Electronic Device and Operation Method Thereof

This application is a continuation application of U.S. patent application Ser. No. 17/828,111, filed on May 31, 2022 which is a continuation application, under 35 U.S.C. § 111(a), of International Application No. PCT/KR2022/006942, filed May 13, 2022, which claims priority to Korean patent application No. 10-2021-0102661, filed Aug. 4, 2021, the entire disclosures of all of which are herein incorporated by reference as a part of this application.

Various embodiments of the disclosure relate to an electronic device and an operation method thereof, and more particularly, to an electronic device for performing image quality processing, and an operation method thereof.

With the increase in the supply of electronic devices capable of outputting high-resolution content according to technological developments, the high-resolution content therefor is also gradually increasing. When the resolution of content input to the electronic devices is increased, the size of an intellectual property (IP) block for image quality processing also needs to be increased. However, when the size of the IP block is increased, various issues, such as costs, complexity, power consumption, heat generation, and the like, may occur.

Accordingly, a high-efficiency image quality processing technology for performing image quality processing on an ultra-high resolution image without increasing a size of an IP block is required.

According to an embodiment of the disclosure, a method of an electronic device, includes: obtaining a low-resolution input image by down-sampling a high-resolution input image; obtaining a low-resolution output image by performing image quality processing on the low-resolution input image; obtaining a low-resolution model from a conversion relationship between the low-resolution input image prior to the image quality processing being performed and the low-resolution output image subsequent to the image quality processing being performed; obtaining a high-resolution model based on the low-resolution model; and obtaining a high-resolution output image from the high-resolution input image, by applying the high-resolution model to the high-resolution input image.

The down-sampling of the high-resolution input image may include: dividing the high-resolution input image into a plurality of pixel groups; and selecting a pixel of which a pixel value change amount in the plurality of pixel groups is equal to or greater than a reference value.

The obtaining of the low-resolution model may include obtaining, as the low-resolution model, a linear regression model indicating a conversion relationship between a pixel value of an input pixel of the low-resolution input image and a pixel value of an output pixel of the low-resolution output image, the output pixel corresponding to the input pixel.

The linear regression model may be obtained by using a conversion relationship between pixel values of a reference region including the input pixel and pixel values of a region corresponding to the reference region and including the output pixel.

The obtaining of the low-resolution model may further include: obtaining pixel value change amounts of pixels included in a standard region in the low-resolution input image, the standard region including the input pixel; and obtaining the reference region with an adjusted size by adjusting a size of the standard region according to the pixel value change amounts.

The obtaining of the high-resolution model, based on the low-resolution model may include: up-sampling the low-resolution model; and obtaining the high-resolution model by modifying the up-sampled low-resolution model, based on a difference between the high-resolution input image and the low-resolution input image.

The up-sampling of the low-resolution model may include obtaining a linear regression model regarding pixels of the high-resolution input image down-sampled to the input pixel by using the linear regression model regarding the input pixel.

The input pixel may be a first input pixel and the up-sampling of the low-resolution model may include obtaining the linear regression model regarding the pixels of the high-resolution input image down-sampled to the first input pixel, by interpolating the linear regression model regarding the first input pixel and a linear regression model regarding a second input pixel adjacent to the first input pixel.

The obtaining of the high-resolution model may include: obtaining a reliability weight for each pixel of the high-resolution input image, based on a pixel value difference between the high-resolution input image and the low-resolution input image; and obtaining the high-resolution model by applying the reliability weight to the up-sampled low-resolution model.

The reliability weight may be obtained based on a pixel value difference between a certain region of the high-resolution input image and the low-resolution input image obtained by being down-sampled from the certain region.

According to another embodiment of the disclosure, an electronic device includes: a signal processor; a memory storing one or more instructions; and a processor configured to execute the one or more instructions stored in the memory to control the signal processor, wherein the signal processor is configured to, according to control by the processor: obtain a low-resolution input image by down-sampling a high-resolution input image; obtain a low-resolution output image by performing image quality processing on the low-resolution input image; obtain a low-resolution model from a conversion relationship between the low-resolution input image prior to the image quality processing being performed and the low-resolution output image subsequent to the image quality processing being performed; obtain a high-resolution model, based on the low-resolution model; and obtain a high-resolution output image from the high-resolution input image, by applying the high-resolution model to the high-resolution input image.

According to an embodiment of the disclosure, a non-transitory computer-readable recording medium has recorded thereon a program for implementing an operation of an electronic device, the operation method including: obtaining a low-resolution input image by down-sampling a high-resolution input image; obtaining a low-resolution output image by performing image quality processing on the low-resolution input image; obtaining a low-resolution model from a conversion relationship between the low-resolution input image prior to the image quality processing being performed and the low-resolution output image subsequent to the image quality processing being performed; obtaining a high-resolution model based on the low-resolution model; and obtaining a high-resolution output image from the high-resolution input image, by applying the high-resolution model to the high-resolution input image.

Embodiments described in the specification and configurations illustrated in the drawings are merely preferred examples of the embodiments of the disclosure, and may be modified in various different ways at the time of filing of the present application to replace the embodiments and drawings of the specification.

Throughout the disclosure, the expression “at least one of a, b or c” indicates only a, only b, only c, both a and b, both a and c, both b and c, all of a, b, and c, or variations thereof.

Hereinafter, embodiments of the disclosure will be described in detail with reference to the accompanying drawings such that one of ordinary skill in the art may easily implement the disclosure. However, the disclosure may be implemented in various different forms and is not limited to the embodiments of the disclosure described herein.

Terms used in the disclosure are described as general terms currently used in consideration of functions described in the disclosure, but the terms may have different meanings according to an intention of one of ordinary skill in the art, precedent cases, or the appearance of new technologies. Thus, the terms used herein should not be interpreted only by its name, but have to be defined based on the meaning of the terms together with the description throughout the specification.

Also, the terms used in the disclosure are only used to describe specific embodiments of the disclosure, and are not intended to limit the disclosure.

Throughout the specification, when a part is “connected” to another part, the part may not only be “directly connected” to the other part, but may also be “electrically connected” to the other part with another element in between.

“The” and similar directives used in the present specification, in particular, in claims, may indicate both singular and plural. Also, unless there is a clear description of an order of operations describing a method according to the disclosure, the operations described may be performed in a suitable order. The disclosure is not limited by the order of description of the described operations.

The phrases “some embodiments of the disclosure” or “an embodiment of the disclosure” appearing in various places in this specification are not necessarily all referring to the same embodiment of the disclosure.

Some embodiments of the disclosure may be represented by functional block configurations and various processing operations. Some or all of these functional blocks may be implemented by various numbers of hardware and/or software configurations that perform particular functions. For example, the functional blocks of the disclosure may be implemented by one or more microprocessors or by circuit configurations for a certain function. Also, for example, the functional blocks of the disclosure may be implemented in various programming or scripting languages. The functional blocks may be implemented by algorithms executed in one or more processors. In addition, the disclosure may employ general techniques for electronic environment setting, signal processing, and/or data processing. Terms such as “mechanism”, “element”, “means”, and “configuration” may be used widely and are not limited as mechanical and physical configurations.

In addition, a connection line or a connection member between components shown in drawings is merely a functional connection and/or a physical or circuit connection. In an actual device, connections between components may be represented by various functional connections, physical connections, or circuit connections that are replaceable or added.

In addition, terms such as “unit” and “module” described in the specification denote a unit that processes at least one function or operation, which may be implemented in hardware or software, or implemented in a combination of hardware and software.

In embodiments of the disclosure, the term “user” denotes a person who controls functions or operations of an electronic device by using the electronic device, and may include a viewer, a consumer, a manager, or an installation engineer.

Provided are an electronic device for obtaining a reliability weight, based on a similarity between a high-resolution image and a low-resolution image obtained by down-sampling the high-resolution image, and an operation method thereof.

Provided are an electronic device for obtaining a low-resolution model, based on a conversion relationship between low-resolution images before and after image processing, and an operation method thereof.

Provided are an electronic device for obtaining a high-resolution model by using a low-resolution model and a reliability weight, and obtaining a high-resolution output image from a high-resolution input image by using the high-resolution model, and an operation method thereof.

An electronic device and an operation method thereof, according to an embodiment of the disclosure, can obtain a reliability weight, based on a similarity between a high-resolution image and a low-resolution image obtained by down-sampling the high-resolution image.

An electronic device and an operation method thereof, according to an embodiment of the disclosure, can obtain a low-resolution model, based on a conversion relationship between low-resolution images before and after image quality processing.

An electronic device and an operation method thereof, according to an embodiment of the disclosure, can obtain a high-resolution model by using a low-resolution model and a reliability weight, and obtain a high-resolution output image from a high-resolution input image by using the high-resolution model.

Hereinafter, embodiments of the disclosure will be described in detail with reference to accompanying drawings.

is a diagram for describing a display deviceoutputting a high-resolution output imageby receiving a high-resolution input image, according to an embodiment of the disclosure.

Referring to, the display devicemay be an electronic device capable of processing and outputting an image. The display devicemay be a fixed type or a movable type, and may be a digital television (TV) capable of receiving a digital broadcast, but is not limited thereto and may be implemented as various types of electronic devices including a display.

The display devicemay include at least one of a desktop computer, a smartphone, a tablet personal computer (PC), a mobile phone, a video phone, an e-book reader, a laptop PC, a netbook computer, a digital camera, a personal digital assistant (PDA), a portable multimedia player (PMP), a camcorder, a navigation device, a wearable device, a smart watch, a home network system, a security system, or a medical device.

The display devicemay be implemented as not only a flat display device, but also a curved display device with a screen having a curvature or a flexible display device with an adjustable curvature. Output definition of the display devicemay include, for example, high definition (HD), full HD, ultra HD, or definition clearer than ultra HD.

The display devicemay output an image. The image is visual information displayable on a screen, and may include a still image or a video. The video may include a plurality of frames. The video may include TV programs provided by content providers or items, such as various movies or TV series, via video-on-demand (VOD) services. The content provider may refer to a terrestrial broadcasting station or cable broadcasting station providing, to a user, various types of content including the image, an over-the-top service provider, or an internet protocol TV (IPTV) service provider.

With the increase in supply of electronic devices capable of outputting high-resolution content, high-resolution content provided by the content provider is also increasing. Because the number of pixels for representing an image is increased when resolution of content is increased, the number of pixels to be processed by an electronic device during a same time is also increased. Accordingly, to process the high-resolution content, a size of a signal processing unit performing image quality processing in the electronic device needs to correspond to the resolution of the content. The signal processing unit may include, for example, an intellectual property (IP) block. The IP block is a semiconductor IP core and may denote a reusable unit of logic, cell, or integrated circuit layout design that is the intellectual property of one party. However, when a size of the IP block is increased, issues, such as complexity, costs, power consumption, and heat generation, may occur.

According to an embodiment of the disclosure, the display devicemay process a high-resolution image with low complexity, by using an image quality processing IP block capable of processing low-resolution content.

In this regard, upon receiving the high-resolution input image, the display devicemay down-sample the received high-resolution input image, thereby obtaining a low-resolution input image. For example, when the high-resolution input imageis an image having resolution of 8 K, and an image processible by the display deviceis an image having resolution of 4 K, the display devicemay obtain the low-resolution input image having resolution of 4 K by down-sampling the high-resolution input image.

According to an embodiment of the disclosure, the display devicemay obtain a low-resolution output image by performing image quality processing on the low-resolution input image. For example, the display devicemay obtain the low-resolution output image of 4 K by performing image quality processing on the low-resolution input image having resolution of 4 K, by using an image quality processing IP block having low complexity.

According to an embodiment of the disclosure, the display devicemay obtain a low-resolution model from a conversion relationship between the low-resolution input image and the low-resolution output image. The display devicemay obtain the low-resolution model for each pixel, by using a relationship between a pixel value of the low-resolution input image and a pixel value of the low-resolution output image.

According to an embodiment of the disclosure, the display devicemay obtain a high-resolution model, based on the low-resolution model.

According to an embodiment of the disclosure, the display devicemay obtain a high-resolution output imagefrom the high-resolution input image, by using the high-resolution model. In other words, in the above example, the display devicemay obtain the high-resolution output imagehaving resolution of 8 K by applying the high-resolution model to the high-resolution input imagehaving resolution of 8 K.

According to another embodiment of the disclosure, the image quality processing IP block may not be included in the display device, but may be included in a device separate from the display device. For example, the image quality processing IP block performing the image quality processing may be included in an external computing device or server. In this case, the display devicemay transmit the high-resolution input imageto the computing device or serverthrough a communication network.

The servermay down-sample the high-resolution input imagereceived from the display deviceto an image having processible resolution, and perform image quality processing on the down-sampled low-resolution input image. The servermay obtain the low-resolution model from the conversion relationship between the low-resolution input image after the image quality processing and the low-resolution input image before the image quality processing, and obtain the high-resolution model, based on the low-resolution model. The servermay obtain the high-resolution output imageby applying the high-resolution model to the high-resolution input image, and transmit the high-resolution output imageto the display devicevia the communication network.

The display devicemay output, on a screen, the high-resolution output imagereceived from the server.