Display apparatus for obtaining output image using edge image and control method thereof

This application is a bypass continuation of International Application No. PCT/KR2023/017133, filed on Oct. 31, 2023, which is based on and claims priority to Korean Patent Application No. 10-2022-0143782, filed on Nov. 1, 2022, in the Korean Intellectual Property Office, the disclosures of which are incorporated by reference herein in their entireties.

The disclosure relates to a display apparatus and a control method thereof, and more particularly, to a display apparatus for obtaining an output image using an edge image and a control method thereof.

Various types of display apparatuses are being developed and widely used.

Recently, display apparatuses having viewing assistive functions have been developed and widely used. The viewing assistive functions may include a function of generating subtitles to be output together with images in order to improve a viewing environment for a user with a hearing impairment, a function of outputting images after improving the visibility of the images in order to improve a viewing environment for a user with a visual impairment, and the like.

However, the conventional viewing assistive functions for a user with a visual impairment are simply limited to functions of providing a high contrast mode, providing an image after being enlarged, and providing an image after being subjected to color inversion, without considering a feature of an image (e.g., a meaningful portion in the image), a user's characteristic (e.g., an impairment grade).

There has been a continuous need for a method of outputting an image after improving the visibility of the image by performing image processing in consideration of a grade of a user's impairment or a portion that a user is likely to be interested in within an image, rather than performing uniform image processing for a user with a visual impairment.

According to an example embodiment of the disclosure, a display apparatus includes: a display; a memory storing at least one instruction; and at least one processor connected to the display and the memory to control the display apparatus, wherein the at least one processors are configured to: pre-process an image; obtain an edge image by identifying a contour corresponding to each of a plurality of objects included in the pre-processed image; perform image processing to adjust at least one of a thickness or a color of each of the plurality of contours included in the edge image; obtain an output image by overlaying or blending the edge image subjected to image processing on or with the pre-processed image; and display the output image through the display.

According to an example embodiment of the disclosure, a method of controlling a display apparatus includes: pre-processing an image; obtaining an edge image by identifying a contour corresponding to each of a plurality of objects included in the pre-processed image; performing image processing to adjust at least one of a thickness or a color of each of the plurality of contours included in the edge image; obtaining an output image by overlaying or blending the edge image subjected to image processing on or with the pre-processed image; and displaying the output image.

According to an example embodiment of the disclosure, a non-transitory computer-readable recording medium having recorded thereon a program which, when executed by at least one processor of an electronic device, causes the electronic device to perform operations comprising: pre-processing an image; obtaining an edge image by identifying a contour corresponding to each of a plurality of objects included in the pre-processed image; performing image processing to adjust at least one of a thickness or a color of each of the plurality of contours included in the edge image; obtaining an output image by overlaying or blending the edge image subjected to image processing on or with the pre-processed image; and displaying the output image.

Hereinafter, the disclosure will be described in greater detail with reference to the accompanying drawings.

The terms used in describing the various example embodiments of the disclosure have been selected from general terms that are currently in wide use in consideration of the functions in the disclosure. However, the terms may have different meanings according to intentions of those skilled in the art, precedents, the emergence of new technologies, or the like. Also, various terms may be arbitrarily selected, and in this case, the meanings of the terms will be described in detail in the corresponding description portion of the disclosure. Therefore, the terms used herein should be defined based on the overall content of the disclosure together the meanings of the terms, not simply by the terms themselves.

In the disclosure, the expression “have”, “may have”, “include”, “may include”, or the like indicates the presence of stated features (e.g., numerical values, functions, operations, or components such as parts) and does not preclude the presence of additional features.

It should be understood that the expression “at least one of A and/or B” refers to “A”, or “B”, or “A and B”.

The expressions “first”, “second”, and the like used in the disclosure may modify various components regardless of order and/or importance, and may be used to distinguish one component from another component, and do not limit the components.

It should further be understood that when a component (e.g., a first component) is referred to as being “(operatively or communicatively) coupled with/to” or “connected to” another component (e.g., a second component), this may refer, for example, to a component being coupled with/to or connected to another component directly or via an intervening component (e.g., a third component).

Singular expressions include plural expressions unless the context clearly indicates otherwise. It should be further understood that the term “include” or “comprise” used herein specify the presence of features, numerals, steps, operations, components, parts, or combinations thereof mentioned in the disclosure, but do not preclude the presence or addition of one or more other features, numerals, steps, operations, components, parts, or combinations thereof.

In the disclosure, a “module” or a “unit” may perform at least one function or operation, and may be implemented as hardware, software, or a combination of hardware and software. In addition, a plurality of “modules” or a plurality of “units” may be integrated into at least one module and may be implemented as at least one processor (not shown) except for “modules” or “units” that need to be implemented in specific hardware.

In the disclosure, the term “user” may refer to a person using an electronic device or a device (e.g., an artificial intelligence electronic device) using an electronic device.

Hereinafter, various embodiments of the disclosure will be described in greater detail with reference to the accompanying drawings.

is a diagram illustrating an example viewing assistive function processing unit according to various embodiments.

A display apparatus(refer to) according to various embodiments of the disclosure may provide an image. The display apparatusmay be implemented as a TV, but is not limited thereto, and any apparatus having a display function, such as a video wall, a large format display (LFD), a digital signage, a digital information display (DID), or a projector display, may be applied without limitation. In addition, the display apparatusmay be implemented using various types of displays, such as a liquid crystal display (LCD), an organic light-emitting diode (OLED), a liquid crystal on silicon (LCoS), a digital light processing (DLP), a quantum dot (QD) display panel, and a quantum dot light-emitting diode (QLED).

The display apparatusaccording to an example of the disclosure may include an image input unit (e.g., including input circuitry), an image processing unit (e.g., including image processing circuitry), a viewing assistive function processing unit (e.g., including processing circuitry), and an image output unit (e.g., including output circuitry).

The image input unitmay be included in a communication interface including various communication circuitry configured to receive an image by communicating with an external device (a set top box, a cloud server, etc.). The image input unitmay obtain an image included in a memory.

According to an example, each of the image processing unitand the viewing assistive function processing unitmay be included in one or more processors to process the received image, and provide the processed image through the image output unit(e.g., a display). As used in the disclosure terms such as “processor”, “processing unit” or the like, may include various processing circuitry. For example, and without limitation, as used herein, including the claims, the term “processor”, “processing unit”, or the like may include various processing circuitry, including at least one processor, wherein one or more processors of the at least one processor may be configured to perform the various functions described herein.

For example, the viewing assistive function processing unitmay include various processing circuitry and/or executable program instructions and perform image processing according to special purposes as well as general image processing for images.

For example, the viewing assistive function processing unitmay generate subtitles to be output together with the image processed through the image processing unitfor the convenience of a user with a hearing impairment.

For example, the viewing assistive function processing unitmay generate an image with improved visibility for the convenience of a user with a visual impairment (e.g., a person with low vision).

As an example, the viewing assistive function processing unitmay receive a processed image from the image processing unit, perform visibility enhancement processing on the processed image, and provide the image subjected to the visibility enhancement processing through a display.

As an example, the viewing assistive function processing unitmay receive an image from the image input unitand perform visibility enhancement processing on the received image, and the display apparatusmay provide an image processed by the image processing unitand an image subjected to visibility enhancement processing by the viewing assistive function processing unittogether through a display. For example, the display apparatusmay overlay or blend an image subjected to visibility enhancement processing by the viewing assistive function processing uniton or with an image processed by the image processing unitand provide the image obtained thereby so through a display.

For the convenience of users with visual impairments, conventional display apparatuses provide images in high contrast mode, enlarged images, or images on which a color inversion function has been performed.

However, there has been a demand for a visibility enhancement processing method in which a degree of visual impairment (a visual impairment grade), a feature of an image (for example, a feature of each of a plurality of objects included in an image), or the like is considered, rather than a uniform visibility enhancement processing method (for example, a method in which a feature of an image is not considered) as in the conventional display apparatuses.

The viewing assistive function processing unitaccording to various embodiments of the disclosure may include a pre-processing unit (e.g., including processing circuitry and/or executable program instructions), a contour detection unit (e.g., including processing circuitry and/or executable program instructions), a contour adjustment unit (e.g., including processing circuitry and/or executable program instructions), and a color and contrast control unit (e.g., including processing circuitry and/or executable program instructions), and may perform visibility enhancement processing in consideration of a degree of visual impairment and a feature of an image. As an example, the viewing assistive function processing unitmay identify a contour corresponding to each of a plurality of objects included in an image, and perform visibility enhancement processing by adjusting at least one of a thickness or a color of the identified contour. The display apparatusmay overlay the image subjected to visibility enhancement processing by the viewing assistive function processing uniton the image processed by the image processing unit, and output the image obtained thereby through the image output unit. The display apparatusmay blend the image subjected to visibility enhancement processing by the viewing assistive function processing unitwith the image processed by the image processing unit, and output the image obtained thereby through the image output unit.

Hereinafter, visibility enhancement processing methods of the viewing assistive function processing unitaccording to various embodiments of the disclosure will be described in greater detail.

is a block diagram illustrating an example configuration of a display apparatus according to various embodiments.

Referring to, the display apparatusincludes a display, a memory, and one or more processors (e.g., including various processing circuitry).

The displayaccording to various embodiments of the disclosure may be implemented in various types of displays such as a liquid crystal display (LCD), an organic light-emitting diode (OLED), a liquid crystal on silicon (LCoS), a digital light processing (DLP), and a quantum dot (QD) display panel, a quantum dot light-emitting diode (QLED), a micro light-emitting diode (μLED), and a mini LED.

The display apparatusmay be implemented as a touch screen combined with a touch sensor, a flexible display, a rollable display, a 3D display, a display in which a plurality of display modules are physically connected to each other, or the like.

The memoryaccording to various embodiments of the disclosure may store data required for various embodiments of the disclosure.

The memorymay be implemented in the form of a memory embedded in the display apparatusor in the form of a memory attachable to and detachable from the display apparatus, depending on the data storage purpose. For example, data for driving the display apparatusmay be stored in a memory embedded in the display apparatus, and data for an extended function of the display apparatusmay be stored in a memory attachable to and detachable from the display apparatus. Meanwhile, the memory embedded in the display apparatusmay be implemented as at least one of a volatile memory (e.g., a dynamic RAM (DRAM), a static RAM (SRAM), or a synchronous dynamic RAM (SDRAM)) or a non-volatile memory (e.g., a one time programmable ROM (OTPROM), a programmable ROM (PROM), an erasable and programmable ROM (EPROM), an electrically erasable and programmable ROM (EEPROM), a mask ROM, a flash ROM, a flash memory (e.g., a NAND flash or a NOR flash), a hard drive, or a solid state drive (SSD)). In addition, the memory attachable to and detachable from the display apparatusmay be implemented in the form of, for example, a memory card (e.g., a compact flash (CF), a secure digital (SF), a micro secure digital (Micro-SD), a mini secure digital (Mini-SD), an extreme digital (xD), or a multi-media card (MMC)) or an external memory (e.g., a USB memory) connectable to a USB port.

According to an example, the memorymay store at least one instruction or a computer program including instructions for controlling the display apparatus.

According to various embodiments of the disclosure, various data may be stored in a memory outside the processor, and some of the data may be stored in a memory inside the processorand the other of the data may be stored in a memory outside the processor.

The one or more processorsaccording to various embodiments of the disclosure control the overall operation of the display apparatus.

According to various embodiments of the disclosure, the processorsmay be implemented as a digital signal processor (DSP) that processes a digital signal, a microprocessor, and a timing controller (TCON). However, the processorsare not limited thereto, and may include one or more of a central processing unit (CPU), a micro controller unit (MCU), a micro processing unit (MPU), a controller, an application processor (AP), a communication processor (CP), an ARM processor, and an artificial intelligence (AI) processor, or may be defined by those terms. In addition, the processormay be implemented as a system on chip (SoC) having a built-in processing algorithm or a large scale integration (LSI), or may be implemented in the form of a field programmable gate array (FPGA). The processormay perform various functions by executing computer executable instructions stored in the memory.

The one or more processorsmay include one or more of a central processing unit (CPU), a graphics processing unit (GPU), an accelerated processing unit (APU), a many integrated core (MIC), a digital signal processor (DSP), a neural processing unit (NPU), a hardware accelerator, or a machine learning accelerator. The one or more processorsmay control one or any combination of the other components of the electronic device, and may perform operations related to communication or data processing. The one or more processorsmay execute one or more programs or instructions stored in the memory. For example, the one or more processorsmay perform a method according to various embodiments of the disclosure by executing one or more instructions stored in the memory.

In a case where the method according to various embodiments of the disclosure includes a plurality of operations, the plurality of operations may be performed by one processor or by a plurality of processors. For example, when a first operation, a second operation, and a third operation are performed by the method according to various embodiments, all of the first operation, the second operation, and the third operation may be performed by the first processor, or the first operation and the second operation may be performed by the first processor (e.g., a general-purpose processor) and the third operation may be performed by the second processor (e.g., an artificial intelligence-dedicated processor).

The one or more processorsmay be implemented as single core processors each including one core, or may be implemented by one or more multi-core processors each including a plurality of cores (e.g., multiple homogeneous cores or multiple heterogeneous cores). In a case where the one or more processorsare implemented as multi-core processors, each of the plurality of cores included in the multi-core processor may include a memory such as a cache memory or an on-chip memory inside the processor, and a common cache shared by the plurality of cores may be included in the multi-core processor. In addition, each of the plurality of cores (or some of the plurality of cores) included in the multi-core processor may independently read and execute program instructions for implementing the method according to various embodiments of the disclosure, or all (or some) of the plurality of cores may be linked to each other to read and execute program instructions for implementing the method according to various embodiments of the disclosure.

In a case where the method according to various embodiments of the disclosure includes a plurality of operations, the plurality of operations may be performed by one of the plurality of cores included in the multi-core processor, or may be performed by two or more of the plurality of cores. For example, when a first operation, a second operation, a third operation are performed by the method according to various embodiments, all of the first operation, the second operation, and the third operation may be performed by a first core included in the multi-core processor, or the first operation and the second operation may be performed by the first core included in the multi-core processor, and the third operation may be performed by a second core included in the multi-core processor.

In the embodiments of the disclosure, the processor may refer to a system on chip (SoC) in which one or more processors and the other electronic components are integrated, a single-core processor, a multi-core processor, or a core included in the single-core processor or the multi-core processor. Here, the core may be implemented as a CPU, a GPU, an APU, an MIC, a DSP, an NPU, a hardware accelerator or a machine learning accelerator, but the embodiments of the disclosure are not limited thereto.

In, the one or more processorsare expressed as image processing unitand viewing assistive function processing unitfor convenience of explanation, but each of the image processing unitand the viewing assistive function processing unitmay be included in the one or more processorsas described above. Hereinafter, an operation of each of the image processing unitand the viewing assistive function processing unitwill be described as an operation of the one or more processors.

For example, the one or more processorsaccording to an example may display a region that is meaningful to a user with a visual impairment in an image in a visibility-enhanced manner and display the other region in a blurred manner.