Artificial Intelligence Camera and Operating Method of Artificial Intelligence Camera

2024 This application claims priority under 35 U.S.C. § 119 to Korean Patent Application No. 10-2024-0167193 filed on Nov. 21,and No. 10-2025-0040465 filed on Mar. 28, 2025, in the Korean Intellectual Property Office, the disclosures of which are incorporated by reference herein in their entireties.

Embodiments of the present disclosure described herein relate to an artificial intelligence (AI) camera, and more particularly, relate to an AI camera that identifies patterns or images in real time by using an AI optical system based on an optical technology, and an operating method of the artificial intelligence camera.

An AI technology is a technology that artificially implements human learning, reasoning, and perception abilities by using computers, and is being applied in many fields. In particular, the AI technology is showing excellent performance in the field of image processing based on the technology of processing large amounts of data.

In the meantime, with the advancement of technology, cameras may obtain high-resolution images or videos, and are expected to expand to more applications in the future.

Nowadays, camera systems that incorporate the AI technology are being developed. These camera systems may analyze and determine the large amount of obtained data by using AI, thereby quickly recognizing special situations such as falls, water leaks, and dangerous situations.

A camera system with embedded AI technology consists of a camera and an AI system that analyzes images obtained by the camera. The AI system is implemented as a separate hardware such as a computer or chip depending on application fields. Accordingly, the volume of the camera system increases; a lot of energy is consumed in the process of calculating large amounts of data; and a time delay occurs due to image recognition and discrimination.

To overcome the above-mentioned issues, a single system is required to perform image acquisition and image analysis.

Embodiments of the present disclosure provide an AI camera that identifies patterns or images in real time by using an AI optical system based on an optical technology, and an operating method of the artificial intelligence camera.

According to an embodiment, an artificial intelligence (AI) camera includes a lens, an AI optical system that generates a processed light by performing a neural network operation based on first input light incident through the lens, and an image sensor that generates image data based on the processed light and identifies an image pattern based on the image data.

For example, the AI camera further includes a coherent optical converter that outputs a converted light based on the first input light. The first input light is incoherent light and the converted light is coherent light.

For example, the coherent optical converter includes a light source that outputs second input light, a smart pixel array that generates the converted light based on the first input light and the second input light, and a beam splitter that outputs the first input light and the second input light to the smart pixel array.

For example, the smart pixel array includes a plurality of smart pixels. Each of the plurality of smart pixels includes a first semiconductor element and a second semiconductor element. The first semiconductor element receives the first input light and generates an electrical signal based on the received first input light. The second semiconductor element receives the second input light and controls transmittance or reflectivity of the received second input light based on the electrical signal.

For example, the AI optical system is based on a diffractive optical neural network.

For example, the AI optical system includes at least one diffractive optical element including a plurality of pixels.

For example, the at least one diffractive optical element adjusts at least one of intensity and a phase of light.

For example, the at least one diffractive optical element adjusts the intensity, and each of the plurality of pixels has different transmittance of light.

For example, the at least one diffractive optical element adjusts the phase, and each of the plurality of pixels has a different phase modulation value.

For example, the AI optical system is based on an optical convolutional artificial neural network using a 4F optical system.

For example, the 4F optical system includes a filter including a plurality of pixels, and a filter value of each of the plurality of pixels is determined by using an AI algorithm so as to modulate intensity or a phase of light.

According to an embodiment, a method of operating an AI camera includes receiving first input light reflected from an object through a lens, generating, by an AI optical system, a processed light by performing a neural network operation based on the first input light, generating, by an image sensor, image data based on the processed light, and identifying, by the image sensor, an image pattern based on the image data.

For example, the receiving of the first input light reflected from the object through the lens includes outputting, by a coherent optical converter, a converted light based on the first input light. The first input light is incoherent light and the converted light is coherent light.

For example, the outputting, by the coherent optical converter, of the converted light based on the first input light includes outputting, by a light source, second input light, and generating, by a smart pixel array, the converted light based on the first input light and the second input light.

For example, the smart pixel array includes a smart pixel including a first semiconductor element and a second semiconductor element. The generating, by the smart pixel array, of the converted light based on the first input light and the second input light includes generating, by the first semiconductor element, an electrical signal based on the first input light, and controlling, by the second semiconductor element, transmittance or reflectivity of the second input light based on the electrical signal.

For example, the AI optical system is based on a diffractive optical neural network.

For example, the AI optical system includes at least one diffractive optical element including a plurality of pixels.

For example, the at least one diffractive optical element adjusts intensity of light, and each of the plurality of pixels has different transmittance of light.

For example, the at least one diffractive optical element adjusts a phase of light, and each of the plurality of pixels has a different phase modulation value.

For example, the AI optical system is based on an optical convolutional artificial neural network using a 4F optical system.

Hereinafter, embodiments of the present disclosure will be described in detail and clearly to such an extent that an ordinary one in the art easily implements the present disclosure.

Hereinafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. The above and other aspects, features and advantages of the present disclosure will become apparent from embodiments to be described in detail in conjunction with the accompanying drawings. However, that the present disclosure is not limited to the following embodiments and may be implemented with various forms. Rather, embodiments introduced herein are provided to ensure that disclosed content is thorough and complete and to sufficiently convey the spirit of the present disclosure to those skilled in the art, and the present disclosure is defined only by the scope of claims. The same reference numerals denote the same elements throughout the specification.

The terms used in the specification are provided to describe the embodiments, not to limit the present disclosure. In the specification, the singular forms include plural forms unless particularly mentioned. The words ‘comprises’ and/or ‘comprising’ as used in the specification do not exclude the presence or addition of one or more other components, operations and/or elements in addition to the mentioned components, operations and/or elements. Moreover, because it is according to a preferred embodiment, the reference signs presented in the order of the description are not necessarily limited to that order.

Furthermore, embodiments described herein will be described with reference to cross-sectional and/or perspective views, which are ideal illustrations of the present disclosure. In the drawings, the thicknesses of films and regions are exaggerated to describe the technical features effectively. As such, variations from the shapes of the illustrations as a result, for example, of manufacturing techniques and/or tolerances, are to be expected. Thus, embodiments of the present disclosure are not limited to the specific shapes shown, but also include variations in shape produced by the manufacturing process.

In the detailed description, components described with reference to the terms “unit”, “module”, “block”, “˜er or ˜or”, etc. and function blocks illustrated in drawings will be implemented with software, hardware, or a combination thereof. For example, the software may be a machine code, firmware, an embedded code, and application software. For example, the hardware may include an electrical circuit, an electronic circuit, a processor, a computer, an integrated circuit, integrated circuit cores, a pressure sensor, an inertial sensor, a microelectromechanical system (MEMS), a passive element, or a combination thereof.

1 FIG. shows an example of an AI-based camera system.

1 FIG. 100 110 120 Referring to, a camera systemmay include a cameraand a data processing unit.

110 111 112 111 10 111 112 The cameramay include a lens optical systemand an image sensor. The lens optical systemmay receive input light reflected from an object(or the outside). The lens optical systemmay deliver input light to the image sensor.

112 111 112 111 112 112 The image sensormay generate image data based on the input light incident through the lens optical system. For example, the image sensormay convert the input light incident through the lens optical systeminto an electrical signal. The image sensormay generate image data, which is digital data, based on the electrical signal. The image sensormay output the image data.

120 121 122 121 112 121 121 The data processing unitmay include an image processing unitand a data identification unit. The image processing unitmay receive image data from the image sensor. The image processing unitmay process the image data based on an AI technology. For example, the image processing unitmay analyze the image data for a specific purpose based on the AI technology.

122 121 122 122 The data identification unitmay receive the analyzed image data from the image processing unit. The data identification unitmay identify the analyzed image data based on the AI technology. For example, the data identification unitmay identify an image pattern or an image from the analyzed image data based on the AI technology.

1 FIG. 120 100 In, the data processing unitmay be implemented as a separate chip or computing device (e.g., a computer). As a result, the volume of the camera systemincreases; a lot of energy is consumed in the process of processing large amounts of data; and, time delay occurs due to image recognition and discrimination.

2 FIG. shows an example of an AI camera, according to an embodiment of the present disclosure.

2 FIG. 200 210 220 230 Referring to, an AI cameramay include a lens optical system, an AI optical system, and an image sensor.

210 10 210 220 210 210 210 210 The lens optical systemmay receive input light reflected from the object(or the outside). The lens optical systemmay deliver the input light to the AI optical system. In an embodiment, the lens optical systemis movable. As the location of the lens optical systemchanges according to the movement of the lens optical system, the focal length of the lens optical systemmay change.

220 210 220 220 230 The AI optical systemmay perform neural network operations based on the input light incident through the lens optical system. For example, the AI optical systemmay perform neural network operations to process and analyze the input light or an optical image formed by the input light. As a result of performing the neural network operations, the AI optical systemmay generate a processed light and may output the processed light thus generated to the image sensor.

220 In an embodiment, the AI optical systemmay be implemented based on a diffractive optical neural network or an optical convolutional artificial neural network. However, the scope of the present disclosure is not limited thereto and may be implemented based on various optical neural networks.

230 230 220 230 230 The image sensormay identify an image pattern or an image based on the processed light. For example, the image sensormay receive the processed light from the AI optical system. The image sensormay generate image data, which is digital data, based on the processed light. The image sensormay identify an image pattern or an image from the image data.

2 FIG. 200 In, the AI cameramay implement an AI technology by using an optical method, thereby reducing power consumption and performing neural network operations at the speed of light.

3 FIG. 2 FIG. 3 FIG. 300 illustrates a specific example of the AI camera of. In, an AI cameramay perform neural network operations based on a diffractive optical neural network.

2 3 FIGS.and 3 FIG. 2 FIG. 2 FIG. 2 FIG. 300 310 320 330 310 210 320 220 330 230 Referring to, the AI cameramay include an lens, an AI optical system, and an image sensor. In, the lensmay correspond to the lens optical systemof; the AI optical systemmay correspond to the AI optical systemof; and the image sensormay correspond to the image sensorof. Accordingly, for convenience of description, redundant descriptions are omitted.

320 321 1 321 321 1 321 321 1 321 320 330 n n n The AI optical systemmay include a plurality of amplitude diffractive optical elements_to_. Each of the plurality of amplitude diffractive optical elements_to_may perform a neural network operation. For example, each of the plurality of amplitude diffractive optical elements_to_may perform a neural network operation based on the diffraction and interference of light. The neural network operation may include a linear operation and a nonlinear operation. As a result of performing the neural network operations, the AI optical systemmay generate the processed light and may deliver the processed light thus generated to the image sensor.

In an embodiment, the linear operation may be performed based on the diffraction of light.

In an embodiment, after the linear operation is performed, the nonlinear operation may be performed based on the interference of light. For example, an activation function operation may be performed based on the interference of light. Light on which the nonlinear operation is performed may have a specific pattern (e.g., a specific optical pattern).

3 FIG. 320 321 1 321 320 320 n In, the AI optical systemis illustrated as including the plurality of amplitude diffractive optical elements_to_, but the AI optical systemmay include one amplitude diffractive optical element. That is, the AI optical systemmay include at least one amplitude diffractive optical element.

300 In an embodiment, as the number of amplitude diffractive optical elements increases, the identification accuracy of the image pattern of the AI cameramay be improved.

4 FIG. shows an example of an amplitude diffractive optical element, according to an embodiment of the present disclosure.

4 FIG. 400 Referring to, an amplitude diffractive optical elementmay include a plurality of pixels PX. At least some of the plurality of pixels PX may have different light transmittances (or reflectivity). That is, when light of the same intensity is incident on the plurality of pixels PX, the intensity of the transmitted light may vary for each pixel PX. For example, at least some of the plurality of pixels PX may have different light transmittances, while the others thereof may have the same light transmittance. Alternatively, the plurality of pixels PX may have different light transmittances, respectively.

In an embodiment, the light transmittance of at least some of the plurality of pixels PX may be 0%. That is, at least some of the plurality of pixels PX may completely block light.

In an embodiment, the light transmittance of at least some of the plurality of pixels PX may be 100 %. That is, at least some of the plurality of pixels PX may completely transmit light.

In an embodiment, the light transmittance of each of the plurality of pixels PX may be determined based on an AI algorithm. For example, the light transmittance of each of the plurality of pixels PX may be determined in advance according to purposes.

5 FIG. shows an example of an AI camera, according to an embodiment of the present disclosure.

5 FIG. 5 FIG. 2 FIG. 2 FIG. 2 FIG. 500 510 520 530 540 510 210 530 220 540 230 Referring to, an AI cameramay include a lens optical system, a coherent optical converter, an AI optical system, and an image sensor. In, the lens optical systemmay correspond to the lens optical systemof; the AI optical systemmay correspond to the AI optical systemof; and the image sensormay correspond to the image sensorof. Accordingly, for convenience of description, redundant descriptions are omitted.

520 520 510 520 The coherent optical convertermay convert light that does not have coherence (hereinafter referred to as ‘incoherent light’) into light that has coherence (hereinafter referred to as ‘coherent light’). For example, the coherent optical convertermay receive first input light incident through the lens optical system. The first input light may be incoherent light. The coherent optical convertermay output a converted light by converting the first input light into coherent light.

530 520 The AI optical systemmay perform a neural network operation based on the converted light output from the coherent optical converter.

6 FIG. 5 FIG. shows a specific example of the coherent light converter of.

5 6 FIGS.and 600 610 620 630 Referring to, a coherent optical convertermay include a light source, a beam splitter, and a smart pixel array.

610 620 The light sourcemay output second input light, which is seed light, to the beam splitter. The seed light may represent light used to convert incoherent light into coherent light. In an embodiment, the seed light may be a perfectly coherent light (hereinafter referred to as ‘fully-coherent light’).

610 In an embodiment, the light sourcemay be a laser.

620 510 610 The beam splittermay receive first input light, which is incoherent light, through the lens optical system, and may receive the second input light from the light source. In an embodiment, the first input light may form a first optical image. In this case, the first optical image may be an incoherent image.

620 620 630 The beam splittermay reflect or transmit light. For example, the beam splittermay transmit at least part of the first input light. The transmitted first input light may propagate toward the smart pixel array.

620 620 630 For example, the beam splittermay reflect at least part of the second input light. As a result of the reflection, the beam splittermay change the path of the second input light. The reflected second input light may propagate toward the smart pixel array.

630 620 630 The smart pixel arraymay receive the first input light and the second input light from the beam splitter. The smart pixel arraymay output a converted light, which is coherent light, based on the first input light and the second input light. In an embodiment, the converted light may form a second optical image. In this case, the second optical image may be a coherent image.

7 FIG. 6 FIG. 7 FIG. 6 FIG. 6 FIG. 6 FIG. shows a specific example of the smart pixel array of. In, it is assumed that first incoherent light, second incoherent light, and third incoherent light are included in the first input light of. It is assumed that first fully-coherent light, second fully-coherent light, and third fully-coherent light are included in the second input light of. Moreover, it is assumed that first coherent light, second coherent light, and third coherent light are included in the converted light of.

6 7 FIGS.and 630 Referring to, the smart pixel arraymay include first to third smart pixels SPX1 to SPX3. Each of the first to third smart pixels SPX1 to SPX3 may convert incoherent light into coherent light.

For example, the first smart pixel SPX1 may generate first coherent light based on the first incoherent light and the first fully-coherent light. The second smart pixel SPX2 may generate second coherent light based on the second incoherent light and the second fully-coherent light. The third smart pixel SPX3 may generate third coherent light based on the third incoherent light and the third fully-coherent light.

In an embodiment, the wavelengths of the first incoherent light, the second incoherent light, and the third incoherent light may be different from each other. Furthermore, the wavelengths of the first coherent light, the second coherent light, and the third coherent light may be different from each other.

In an embodiment, each of the first to third smart pixels SPX1 to SPX3 may control the intensity of light transmitted or reflected.

7 FIG. 630 630 illustrates that the smart pixel arrayincludes the first to third smart pixels SPX1 to SPX3. However, the scope of the present disclosure is not limited thereto. The smart pixel arraymay include a plurality of smart pixels.

8 FIG. 7 FIG. illustrates a specific example of the first smart pixel of.

6 8 FIGS.to Referring to, the first smart pixel SPX1 may include a first semiconductor element and a second semiconductor element. Each of the first semiconductor element and the second semiconductor element may include an N-type semiconductor layer, an intrinsic semiconductor layer, and a P-type semiconductor layer.

A bias voltage may be applied to the N-type semiconductor layer of the first semiconductor element, and the P-type semiconductor layer of the first semiconductor element may be electrically connected to the N-type semiconductor layer of the second semiconductor element. A ground voltage may be applied to the P-type semiconductor layer of the second semiconductor element.

The first semiconductor element may receive first incoherent light. The first semiconductor element may generate an electrical signal based on the first incoherent light. The generated electrical signal may be delivered from the P-type semiconductor layer of the first semiconductor element to the N-type semiconductor layer of the second semiconductor element.

The second semiconductor element may control the transmittance or reflectivity of the first fully-coherent light based on the electrical signal delivered from the first semiconductor element. As the result of controlling the transmittance or reflectivity of the first fully-coherent light, the intensity of the first coherent light may be adjusted.

In an embodiment, the wavelengths of the first incoherent light and the first fully-coherent light may be different from or the same as each other.

8 FIG. 630 In, the first smart pixel SPX1 is described as an example, but each of the remaining smart pixels SPX2 and SPX3 included in the smart pixel arrayoperates identically or similarly to the first smart pixel SPX1.

9 FIG. 5 FIG. 9 FIG. 700 shows a specific example of the AI camera of. In, an AI cameramay perform neural network operations based on a diffractive optical neural network.

5 9 FIGS.to 9 FIG. 5 FIG. 5 FIG. 5 FIG. 5 FIG. 700 710 720 730 740 710 510 720 520 730 530 740 540 Referring to, the AI cameramay include a lens, a coherent optical converter, an AI optical system, and an image sensor. In, the lensmay correspond to the lens optical systemof; the coherent optical convertermay correspond to the coherent optical converterof; the AI optical systemmay correspond to the AI optical systemof; and the image sensormay correspond to the image sensorof. Accordingly, for convenience of description, redundant descriptions are omitted.

730 731 1 731 731 1 731 731 1 731 730 740 n n n The AI optical systemmay include a plurality of phase diffractive optical elements_to_. Each of the plurality of phase diffractive optical elements_to_may perform a neural network operation. For example, each of the plurality of phase diffractive optical elements_to_may perform neural network operations based on diffraction, phase modulation, and interference of light. As a result of performing the neural network operations, the AI optical systemmay generate the processed light and may deliver the processed light thus generated to the image sensor.

In an embodiment, linear operations may be performed based on light diffraction and phase modulation. In this case, a weighting operation of the linear operation may be performed based on a preset phase modulation value (e.g., a trained phase modulation value).

731 1 731 731 1 731 n n Each of the plurality of phase diffractive optical elements_to_may include a plurality of pixels. In each of the plurality of phase diffractive optical elements_to_, at least some of a plurality of pixels may have different phase modulation values. That is, when light having the same phase is incident on the plurality of pixels, the phase of the transmitted light may vary for each pixel. For example, at least some of the plurality of pixels may have different phase modulation values, and the others thereof may have the same phase modulation value. Alternatively, each of the plurality of pixels may have different phase modulation values from each other.

In an embodiment, the phase modulation value of each of the plurality of pixels may be determined based on an AI algorithm. For example, the phase modulation values of each of a plurality of pixels may be predetermined according to purposes.

730 731 1 731 n In an embodiment, the AI optical systemmay train weights and may continuously adjust values of the weights based on the neural network operations of the plurality of phase diffractive optical elements_to_.

9 FIG. 730 731 1 731 730 730 n In, the AI optical systemis illustrated as including the plurality of phase diffractive optical elements_to_, but the AI optical systemmay include one phase diffractive optical element. That is, the AI optical systemmay include at least one phase diffractive optical element.

730 731 1 731 n. In an embodiment, the AI optical systemmay include a plurality of diffractive optical elements that simultaneously modulate the intensity and phase of light instead of the plurality of phase diffractive optical elements_to_

10 FIG. 5 FIG. 10 FIG. 800 illustrates a specific example of the AI camera of. In, an AI cameramay perform a neural network operation based on an optical convolutional artificial neural network.

5 8 10 FIGS.toand 10 FIG. 5 FIG. 5 FIG. 5 FIG. 5 FIG. 800 810 820 830 840 810 510 820 520 830 530 840 540 Referring to, the AI cameramay include a lens, a coherent optical converter, an AI optical system, and an image sensor. In, the lensmay correspond to the lens optical systemof; the coherent optical convertermay correspond to the coherent optical converterof; the AI optical systemmay correspond to the AI optical systemof; and the image sensormay correspond to the image sensorof. Accordingly, for convenience of description, redundant descriptions are omitted.

830 830 831 832 832 The AI optical systemmay be implemented based on a 4F optical system. The AI optical systemmay include a first lens, a filter, and a second lens.

832 832 720 The filtermay include a plurality of pixels. The filter value of each of the plurality of pixels may be determined by using an AI algorithm so as to modulate the intensity or phase of light. When the filtermodulates the intensity of light, the coherent optical convertermay be omitted.

In the above embodiments, components according to the present disclosure are described by using the terms “first”, “second”, “third”, etc. However, the terms “first”, “second”, “third”, etc. may be used to distinguish components from each other and do not limit the present disclosure. For example, the terms “first”, “second”, “third”, etc. do not involve an order or a numerical meaning of any form.

The above-mentioned description refers to embodiments for implementing the scope of the present disclosure. Embodiments in which a design is changed simply or which are easily changed may be included in the scope of the present disclosure as well as an embodiment described above. In addition, technologies that are easily changed and implemented by using the above-mentioned embodiments may be also included in the scope of the present disclosure.

An AI camera according to an embodiment of the present disclosure may reduce the volume of the entire system, by utilizing an AI optical system instead of separate hardware for applying an AI technology.

The AI camera according to an embodiment of the present disclosure processes data by using an optical system, thereby reducing power consumption and eliminating latency. Accordingly, the AI camera may recognize and discriminate image patterns or images in real time.

While the present disclosure has been described with reference to embodiments thereof, it will be apparent to those of ordinary skill in the art that various changes and modifications may be made thereto without departing from the spirit and scope of the present disclosure as set forth in the following claims.