Method, Apparatus, and Recording Medium Storing Commands for Processing Scanned Image of Intraoral Scanner

The present disclosure relates to a method for processing a scanned image from an intraoral scanner, an apparatus for performing the method, and a recording medium in which commands for performing the method are recorded. More specifically, the present disclosure relates to a method for filtering out noise existing in a three-dimensional image generated based on a scanned image from an intraoral scanner, etc.

A three-dimensional intraoral scanner is an optical instrument that is inserted into an oral cavity of a patient, is used for scanning of the oral cavity, and thus acquires a three-dimensional image of the oral cavity. Specifically, the three-dimensional intraoral scanner may acquire multiple two-dimensional images of the oral cavity of the patient, and the acquired multiple two-dimensional images are used for post-image processing, whereby a three-dimensional image for the oral cavity of the patient can be generated.

The generated three-dimensional image is an image based on the multiple two-dimensional images, and thus whether there is noise (e.g., a doctor's finger, a treatment instrument, a cheek soft tissue, the tongue soft tissue, etc.) represented in the two-dimensional images has a considerable effect on the quality of the three-dimensional image.

The quality of the three-dimensional image of an oral cavity of a patient has a direct effect on a series of dental treatment processes such as establishment of the patient's treatment plan, review of the patient's treatment process, and review of the patient's treatment results, and thus there is a need for a technology for generating a high-quality three-dimensional image of the oral cavity of the patient. In addition, as a technology for generating a high-quality three-dimensional image, a technology for appropriately filtering out noise represented in a two-dimensional image is required. Above all, since soft tissue, which is a type of noise, is represented by an attribute on a two-dimensional image that is similar to that of a region of interest (e.g., a tooth, the gingiva, etc.), a technology for effectively filtering out such soft tissue is required.

The present disclosure provides a method for determining a region of interest on a two-dimensional image, the region being obtained after filtering out noise, based on a difference in depth from a region represented on the two-dimensional image of an oral cavity of a patient. In addition, the present disclosure provides a method for generating a high-quality three-dimensional image of an oral cavity of a patient by using a two-dimensional image for generation of the three-dimensional image, the two-dimensional image being obtained after filtering out noise.

The technical problems to be solved in the present disclosure are not limited to the mentioned technical problems, and other unmentioned technical problems can be clearly understood by those skilled in the art from the description below.

An image processing method performed by an electronic device according to some embodiments may include: identifying a tooth region in a two-dimensional image of a target oral cavity; identifying, in the two-dimensional image, a first neighboring region located within a predetermined distance from a boundary of the tooth region; determining, based on a difference in depth between the tooth region and the first neighboring region, whether to include the first neighboring region in a region of interest; and generating a three-dimensional image of the target oral cavity from the two-dimensional image which includes the region of interest.

In some embodiments, the identifying the tooth region may include identifying the tooth region in the two-dimensional image by using a tooth segmentation model constructed according to a machine learning algorithm, and the tooth segmentation model may correspond to a model trained by modeling a correlation between a training image set of a tooth and a segmentation result image set corresponding to the training image set.

In some embodiments, the determining whether to include the first neighboring region in the region of interest may include comparing a first coordinate corresponding to the tooth region with a second coordinate corresponding to the first neighboring region to calculate the difference in depth, and each of the first coordinate and the second coordinate may correspond to a coordinate obtained using an intraoral scanner linked to the electronic device. Here, each of the first coordinate and the second coordinate may be a coordinate calculated based on a position of a first camera, a position of a second camera distinguished from the first camera, an image captured by the first camera, and an image captured by the second camera, and the first camera and the second camera may be cameras provided in the intraoral scanner. Further, each of the first coordinate and the second coordinate may be a coordinate obtained through monocular depth estimation of the two-dimensional image captured from the intraoral scanner.

In some embodiments, the determining of whether to include the first neighboring region in the region of interest may include excluding the first neighboring region from the region of interest when the difference in depth is equal to or greater than a threshold. Here, the excluding the first neighboring region from the region of interest may include even when a difference in depth between the first neighboring region and a second neighboring region located within the predetermined distance from the boundary of the first neighboring region is less than the threshold, excluding the second neighboring region from the region of interest.

In some embodiments, the determining whether to include the first neighboring region in the region of interest may include when the difference in depth is less than a threshold, including the first neighboring region in the region of interest. Here, the including the first neighboring region in the region of interest may include repeatedly expanding the region of interest until the difference in depth between the region of interest and a third neighboring region located within the predetermined distance from the boundary of the region of interest becomes equal to or greater than the threshold. Further, a distance between the boundary of the region of interest and the third neighboring region may increase as an expansion count increases. Furthermore, the repeatedly expanding of the region of interest may include when an expansion count is equal to or greater than a reference count, suspending an expansion of the region of interest even when the difference in depth between the region of interest and the third neighboring region is less than the threshold.

In some embodiments, the image processing method may further include displaying the region of interest by highlighting the region of interest on the two-dimensional image or displaying the region of interest by highlighting the region of interest on the three-dimensional image.

An electronic device according to some embodiments may include a processor, a network interface communicatively connected to an intraoral scanner, a display, a memory, and a computer program loaded onto the memory and executed by the processor, wherein the computer program comprises instructions for: identifying a tooth region in a two-dimensional image of a target oral cavity; identifying, in the two-dimensional image, a first neighboring region located within a predetermined distance from a boundary of the tooth region; determining, based on a difference in depth between the tooth region and the first neighboring region, whether to include the first neighboring region in a region of interest, and generating a three-dimensional image of the target oral cavity from the two-dimensional image which includes the region of interest.

In some embodiments, the instruction for determining whether to include the first neighboring region in the region of interest may comprise an instruction for when the difference in depth is equal to or greater than a threshold, excluding the first neighboring region from the region of interest, and when the difference in depth is less than the threshold, including the first neighboring region in the region of interest.

In some embodiments, the computer program may further comprise an instruction for displaying the region of interest by highlighting the region of interest on the two-dimensional image or displaying the region of interest by highlighting the region of interest on the three-dimensional image.

According to some embodiments, a non-transitory computer-readable recording medium in which a computer program to be executed by a processor is recorded, the computer program may comprise instructions for: identifying a tooth region in a two-dimensional image of a target oral cavity; identifying, in the two-dimensional image, a first neighboring region located within a predetermined distance from a boundary of the tooth region; determining, based on a difference in depth between the tooth region and the first neighboring region, whether to include the first neighboring region in a region of interest; and generating a three-dimensional image of the target oral cavity from the two-dimensional image which includes the region of interest.

In some embodiments, the computer program may further comprise an instruction for displaying the region of interest by highlighting the region of interest on the two-dimensional image or displaying the region of interest by highlighting the region of interest on the three-dimensional image.

Embodiments of the present disclosure are illustrated for describing the technical spirit of the present disclosure. The scope of the claims according to the present disclosure is not limited to the embodiments described below or to the detailed descriptions of these embodiments.

All technical or scientific terms used herein have meanings that are generally understood by a person having ordinary knowledge in the art to which the present disclosure pertains, unless otherwise specified. The terms used herein are selected for more clear illustration of the present disclosure, and are not intended to limit the scope of the claims in accordance with the present disclosure.

The expressions “include,” “provided with,” “have” and the like used herein should be understood as open-ended terms connoting the possibility of including other embodiments, unless otherwise mentioned in a phrase or sentence including the expressions.

A singular expression can include meanings of plurality, unless otherwise mentioned, and the same is applied to a singular expression stated in the claims. In addition, the terms “first,” “second,” etc. used herein are used to distinguish a plurality of components from one another, and are not intended to limit the order or importance of the relevant components.

The term “unit” used in these embodiments means a software component or hardware component, such as a field-programmable gate array (FPGA) and an application specific integrated circuit (ASIC). However, a “unit” is not limited to software and hardware, and it may be configured to be an addressable storage medium or may be configured to run on one or more processors. Accordingly, as examples what a “unit” may mean, a “unit” may include components, such as software components, object-oriented software components, class components, and task components, as well as processors, functions, attributes, procedures, subroutines, segments of program codes, drivers, firmware, micro-codes, circuits, data, databases, data structures, tables, arrays, and variables. In addition, functions provided in components and a “unit” may be combined into a smaller number of components and “units” or further subdivided into additional components and “units.”

The expression “based on” used herein is used to describe one or more factors that influences a decision, an action of judgment or an operation described in a phrase or sentence including the relevant expression, and this expression does not exclude additional factors influencing the decision, the action of judgment or the operation.

When a certain component is described as “coupled to” or “connected to” another component, this should be understood as having meaning that the certain component may be coupled or connected directly to the other component or that the certain component may be coupled or connected to the other component via a new intervening component.

In the present disclosure, artificial intelligence (AI) refers to a technology that imitates human learning ability, reasoning ability, and perception ability and implements them with a computer, and may include the concepts of machine learning and symbolic logic. Here, the machine learning (ML) may be an algorithm technology that classifies or learns features of input data by itself. Specifically, artificial intelligence technology is a machine learning algorithm that can analyze input data, learn the result of the analysis, and make judgments or predictions based on the result of the learning. In addition, technologies that use the machine learning algorithm to imitate the cognitive and judgmental functions of the human brain can also be understood as a category of artificial intelligence. For example, technical fields of linguistic understanding, visual understanding, inference/prediction, knowledge expression, and motion control may be included in the category of artificial intelligence.

The term “machine learning” used herein may refer to a process of training a neural network model using experience of processing data. Through the machine learning, computer software may mean improving its own data processing capabilities. Here, the neural network model is constructed by modeling the correlation between data, and the correlation may be expressed by a plurality of parameters. The neural network model may derive the correlation between data by extracting and analyzing features from given data, and optimizing the parameters of the neural network model by repeating this process may be referred to as machine learning. For example, the neural network model may learn mapping (correlation) between an input and an output with respect to data given as an input/output pair. Alternatively, even when only input data are given, the neural network model may learn the relationship by deriving the regularity between given data.

The terms “artificial intelligence learning model,” “machine learning model,” or “neural network model” used herein may be designed to implement a human brain structure on a computer, and may include a plurality of network nodes that simulate neurons of a human neural network and have weights. Here, the plurality of network nodes may have a connection relationship between them by simulating synaptic activities of neurons that exchange signals through synapses. Specifically, in the artificial intelligence learning model, a plurality of network nodes may exchange data according to a convolution connection relationship while being located in layers of different depths. The artificial intelligence learning model may be, for example, an artificial neural network model, a convolution neural network model, or the like, but the scope of the present disclosure is not limited to the above-described examples, and it should be noted that various known neural network models are applicable to the present disclosure.

Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. In the accompanying drawings, like or relevant components are indicated by like reference numerals. In addition, in the following description of embodiments, repeated descriptions of the identical or relevant components will be omitted. However, even if a description of a component is omitted, such a component is not intended to be excluded in an embodiment.

1 FIG. 1 FIG. 200 200 20 200 is an exemplary view illustrating a scanning environment according to some embodiments of the present disclosure. Specifically, the scanning environment illustrated inis an environment in which an image of an oral cavity of a patient is obtained using an intraoral scanneraccording to some embodiments of the present disclosure, wherein the intraoral scannermay be a dental instrument for obtaining an intraoral image of a subject(e.g., a patient) of the intraoral scanner.

1 FIG. 10 20 20 200 10 20 20 20 20 20 20 As illustrated in, a user(e.g., a dentist or a dental hygienist) may obtain an image of an oral cavity of the subjectfrom the subjectby using the intraoral scanner. In another example, the usermay also obtain an image of an oral cavity of the subjectfrom a diagnosis model (e.g., a plaster model or an impression model) obtained by modeling after the shape of the oral cavity of the subject. Hereinafter, for convenience of description, a description is made based on an example of obtaining the oral cavity of the image of the subject, but the scope of the present disclosure is not limited to the example. In addition, a part subject to image acquisition is not limited to the oral cavity of the subject, and it is also possible to obtain images of other parts of the subject(e.g., the ear of the subject).

1 FIG. Hereinafter, an operation of elements illustrated inis described in detail.

100 20 200 100 200 100 10 20 100 10 20 100 1 FIG. 4 FIG. An electronic deviceillustrated inmay receive a two-dimensional image of the oral cavity of the subjectfrom the intraoral scanner. Here, the electronic devicemay communicate with the intraoral scannervia a wired or wireless communication network. In addition, the electronic devicemay generate a three-dimensional image of the oral cavity by performing three-dimensional modeling of an internal structure of the oral cavity, based on the received two-dimensional image of the oral cavity, and a detailed operation related thereto is described below with reference to. The generated three-dimensional image of the oral cavity may be displayed to the useror the subjectthrough a display of the electronic device. In this case, the usermay provide a proper dental treatment service to the subjectwith reference to the three-dimensional image displayed on the display of the electronic device.

100 200 200 20 In some embodiments, the electronic devicemay receive the three-dimensional image of the oral cavity generated by the intraoral scanner. Here, the intraoral scannermay obtain a two-dimensional image of the oral cavity by scanning the oral cavity of the subject, and may generate a three-dimensional image of the oral cavity based on the obtained two-dimensional image of the oral cavity. That is, it should be noted that regardless of the position where the operation of generating the three-dimensional image is processed, the operation is included in the scope of the present disclosure.

100 100 20 20 100 In addition, the electronic devicemay be communicatively connected to a cloud server (not shown). In this case, the electronic devicemay transmit the two-dimensional image of the oral cavity or the three-dimensional image of the oral cavity of the subjectto the cloud server, and the cloud server may store the two-dimensional image of the oral cavity of the subject, received from the electronic device, or the three-dimensional image of the oral cavity.

100 2 FIG. The above-described electronic devicemay be implemented as a computing device, and such a computing device is described below in detail with reference to.

200 10 1 FIG. The intraoral scannerillustrated inmay have a shape capable of being inserted into and drawn out of an oral cavity, and may be a handheld scanner which enables a userto freely adjust a scanning distance and a scanning angle.

200 200 20 20 Such an intraoral scannermay obtain the image of the oral cavity by being inserted into the oral cavity and scanning the oral cavity in a non-contact manner. The image of the oral cavity may include at least one tooth, the gingiva, and an artificial structure which can be inserted into the oral cavity (e.g., an orthodontic device including a bracket and a wire, an implant, a denture, an orthodontic aid inserted into the oral cavity, etc.). Specifically, the intraoral scannermay emit light to the oral cavity of the subjectby using a light source (or a projector), and may receive light reflected from the oral cavity of the subjectthrough a camera (or at least one sensor).

200 20 20 In addition, the intraoral scannermay obtain, as the two-dimensional image, a surface image of the oral cavity of the subject, based on information received through the camera. Here, the surface image of the oral cavity of the subjectmay include at least one of a tooth, the gingiva, an artificial structure, a cheek, the tongue, or a lip.

200 As illustrated above, in some embodiments, the intraoral scannermay obtain the two-dimensional image of the oral cavity by scanning the oral cavity, and may generate the three-dimensional image of the oral cavity based on the obtained two-dimensional image of the oral cavity.

200 2 FIG. The above-described intraoral scannermay be implemented as a computing device, and an example of such a computing device is described below in detail with reference to.

2 FIG. 1 FIG. 2 FIG. 2 FIG. 100 200 is an exemplary block diagram illustrating each of the electronic deviceand the intraoral scannerillustrated in. The block diagram illustrated inillustrates an exemplary embodiment for achieving the objective of the present disclosure, and some elements may be added or deleted as necessary. In addition, elements of the block diagram illustrated inare elements that are functionally distinguished, and it should be noted that multiple different elements may be implemented to be integrated in an actual physical environment. Hereinafter, each element illustrated in the block diagram is described in detail.

100 101 103 105 107 109 100 100 100 2 FIG. The electronic deviceillustrated inmay include one or more processors, one or more memories, a communication circuit, a display, and/or an input device. As illustrated above, at least one of the elements included in the electronic devicemay be omitted, or other elements may be added to the electronic device. In addition, additionally or alternatively, some of the elements may be implemented to be integrated, or may be implemented as a single entity or multiple entities. At least some of the elements in the electronic devicemay be connected to each other through a bus, a general purpose input/output (GPIO), a serial peripheral interface (SPI), a mobile industry processor interface (MIPI), etc. and transmit and receive data and/or a signal.

101 100 103 100 101 100 101 100 103 103 The one or more processorsof the electronic devicemay be elements capable of performing data processing or operation for control and/or communication of each element (e.g., the memory)) of the electronic device. The one or more processorsmay be operatively connected to, for example, the elements of the electronic device. In addition, the one or more processorsmay load data or a command received from other elements of the electronic devicein the one or more memories, process the data or command stored in the one or more memories, and store result data.

103 100 103 101 103 103 200 Next, the one or more memoriesof the electronic devicemay store various data, commands, and/or information. As a specific example, the one or more memoriesmay store instructions for operation of the processoras a computer program. In addition, the one or more memoriesmay store correlation models constructed according to a machine learning algorithm. In addition, the one or more memoriesmay store data received from the intraoral scanner(e.g., the two-dimensional image of the oral cavity and the three-dimensional image of the oral cavity).

105 100 200 105 105 105 105 105 100 Next, the communication circuit (the network interface) of the electronic devicemay establish a wired or wireless communication channel with an external device (e.g., the intraoral scannerand the cloud server (not shown)), and transmit and receive various data to or from the external device. In some embodiments, to communicate with the external device by wire, the communication circuitmay include at least one port for connection with the external device via a wired cable. In this case, the communication circuitmay perform communication with the external device connected to the wired cable via the at least one port. In some other embodiments, the communication circuitmay include a cellular communication module and may be configured to be connected to a cellular network (e.g., 3G, LTE, 5G, WiBro, or WiMAX). In some other embodiments, the communication circuitmay include a short-distance communication module, and may transmit or receive data to or from the external device by using short-distance communication (e.g., Wi-Fi, Bluetooth, Bluetooth low energy (BLE), or UWB). In some other embodiments, the communication circuitmay include a non-contact communication module for non-contact communication. Here, the non-contact communication may include, for example, at least one non-contact type proximity communication technology such as near field communication (NFC), radio frequency identification (RFID) communication, or magnetic secure transmission (MST) communication. In addition to the above-described various examples, the electronic devicemay be implemented in various known methods for communication with the external device, and it should be noted that the above-described examples do not limit the scope of the present disclosure.

107 100 101 101 20 200 20 107 107 100 10 Next, the displayof the electronic devicemay display various screens based on control of the processor. Here, based on control of the processor, the two-dimensional image of the oral cavity of the subject, received from the intraoral scanner, and/or the three-dimensional image of the oral cavity of the subject, obtained by performing three-dimensional modeling of the inner structure of the oral cavity, may be displayed through the display. In this case, to display the two-dimensional image and/or the three-dimensional image of the oral cavity through the display, for example, a web browser or a dedicated application may be installed in the electronic device. In some embodiments, the above-described web browser or dedicated application may be implemented to provide the userwith an edition function, a storage function, and a deletion function of the two-dimensional image and/or the three-dimensional image of the oral cavity through a user interface.

109 100 101 100 100 109 109 107 109 Next, the input deviceof the electronic devicemay receive a command or data to be used for an element (e.g., the processor) of the electronic devicefrom outside (e.g., a user) of the electronic device. The input devicemay include, for example, a microphone, a mouse, a keyboard, or the like. In some embodiments, the input devicemay be coupled to the displayand implemented in the form of a touch sensor panel capable of recognizing contact or proximity of various external objects. However, the scope of the present disclosure is not limited to the above-described examples, and various known input devicesmay be included in the scope of the present disclosure for convenience of the user.

200 201 202 203 204 205 206 200 200 200 2 FIG. The intraoral scannerillustrated inmay include a processor, a memory, a communication circuit, a light source, a camera, and/or an input device. As described above, at least one of the elements included in the intraoral scannermay be omitted, or other elements may be added to the intraoral scanner. In addition, additionally or alternatively, some of the elements may be implemented to be integrated, or may be implemented as a single entity or multiple entities. At least some of the elements in the intraoral scannermay be connected to each other through a bus, a general purpose input/output (GPIO), a serial peripheral interface (SPI), a mobile industry processor interface (MIPI), etc. and transmit and receive data and/or a signal.

201 200 200 200 201 200 202 202 The processorof the intraoral scannermay be an element capable of performing data processing or operation for control and/or communication of each element of the intraoral scanner, and may be operatively connected to the elements of the intraoral scanner. In addition, the processormay load data or a command received from other elements of the intraoral scannerin the memory, process the data or command stored in the memory, and store result data.

202 200 201 Next, the memoryof the intraoral scannermay store instructions for the above-described operation of the processor.

203 200 100 203 203 203 203 203 200 Next, the communication circuitof the intraoral scannermay establish a wired or wireless communication channel with an external device (e.g., the electronic device), and transmit and receive various data to or from the external device. In some embodiments, to communicate with the external device by wire, the communication circuitmay include at least one port for connection with the external device via a wired cable. In this case, the communication circuitmay perform communication with the external device connected by the wired cable via the at least one port. In some other embodiments, the communication circuitmay include a cellular communication module and may be configured to be connected to a cellular network (e.g., 3G, LTE, 5G, WiBro, or WiMAX). In some other embodiments, the communication circuitmay include a short-distance communication module, and may transmit or receive data to or from the external device by using short-distance communication (e.g., Wi-Fi, Bluetooth, Bluetooth low energy (BLE), or UWB). In some other embodiments, the communication circuitmay include a non-contact communication module for non-contact communication. Here, the non-contact communication may include, for example, at least one non-contact type proximity communication technology such as near field communication (NFC), radio frequency identification (RFID) communication, or magnetic secure transmission (MST) communication. In addition to the above-described various examples, the intraoral scannermay be implemented in various known methods for communication with the external device, and it should be noted that the above-described examples do not limit the scope of the present disclosure.

204 200 20 204 Next, the light sourceof the intraoral scannermay emit light to the oral cavity of the subject. For example, the light emitted from the light sourcemay be structured light having a predetermined pattern (e.g., a stripe pattern in which straight lines of different colors continuously appear). Here, the pattern of the structured light may be generated using, for example, a pattern mask or a digital micro-mirror device (DMD), but is not limited thereto.

205 200 20 20 205 205 Next, the cameraof the intraoral scannermay obtain an image of the oral cavity of the subjectby receiving reflected light reflected by the oral cavity of the subject. Here, the cameramay include a left camera corresponding to a left eye field and a right camera corresponding to a right eye field to construct a three-dimensional image according to an optical triangulation method. In addition, here, the cameramay include at least one sensor such as a CCD sensor or a CMOS sensor.

206 200 200 206 10 10 10 206 Next, the input deviceof the intraoral scannermay receive a user input for controlling the intraoral scanner. For example, the input devicemay include a button for receiving a push operation of the user, a touch panel for detecting a touch of the user, and a voice recognition device including a microphone. In this case, the usermay control scanning start or stop by using the input device.

200 206 200 206 200 206 100 201 200 101 100 10 20 200 200 20 20 100 100 20 107 100 20 20 107 100 107 To describe in more detail the operation of the intraoral scannercontrolled through the input device, the intraoral scannermay receive a user input for starting scanning through the input deviceof the intraoral scanneror the input deviceof the electronic device, and start scanning according to processing by the processorof the intraoral scanneror the processorof the electronic device. Here, when the userscans the oral cavity of the subjectthrough the intraoral scanner, the intraoral scannermay generate a two-dimensional image of the oral cavity of the subject, and may transmit the two-dimensional image of the oral cavity of the subjectto the electronic devicein real time. In this case, the electronic devicemay display the received two-dimensional image of the oral cavity of the subjectthrough the display. In addition, the electronic devicemay generate (construct) a three-dimensional image of the oral cavity of the subjectbased on the two-dimensional image of the oral cavity of the subject, and may display the three-dimensional image of the oral cavity through the display. In this case, the electronic devicemay also display the three-dimensional image that is being generated in real time through the display.

207 200 200 207 10 207 10 200 207 200 201 Next, the sensor moduleof the intraoral scannermay detect an operational state of the intraoral scanneror an external environmental state (e.g., the user's operation), and generate an electrical signal corresponding to the detected state. The sensor modulemay include, for example, at least one of a gyro sensor, an acceleration sensor, a gesture sensor, a proximity sensor, or an infrared sensor. Here, the usermay control scanning start or stop by using the sensor module. In a specific example, in a case where the userholds the intraoral scannerwith a hand and moves the same, when an angular speed measured through the sensor moduleexceeds a configuration value, the intraoral scannermay control the processorto start the scanning operation.

3 FIG. 3 FIG. 2 FIG. 200 200 Hereinafter, referring to, the above-described intraoral scanneris described in more detail.is an exemplary view illustrating the intraoral scannerdescribed with reference to.

200 210 220 210 200 10 220 20 210 220 200 210 210 204 20 20 20 10 206 200 10 206 204 20 3 FIG. The intraoral scannerillustrated inmay include a bodyand a probe tip. Here, the bodyof the intraoral scannermay have a shape that is easy for the userto grip and use, and the probe tipmay have a shape that facilitates insertion into and withdrawal from the oral cavity of the subject. In addition, the bodymay be coupled to or detached from the probe tip. In addition, the above-described elements of the intraoral scannermay be arranged inside the body. An opening may be disposed at one end of the bodyso that light output from the light sourcecan be emitted to the subject. The light emitted through the opening may be reflected by the subjectand introduced again through the opening. Here, the reflected light introduced through the opening may be captured by a camera and an image of the subjectmay be generated. In addition, the usermay start scanning by using the input device(e.g., a button) of the intraoral scanner. For example, when the usertouches or presses the input device, light may be emitted from the light sourceto the subject.

1 3 FIGS.to 4 FIG. 20 The scanning environment and the elements included therein according to some embodiments of the present disclosure are described in detail with referenceabove. Before describing an image processing method according to some embodiments of the present disclosure, an operation of generating a three-dimensional image of the oral cavity of the subject, which can be referred to in some embodiments of the present disclosure, is described below in detail with reference to.

4 FIG. 20 10 20 200 200 310 20 200 20 20 200 310 100 is an exemplary view illustrating in detail an operation of generating a three-dimensional image of the oral cavity of the subject, which can be referred to in some embodiments of the present disclosure. As described above, the usermay scan the oral cavity of the subjectwhile moving the intraoral scanner, and in this case, the intraoral scannermay obtain multiple two-dimensional imagesof the oral cavity of the subject. For example, the intraoral scannermay obtain a two-dimensional image of a region including a front tooth of the subjectand a two-dimensional image of a region including a molar tooth of the subject. In this case, the intraoral scannermay transmit the obtained multiple two-dimensional imagesto the electronic device.

100 310 20 310 100 310 The electronic devicemay convert each of the multiple two-dimensional imagesof the oral cavity of the subjectinto a set of multiple points having three-dimensional coordinate values by using the received multiple two-dimensional images. For example, the electronic devicemay convert each of the multiple two-dimensional imagesinto a point cloud corresponding to a set of data points having three-dimensional coordinate values.

310 20 100 Here, the point cloud set having three-dimensional coordinate values based on the multiple two-dimensional imagesmay be stored as raw data of the oral cavity of the subject. In addition, the electronic devicemay complete the entire tooth model by aligning the point cloud, which is a set of data points having three-dimensional coordinate values.

100 100 320 20 In some embodiments, the electronic devicemay reconfigure (reconstruct) the three-dimensional image of the oral cavity. For example, by merging the point cloud set stored as raw data by using a Poisson algorithm, the electronic devicemay reconfigure multiple points and convert the multiple points into a closed three-dimensional surface to reconfigure a three-dimensional imageof the oral cavity of the subject. However, unlike the present example, the raw data may be processed according to various known methods, and thus it should be noted that any methods for reconfiguring the three-dimensional image of the oral cavity can be included in the scope of the present disclosure.

4 FIG. The operation of generating a three-dimensional image, which can be referred to in some embodiments of the present disclosure, is additionally described with reference to. Hereinafter, methods according to various embodiments of the present disclosure are described in detail.

100 200 1 FIG. 1 FIG. Respective operations of the methods to be described below may be performed by a computing device. In other words, the respective operations of the methods may be implemented by one or more instructions executed by a processor of a computing device. All operations included in such methods may be executed by one physical computing device, but first operations of the method may be performed by a first computing device and second operations of the method may be performed by a second computing device. Hereinafter, the description is made under the assumption that the respective operations of the methods are performed by the electronic deviceillustrated in. However, for convenience of description, operation structures of the respective operations included in the methods may be omitted. In addition, it should be noted that performing the respective operations of the methods by the intraoral scannerillustrated inis not excluded from the scope of the present disclosure.

5 FIG. is an exemplary flow chart illustrating an image processing method according to some embodiments of the present disclosure.

5 FIG. 1 FIG. 110 10 20 Referring to, in operation S, a tooth region may be identified in a two-dimensional image of a target oral cavity. Here, as described with reference to, the target oral cavity may mean an oral cavity of the subject, but the meaning of the target oral cavity is not limited to the present example, and may mean a diagnosis model obtained by modeling after the shape of the oral cavity of the subject. In addition, here, the tooth region corresponds to an image region included in the two-dimensional image of the target oral cavity, and may mean a region in which at least one tooth is represented.

110 10 200 10 100 In relation to operation S, the identification of the tooth region may be performed according to various methods. For example, the tooth region may be identified by directly selecting, by the user, the tooth region of the two-dimensional image obtained from the intraoral scanner. In this case, the usermay select the tooth region from the two-dimensional image through a user interface provided in the electronic device. In another example, through image processing of detecting a unique attribute of the tooth region represented in the two-dimensional image, the tooth region may be identified. Specifically, the tooth region in the two-dimensional image of the oral cavity is represented in a white-based color unlike other regions, and the tooth region may be identified through image processing of detecting a color as a unique attribute. The tooth region may be identified by other various methods, and it should be noted that any operations of identifying the tooth region from the two-dimensional image can be included in the scope of the present disclosure. Hereinafter, an operation of identifying the tooth region from the two-dimensional image by using a neural network model constructed according to a machine learning algorithm is described in more detail.

110 6 FIG. In relation to operation S, in some embodiments, the identifying the tooth region may include identifying a tooth region in the two-dimensional image by using a tooth segmentation model constructed according to a machine learning algorithm. Here, the tooth segmentation model may be a model trained by modeling a correlation between a learning image set for a tooth and a segmentation result image set corresponding to the training image set. A detailed description of the tooth segmentation model is described below with reference to.

120 10 FIG. In operation S, a first neighboring region located within a predetermined distance from a boundary of the tooth region may be identified in the two-dimensional image of the target oral cavity. Here, the predetermined distance may mean, for example, one pixel on the two-dimensional image, but the scope of the present disclosure is not limited to the present example, and the distance may change according to the implementation example thereof. That is, as described above, the tooth region is a region which can be a reference for image processing, and thus in this operation, the first neighboring region may be identified with reference to the boundary of the tooth region. A detailed description related thereto is made below with reference to.

130 200 204 200 10 10 1 FIG. 11 FIG. In operation S, whether to include the first neighboring region in a region of interest may be determined based on a difference in depth between the tooth region and the first neighboring region. Here, the depth is a value represented on an axis corresponding to a scanning direction of the intraoral scannerillustrated in, and may mean, for example, a distance from a reference point to each region, and in this case, the reference point may be a light sourceof the intraoral scanner. However, the scope of the present disclosure is not limited to the present example, and it should be noted that if a value of each region can correspond to the axis corresponding to the scanning direction, any methods can be included in the scope of the present disclosure. In addition, here, the region of interest means a region which can be a target of interest of the user, and for example, a tooth region and a gingiva region required for dental treatment may be a region of interest. The region of interest may be determined according to the operations to be described with reference tobelow so that a region satisfying the intention of the usermay be determined, and the target of the region of interest may be adjusted by changing a configuration value (e.g., a threshold, a distance between the boundary of the region of interest and a third neighboring region, a reference count, etc.).

140 4 FIG. 4 FIG. In operation S, a three-dimensional image of the target oral cavity may be generated from the two-dimensional image including the region of interest. Here, the generation of the three-dimensional image may refer to the description of. However, the description ofis made according to an example of converting the entire region of the two-dimensional image into a point cloud, but unlike this, in this operation, it can be understood that the region of interest, which is a partial region included in the two-dimensional image, is converted into the point cloud.

6 FIG. 5 FIG. 6 FIG. 500 410 420 410 is an exemplary view illustrating in detail an operation of identifying a tooth region, described with reference to. Specifically,illustrates an example of constructing a tooth segmentation modelby using a training image setfor a tooth and a segmentation result image setcorresponding to the training image set.

410 410 410 410 Here, the training image setmay mean multiple two-dimensional images of the oral cavity. In this case, the training image setmay mean multiple two-dimensional images randomly extracted from a two-dimensional image pool of the oral cavity of various subject groups (e.g., a male group, a female group, a group by generation, etc.). It can be understood that when a training image is extracted by limiting a two-dimensional image pool of a specific group, a training image setcustomized for the group can be provided, and when a training image is extracted from two-dimensional image pools of various groups, a generalized training image setcan be provided.

420 410 10 In addition, the segmentation result image setis an image set corresponding to the training image set, and may mean multiple two-dimensional images in which at least one region to be identified is masked (or segmented). In a specific example, the segmentation result image may include a tooth region mask and a gingiva region mask. However, the scope of the present disclosure is not limited thereto, the segmentation result image may further include a soft tissue region (e.g., a check region, a tongue region, etc.) mask, and for any region to be identified by the user, the segmentation result image can be provided to include a mask corresponding to the region. Here, the mask may be understood as an identifier enabling a specific region represented in the two-dimensional image to be distinguished from other regions, and may be included in the segmentation result image in the form of an image or metadata.

420 410 109 100 10 The segmentation result image setmay be generated from the training image setaccording to various methods. For example, by overlaying masking on the training image to correspond to a user input received through the input deviceof the electronic device, the segmentation result image may be generated. In addition, the segmentation result image may be generated from the training image in an automated method, and any methods of masking a specific region intended by the usercan be included in the scope of the present disclosure.

500 410 420 100 500 100 1 FIG. The tooth segmentation modelcan be trained using the above-described training image setas input data and using the segmentation result image setas output data. The machine learning may be performed by the electronic deviceillustrated in, but a case where the machine learning is performed by an external device and the trained tooth segmentation modelis loaded to the electronic deviceis not excluded from the scope of the present disclosure.

500 500 500 410 420 Here, the tooth segmentation modelmay be trained to extract various features which can be extracted from the training image, such as the texture, density, and color of the region included in the training image, the shape of the tooth, the shape of the gingiva, and the shape of the oral cavity, and derive a correlation between the segmentation result image and the training image based on the extracted features. As a machine learning algorithm which can be used for training the tooth segmentation model, for example, a deep neural network algorithm, a recurrent neural network algorithm, a convolutional neural network algorithm, a classification-regression analysis algorithm, reinforcement learning algorithm, or the like can be referred to, and it should be noted that all known artificial intelligence technologies for constructing the tooth segmentation modelby using the above-described training data having a pair of an input and an output (i.e., the training image setand the segmentation result image set) are applicable to the present disclosure.

6 FIG. 500 410 420 410 500 As described above, referring back to, it can be understood that the tooth segmentation modelcan be trained by using both the training image setand the segmentation result imagehaving the masked tooth region and gingiva region, as an image corresponding to the training image set. That is, it can be understood that the trained tooth segmentation modelis a model for identifying a tooth region and a gingiva region in a two-dimensional image of the oral cavity.

6 FIG. 10 Referring to, the operation of identifying a tooth region based on an artificial intelligence model has been described above in detail. According to the above-described operation, a region intended by the user may be identified in the two-dimensional image of the oral cavity in an automated method of minimizing the involvement of the user. In addition, a first region (e.g., the tooth region) and a second region (e.g., the gingiva region) may be identified in the two-dimensional image of the oral cavity. That is, by filtering out other regions (e.g., a noise region) of the two-dimensional image excluding the identified regions, and generating a three-dimensional image based on the image obtained by performing the filtering, a three-dimensional image obtained by removing the noise region may be generated. For example, the three-dimensional image may be generated to include only the tooth region and the gingiva region.

500 10 7 8 FIGS.and However, in an actual implementation case using the tooth segmentation modeltrained to identify only the tooth region and the gingiva region, a soft tissue region, which can be a type of a noise region, is represented to have a similar attribute (e.g., color, texture, or the like) to that of the gingiva region on the two-dimensional image, thus the soft tissue region may be identified in the two-dimensional image differently from the intention of the user, and accordingly, the soft tissue region could have been included in the three-dimensional image. A detailed description related thereto is made below with reference to.

7 FIG. 6 FIG. 8 FIG. 7 FIG. 500 500 is an exemplary view illustrating an actual output image of the tooth segmentation modeldescribed with reference to, andis an exemplary view illustrating an ideal output image of the tooth segmentation modeldescribed with reference to.

7 FIG. 8 FIG. 610 500 620 610 610 610 610 610 620 620 620 620 630 630 630 630 a, b, c, a, b, c. a, b, c. illustrates an input imageof the tooth segmentation result modeltrained to identify only the tooth region and the gingiva region and an actual output imagecorresponding to the input image. Here, the input imageincludes a tooth regiona gingiva regionand a soft tissue regionand similarly, the actual output imageincludes a tooth regiona gingiva regionand a soft tissue regionIn addition, the ideal output imageillustrated inalso includes a tooth regiona gingiva regionand a tissue region

7 FIG. 8 FIG. 9 10 FIGS.and 620 630 620 630 620 630 620 630 620 620 c c, c c, b b c Comparingand, the actual output imageand the ideal output imagediffer from each other in their soft tissue regionsandand as described above, it can be construed as a result that the soft tissue regionsandwhich can be a type of a noise region, are represented in attributes similar to those of the gingiva regionsandon the two-dimensional image. Due to the soft tissue regionof the actual output image, a region not intended by the user is identified in the two-dimensional image, and accordingly, a region not intended by the user may also be included in the three-dimensional image. As operations for compensating such a limitation, detailed operations of processing the two-dimensional image with reference to the tooth region in which the accuracy of the identification can be secured due to the distinctiveness of the attribute presented in the two-dimensional image are described with reference to.

9 FIG. 10 FIG. 5 FIG. 9 FIG. is an exemplary view illustrating a two-dimensional image of a target oral cavity, which can be referred to in some embodiments of the present disclosure, andis an exemplary view illustrating in detail the operation of identifying a first neighboring region, described with reference to, as the operation performed on the two-dimensional image of the target oral cavity, described with reference to.

610 610 610 610 640 640 610 643 643 641 640 643 643 643 643 641 640 643 643 641 640 10 10 10 9 FIG. 10 FIG. 10 FIG. 10 FIG. a, b, c, a a b a a b a b a a b a The input imageillustrated inincludes a tooth regiona gingiva regionand a soft tissue regionand an output imagein which a tooth regionis identified to correspond to the input imageis illustrated in. Referring to, it can be understood that first neighboring regionsandlocated within a predetermined distance can be identified with reference to a boundaryof the identified tooth region. Here,illustrates an example in which two first neighboring regionsandare identified, and is merely provided for convenience of description, and the number of the first neighboring regionsandwhich can be identified with reference to the boundaryof the tooth regionmay vary. That is, all of the first neighboring regionsandlocated within a predetermined distance with reference to the boundaryof the tooth regioncan be identified, and only some of them may be identified. The selection of all or some of the first neighboring regions by the useris a selection by the userin a trade-off relationship between a consumption of computing resources and the accuracy of image processing to be performed below, and the usermay select either one according to an actual implementation case.

11 FIG. 5 FIG. 11 FIG. 5 FIG. 130 Hereinafter, referring to, a more detailed explanation of operation Sofis provided below.is an exemplary flow chart specifically illustrating the operation of determining a region of interest described with reference to.

11 FIG. 4 FIG. 131 200 100 200 200 Referring to, in operation S, a difference in depth may be calculated by comparing a first coordinate corresponding to a tooth region and a second coordinate corresponding to a first neighboring region. Here, each of the first coordinate and the second coordinate may be a coordinate obtained by using the intraoral scannerlinked to the electronic device. As described with reference to, when a three-dimensional coordinate is obtained using the intraoral scanner, a difference in depth between the respective regions may be calculated using a value (e.g., the first coordinate and the second coordinate) represented on an axis corresponding to a scanning direction of the intraoral scanneramong the obtained three-dimensional coordinates.

200 200 The coordinates corresponding to the respective regions described above may be obtained in various methods. In some embodiments, the coordinates corresponding to the respective regions may be obtained based on a position of a first camera provided in the intraoral scanner, a position of a second camera distinguished from the first camera, and respective images captured by the cameras. Specifically, by comparing and analyzing respective images captured by a left camera (e.g., the first camera) corresponding to a left eye field and a right camera (e.g., the second camera) corresponding to a right eye field according to an optical triangulation method with the positions of the cameras, the coordinates corresponding to the respective regions may be obtained. However, regarding a detailed operation using the optical triangulation method, a detailed description is omitted so as not to blur the point of the present disclosure. In some other embodiments, the coordinates corresponding to the respective regions may be obtained through monocular depth estimation for the two-dimensional image captured from the intraoral scanner. Here, the monocular depth estimation is a three-dimensional depth estimation method using a two-dimensional image captured by a single camera. In a specific example, a three-dimensional image is restored from a two-dimensional image by using a DenseDepth model, the depth of a region on the two-dimensional image can be estimated accordingly, and in addition to the examples mentioned above, all known methods of estimating a depth of a two-dimensional image through a two-dimensional image captured by a single camera may be applied to the present disclosure.

132 133 132 134 10 Next, when the difference in depth is equal to or greater than a threshold (S), in operation S, the first neighboring region may be excluded from the region of interest, and when the difference in depth is less than the threshold (S), in operation, the first neighboring region may be included in the region of interest. Here, the threshold is a type of a configuration value which can be a reference for exclusion from and inclusion in the region of interest, and the usermay change the threshold according to an actual implementation case. When the gingiva region is included in the region of interest and the soft tissue region is excluded from the region of interest, a proper threshold which can distinguish two regions may be obtained through experiment.

12 12 FIGS.A toD 11 FIG. 12 12 FIGS.A toD 11 FIG. Hereinafter, referring to, a detailed description of the operation of determining a region of interest described with reference tois made.are exemplary views illustrating in detail the operation of determining a region of interest, described with reference to.

12 FIG.A 10 FIG. 11 FIG. 12 FIG.A 12 FIG.A 11 FIG. 12 FIG.A 700 20 710 720 730 740 200 720 750 730 750 740 731 741 750 750 720 720 731 741 720 731 741 731 741 731 741 731 730 720 800 741 74 720 800 741 740 731 730 800 a b a b is an example of a virtual sectional imageof the subject, and illustrates a scanning direction, a tooth region, a gingiva region, and a soft tissue regionof the intraoral scanner. Here, it can be understood that the tooth regionis distinguished from a first boundaryfrom the gingiva regionand a second boundaryfrom the soft tissue region. As described above with reference to, first neighboring regionsandmay be identified from the boundariesandof the tooth region, respectively. In addition, as described above with reference to, a difference in depth between a first coordinate corresponding to the tooth regionand a second coordinate corresponding to the first neighboring regionsandmay be calculated. Here, the first coordinate corresponding to the tooth regionmay be a coordinate of a minute region included in a tooth region within the closest distance from the first neighboring regionsand, and referring to, it can be understood that the depths between the minute regions in the closest distance from the respective first neighboring regionsandand the first neighboring regionsandare compared. According to the example illustrated in, a difference in depth between the first neighboring regionlocated in the gingiva regionand the tooth regionis less than a threshold, and a difference in depth between the first neighboring regionlocated in the soft tissue regionand the tooth regionis equal to or greater than the threshold. Accordingly, the first neighboring regionlocated in the soft tissue regionis excluded from the region of interest, and the first neighboring regionlocated in the gingiva regionis included in the region of interest. As described above with reference to, by configuring a proper threshold, the operation of determining the region of interest may be performed as in the example illustrated in.

The operation of determining the region of interest may be performed as the region of interest gradually expands according to a predetermined rule. Hereinafter, a more detailed description of the expansion of the region of interest will be provided.

133 120 11 FIG. 5 FIG. 11 FIG. With regard to operation Sillustrated in, in some embodiments, the operation of excluding the first neighboring region from the region of interest when the difference in depth from the tooth region is equal to or greater than a threshold may include an operation of excluding a second neighboring region from the region of interest even though the difference in depth between the first neighboring region and the second neighboring region located within a predetermined distance from the boundary of the first neighboring region is less than the threshold. Here, with regard to the operation of identifying the second neighboring region, the description of operation Sinmay be referred to, and for the operation of calculating the difference in depth, the description ofmay be referred to.

12 FIG.B 741 740 743 800 740 741 740 Referring tofor describing the present embodiment in more detail, it can be understood that even though a difference in depth between a first neighboring regionlocated in a soft tissue regionand a second neighboring regionlocated in the soft tissue region is less than a threshold, a second neighboring region located in the soft tissue regionis excluded from the region of interest. That is, the rule described here is a rule disallowing incorporation of a region expanded from the boundary of the region excluded once (e.g., the first neighboring regionlocated in the soft tissue region) into the region of interest, and through such a rule, computing resources unnecessarily consumed for operation for the region which cannot be included in the region of interest can be saved.

134 120 11 FIG. 5 FIG. 11 FIG. With regard to operation Sillustrated in, in some embodiments, including the first neighboring region in the region of interest when the difference in depth from the tooth region is less than the threshold may include repeatedly expanding the region of interest until the difference in depth between the region of interest and a third neighboring region located within a predetermined distance from the boundary of the region of interest becomes equal to or greater than the threshold. Here, for the operation of identifying the third neighboring region, the description of operation Sinmay be referred to, and for the operation of calculating the difference in depth, the description ofmay be referred to.

12 FIG.C 12 FIG.A 12 FIG.C 12 FIG.D 731 730 731 730 733 730 731 730 733 730 731 730 733 730 800 733 730 731 733 735 737 10 Referring tofor describing the present embodiment in more detail, a first neighboring regionlocated in a gingiva regionis a region included in the region of interest, as described with reference to. Accordingly, the first neighboring regionlocated in the gingiva regionmay be understood as a minute region corresponding to the region of interest, and whether to include a third neighboring regionlocated in the gingiva regionin the region of interest may be determined through comparison of a difference in depth between the first neighboring regionlocated in the gingiva regionand the third neighboring regionlocated in the gingiva region. According to the example illustrated in, since the difference in depth between the region of interest (e.g., the first neighboring regionlocated in the gingiva region) and the third neighboring regionlocated in the gingiva regionis less than a threshold, it may be understood that the third neighboring regionlocated in the gingiva regionis included in the region of interest. Similarly, according to the rule, the region of interest can be repeatedly expanded, and referring to, it can be understood that minute regions,,, andof the gingiva region are repeatedly included in the region of interest. That is, the rule described herein is a rule of repeatedly expanding the region of interest by repeatedly applying the same rule corresponding to the comparison of depth between the expanded region of interest and the neighboring region thereof, and through such a rule, the region of interest can be determined to satisfy the intention of the user.

To describe the above-described operation of expanding the region of interest with a specific example, gingiva regions consecutively connected from a tooth region may be incorporated into a region of interest, and a noise region (e.g., a soft tissue region, etc.) having a distance from the tooth region may be excluded from the region of interest. In the above-described operation of expanding the region of interest, the distal end of the region of interest is determined by a single rule of calculating a difference in depth between the region to be expanded (e.g., the third neighboring region) and the region of interest, but hereinafter, other operations of determining the distal end of the region of interest are described.

10 In some embodiments, in the operation of repeatedly expanding the region of interest, even though the difference in depth between the region of interest and the third neighboring region is less than the threshold, the expansion of the region of interest can be suspended when an expansion count is equal to or greater than a reference count. Here, the reference count is a type of a configuration value which can be a reference for exclusion from and inclusion in the region of interest, and the usermay change the reference count according to an actual implementation case. In a case of including only a predetermined part of the gingiva region in the region of interest, a proper reference count may be obtained through experiment. According to the present embodiment, the region of interest may be determined to satisfy the intention of the user, and for example, even in a case of the gingiva region, only a partial region of the gingiva region necessary for dental treatment may be included in the region of interest. In addition, computing resources consumed by the operation for a region unnecessary for dental treatment can be saved.

10 13 FIG. In some other embodiments, when the region of interest is repeatedly expanded, the distance between the boundary of the region of interest and the region to be expanded (e.g., the third neighboring region) may increase as the expansion count increases. Here, the margin of increase in the distance is a type of a configuration value which can be a reference for exclusion from and inclusion in the region of interest, and the usermay change the margin of increase in the distance according to an actual implementation case. In a case of including only a predetermined part of the gingiva region in the region of interest, a proper margin of interest in the distance may be obtained through experiment. A detailed description related thereto is provided below with reference to.

13 FIG. 13 FIG. 12 12 FIGS.A toD 13 FIG. 20 720 730 750 771 773 775 777 779 761 763 765 767 769 761 763 765 767 769 710 761 763 765 767 769 761 763 765 767 769 771 773 775 777 779 761 763 765 767 769 a. is an exemplary view illustrating in detail an operation of expanding a region of interest and an operation of determining a neighboring region upon the expansion, which can be referred to in some embodiments of the present disclosure.is an example of a virtual sectional image obtained by simplifying the oral cavity of the subject, and illustrates an example in which a tooth regionand a gingiva regionare divided by a first boundaryHere, it can be identified that the distance,,,, orbetween the boundary of the region of interest and the region to be extended,,,, orincreases as the extension count increases, and in this case, as described with reference to, whether to include the region to be extended,,,, orin the region of interest may be determined by calculating a difference in depth in a scanning direction. As in the example illustrated in, even though all the regions to be expanded,,,, andare included in the gingiva region, it can be understood that first minute regions,,, andare included in the region of interest and a second minute regionis excluded from the region of interest. According to the present embodiment, the region of interest may be determined to satisfy the intention of the user, and for example, even in a case of the gingiva region, only a partial region of the gingiva region necessary for dental treatment may be included in the region of interest. In addition, computing resources consumed by the operation for a region unnecessary for dental treatment can be saved. Specifically, the distance,,,, orbetween the boundary of the region of interest and the region to be expanded,,,, orincreases as the expansion count increases, and thus the distal end of the region of interest may be determined more promptly by calculating a smaller number of expansion counts than that in a case where the distance between the boundary of the regio and the region to be expanded remains the same as the region of interest is expanded.

14 FIG.A The operations of determining the distal end of the region of interest are described above. According to the various rules described above, computing resources can be saved while determining the region of interest satisfying the intention of the user. Hereinafter, a specific example of a region of interest determined according to the above-described operations is described with reference to.

14 FIG.A 9 FIG. 14 FIG.A 9 FIG. 14 FIG.A 9 FIG. 14 FIG.A 650 610 650 650 650 650 610 610 610 650 610 610 651 650 a. a a b b c a. is an exemplary view illustrating a region of interest determined for a two-dimensional image of a target oral cavity as described with reference to.illustrates an example of an output imagecorresponding to the input imageillustrated in, and the output imageillustrated inincludes a region of interestComparingand, it can be understood that on the output image, the region of interest(corresponding to the tooth regionand the gingiva regionof the input image) and a noise region(corresponding to the soft tissue regionof the input image) are divided by a boundaryof the region of interest

5 FIG. 11 FIG. 15 19 FIGS.to 10 10 According to the image processing method according to some embodiments of the present disclosure described with reference to,, and related exemplary views, a region of interest of a two-dimensional image, which is necessary for dental treatment, i.e., which is an object of interest of the user, may be dynamically determined, and a three-dimensional image may be generated based on the determined two-dimensional image. In addition, according to the above-described image processing method, a high-quality three-dimensional image (i.e., an image including less noise) may be generated while minimizing the involvement of the user. Hereinafter, with reference to, other exemplary operations illustrating the image processing method according to some embodiments of the present disclosure will be described.

15 FIG. 15 FIG. 5 FIG. 210 110 120 130 First,is another exemplary flow chart illustrating an image processing method according to some embodiments of the present disclosure. Referring to, in operation S, a region of interest may be determined in a two-dimensional image of a target oral cavity. This operation can be understood with reference to the description of operations S, S, and Sof.

220 660 660 10 14 FIG.B a Next, in operation S, the region of interest on the two-dimensional image may be highlighted and displayed. Here, for highlighting of the region of interest, various known graphic visualization technologies may be referred to. For example, as in the output imageillustrated in, a maskmay be overlaid over the region of interest, but the scope of the present disclosure is not limited thereto, and any technologies which enable the userto visually recognize the determined region of interest and any methods of highlighting the region of interest may be included in the scope of the present disclosure.

900 100 930 931 910 900 920 900 10 910 930 900 16 FIG. In some embodiments, an output image obtained by highlighting the region of interest and a three-dimensional image corresponding to the output image may be displayed together. For example, as in a screenof the electronic deviceillustrated in, an output imageobtained by highlighting a region of interestand a three-dimensional imagecorresponding to the output image may be displayed in one screen. Here, a reference boxmay be further displayed on the screento enable the userto easily recognize a region on the three-dimensional imagecorresponding to the output imageoutput on the screen.

15 FIG. 2 FIG. 10 10 10 10 10 10 100 According to the image processing method according to some embodiments of the present disclosure described with reference toand related exemplary views, the usermay visually identify a region of interest utilized for conversion into a three-dimensional image. As the uservisually identifies such a region of interest, the usermay identify an error which may occur in determining the region of interest, and may correct the error through the user interface described above with reference to. Specifically, as the useridentifies an output image obtained by highlighting a region of interest and a three-dimensional image corresponding to the output image together in one screen, the usermay identify in more detail an error which may occur in determining the region of interest. In addition, after completion of scanning or during the scanning process, the userperforms dental treatment while identifying a region of interest of a two-dimensional image displayed on the screen of the electronic device, and thus the user convenience can be maximized.

17 FIG. 17 FIG. 5 FIG. 5 FIG. 310 110 120 130 320 140 Next,is another exemplary flow chart illustrating an image processing method according to some embodiments of the present disclosure. Referring to, in operation S, a region of interest may be determined in a two-dimensional image of a target oral cavity. This operation can be understood with reference to the description of operations S, S, and Sof. Next, in operation S, a three-dimensional image of the target oral cavity may be generated from the two-dimensional image including the region of interest. This operation can be understood with reference to the description of operation Sof.

330 1000 110 1200 1300 10 18 FIG. Next, in operation S, the region of interest may be highlighted and displayed on the three-dimensional image. Here, for highlighting the region of interest, various known graphic visualization technologies may be referred to. For example, as in a screenincluding a three-dimensional imageillustrated in, a mask may be overlaid over the region of interest including a tooth regionand a gingiva region. The scope of the present disclosure is not limited to this example, and any technologies which enable the userto visually recognize the determined region of interest and any methods of highlighting the region of interest may be included in the scope of the present disclosure.

1000 1100 1210 1310 10 19 FIG. In some embodiments, a tooth region which is more essential for dental treatment may be highlighted and displayed on the three-dimensional image. For example, as in the screenincluding a three-dimensional imageillustrated in, a mask may be overlaid over a tooth regionand may not be overlaid over a gingiva region, but the scope of the present disclosure is not limited thereto, and any technologies which enable the userto visually recognize the tooth region and any methods of highlighting the tooth region may be included in the scope of the present disclosure.

17 FIG. 10 10 10 100 According to the image processing method according to some embodiments of the present disclosure described with reference toand related exemplary views, the usermay visually identify a region of interest on a three-dimensional image. The usermay identify a region which is essential for dental treatment by visually identifying such a region of interest. That is, after completion of scanning or during the scanning process, the userperforms dental treatment while identifying a region of interest of a two-dimensional image displayed on the screen of the electronic device, and thus the user convenience can be maximized.

1 19 FIGS.to With reference to, various embodiments of the present disclosure and effects according to the embodiments are mentioned. The effects according to the technical spirit of the present disclosure are not limited to the above-mentioned effects, and other unmentioned effects can be clearly understood by those skilled in the art from the description above.

1 19 FIGS.to The technical spirit of the present disclosure described with reference tomay be implemented as computer-readable code on a recording medium readable by a computing device (e.g., electronic device). Here, a computer program implemented as code may include, when being loaded to a memory of the computing device, at least one instruction which causes a processor of the computing device to perform operations according to various embodiments of the present disclosure. Such a computer-readable recording medium may be, for example, a removable recording medium (e.g., CD, DVD, Blue-ray disk, USB drive, removable hard disk, etc.) or a fixed recording medium (e.g., ROM, RAM, built-in hard disk, etc.). In addition, the recording medium may be a non-transitory recording medium. Here, the non-transitory recording medium refers to a tangible medium regardless of whether data is stored semi-permanently or temporarily, and does not include a temporarily transmitted signal. Moreover, the computer program recorded in the recording medium may be transmitted to another computing device via a network such as Internet and installed in another computing device, and may be thus used in other computing devices.

Although the technical spirit of the present disclosure has been described by the examples described in some embodiments and illustrated in the accompanying drawings, it should be noted that various substitutions, modifications, and changes can be made without departing from the scope of the present disclosure which can be understood by those skilled in the art to which the present disclosure pertains. In addition, it should be noted that that such substitutions, modifications and changes are intended to fall within the scope of the appended claims.