Method and Device for Outputting Pathology Slide Image

This application is a continuation of U.S. Application No. 18/087,382 filed December 22, 2022, which is based on and claims priority under 35 U.S.C. §119 to Korean Patent Applications No. 10-2021-0187098, filed on December 24, 2021, and No. 10-2022-0028108, filed on March 4, 2022, in the Korean Intellectual Property Office, the disclosure of which is incorporated by reference herein in its entirety.

The present disclosure relates to a method and device for outputting a pathology slide image.

Digital pathology is a field for obtaining histological information of a patient or predicting a prognosis by using a whole slide image generated by scanning a pathology slide image.

In conventional pathology, pathologists mount a glass slide on an optical microscope, adjust the magnification and focus of the microscope, and examine the slide image while moving the glass slide up and down or left and right. On the other hand, in digital pathology, pathologists output a pathology slide image converted into an image file on a monitor screen, adjust the magnification by using a mouse or a keyboard, and examine the slide image while moving the slide.

Accordingly, there is an increasing demand for a technique for solving side effects that may occur due to the difference between an image examination method in conventional pathology and an image examination method in digital pathology.

Provided are a method and device for outputting a pathology slide image. Provided is a computer-readable recording medium having recorded thereon a program for cause a computer to execute the method. The technical objectives to be solved are not limited to those described above, and other technical objects may be inferred from the following embodiments.

Additional aspects will be set forth in part in the description which follows and, in part, will be apparent from the description, or may be learned by practice of the presented embodiments of the present disclosure.

According to an aspect of an embodiment, a method of outputting a pathology slide image includes receiving a user input related to a method of outputting at least one region included in the pathology slide image, based on the user input, determining an area of a guide to be output on the pathology slide image and a region of the pathology slide image included in the guide, and based on the determined area and region, outputting the pathology slide image on which the guide is overlaid.

According to an aspect of another embodiment, a computer-readable recording medium includes a recording medium having recorded thereon a program for causing a computer to execute the method described above.

According to an aspect of another embodiment, a computing device includes at least one memory, and at least one processor, wherein the at least one processor is configured to determine, based on a user input related to a method of outputting at least one region included in a pathology slide image, an area of a guide to be output on the pathology slide image and a region of the pathology slide image included in the guide, and control a display device to output the pathology slide image on which the guide is overlaid, based on the determined area and region.

Reference will now be made in detail to embodiments, examples of which are illustrated in the accompanying drawings, wherein like reference numerals refer to like elements throughout. In this regard, the present embodiments may have different forms and should not be construed as being limited to the descriptions set forth herein. Accordingly, the embodiments are merely described below, by referring to the figures, to explain aspects. As used herein, the term "and/or" includes any and all combinations of one or more of the associated listed items.

Although the terms used in the embodiments are selected from among common terms that are currently widely used, the terms may be different according to an intention of one of ordinary skill in the art, a precedent, or the advent of new technology. Also, in particular cases, the terms are discretionally selected by the applicant of the present disclosure, in which case, the meaning of those terms will be provided in detail in the corresponding description. Therefore, the terms used herein are not merely designations of the terms, but the terms are defined based on the meaning of the terms and content throughout the present specification.

Throughout the present specification, when a part "includes" a component, it means that the part may additionally include other components rather than excluding other components as long as there is no particular opposing recitation. Also, the terms described in the present specification, such as "...er (or)", "... unit", "... module", etc., denote a unit that performs at least one function or operation, which may be implemented as hardware or software or a combination thereof.

In addition, although the terms such as "first" or "second" may be used herein to describe various elements, these elements should not be limited by these terms. These terms may be only used to distinguish one element from another element.

Hereinafter, embodiments are described in detail with reference to the accompanying drawings. The embodiments may, however, be embodied in many different forms and should not be construed as being limited to the embodiments set forth herein.

1 FIG. is a diagram for describing an example of a system for outputting a pathology slide image according to an embodiment.

1 FIG. 1 10 20 10 20 Referring to, a systemincludes a user terminaland a server. For example, the user terminaland the servermay be connected to each other in a wired or wireless communication manner to transmit and receive data (e.g., image data) to and from each other.

1 FIG. 1 10 20 1 10 20 10 20 Althoughillustrates, for convenience of description, that the systemincludes the user terminaland the server, the present disclosure is not limited thereto. For example, an external devices (not shown) may be included in the system, and operations of the user terminaland the serverto be described below may be performed by a single device (e.g., the user terminalor the server).

10 40 30 10 40 The user terminalmay output a pathology slide image, and a usermay manipulate the user terminalin various ways to observe the pathology slide imagefrom various points of view.

40 40 40 Here, the pathology slide imagemay refer to an image obtained by scanning a pathology slide that is fixed and stained through a series of chemical treatments in order to observe a tissue removed from a human body with a microscope. For example, the pathology slide imagemay refer to a whole slide image including a high-resolution image of a whole slide. As another example, the pathology slide imagemay refer to a portion of such a high-resolution whole slide image.

40 In addition, the pathology slide imagemay refer to a patch region divided into patch units from the whole slide image. For example, the patch may have a size of a certain area. Alternatively, the patch may refer to a region including each of objects included in the whole slide.

40 In addition, the pathology slide imagemay refer to a digital image obtained by converting a pathology slide into an image file by using a digital scanner, and may include information about cells, tissues, and/or structures in a human body.

10 40 30 40 10 40 30 40 10 40 30 30 10 40 The user terminalmay store and output the pathology slide image, and receive a user input from the userto overlay a guide on the pathology slide image. For example, the user terminalmay enlarge or reduce at least one region of the pathology slide image, and may identify a region observed by the userfrom the pathology slide image. Also, the user terminalmay store the pathology slide imagethat is updated by a manipulation by the user. In addition, the usermay record and store, in the user terminal, information about a finding from the pathology slide image.

10 40 20 10 20 The user terminalmay transmit and receive various pieces of data as well as the pathology slide imageto and from the serverand/or the external device (not shown) through communication therewith. In addition, the user terminalmay store the transmitted or received data therein, or may request the serverand/or the external device (not shown) to store the transmitted or received data.

10 10 The user terminalmay be a computing device provided with a display device and a device for receiving a user input (e.g., a keyboard or a mouse), and including a memory and a processor. For example, the user terminalmay be a notebook personal computer (PC), a desktop PC, a laptop computer, a tablet computer, a smart phone, or the like, but is not limited thereto.

20 10 20 40 20 20 20 10 20 1 14 FIGS.to The servermay be a device that communicates with the user terminaland the external device (not shown). For example, the servermay be a device that stores various pieces of data including the pathology slide image. As another example, the servermay be a computing device including a memory and a processor, and having a computing capability. In a case in which the serveris a computing device, the servermay perform at least some of operations of the user terminalto be described below with reference to. For example, the servermay be a cloud server, but is not limited thereto.

30 40 10 10 40 40 30 The usermay be a person that observes the pathology slide imageoutput on the user terminal, manipulate the user terminaloutputting the pathology slide image, or records a finding from the pathology slide image. For example, the usermay be a pathologist, but is not limited thereto.

30 40 In the traditional pathology, pathologists examine an image by observing a circular image through a lens of an optical microscope while manipulating an objective lens and a drive adjustment knob by hand. On the contrary, in the digital pathology, the userexamines an image while observing the pathology slide imageoutput on a display device by using a computer and a mouse.

40 It is difficult for pathologists, who are accustomed to examining images with conventional optical microscopes, to adapt to a method of adjusting the magnification of a slide image or searching a region of interest (e.g., a region of lesion), due to the interface in the digital pathology. Accordingly, it may take a long time to examine the pathology slide image, or a result of the examining may be inaccurate.

10 40 40 For example, the histological grade of breast cancer is determined based on the number of mitoses of cancer cells identified in an areatimes the range that may be observed in a high-power field (HPF) of an optical microscope. However, because it is difficult to estimate, from the pathology slide image, the range of HPF seen on a microscope, it may not be easy for a pathologist to determine an accurate histological grade based on the pathology slide image.

10 40 40 10 10 30 The user terminalaccording to an embodiment determines, based on a user input, the area of the guide to be output on the pathology slide imageand a region included in the guide (i.e., a region of the pathology slide imageto be included in the guide). Then, the user terminaloutputs a pathology slide image on which the guide is overlaid, according to a result of the determining. Accordingly, the user terminalmay output an image that may be examined in the same manner as in a method of examining an image by using an optical microscope. Accordingly, even the user, who is accustomed to examining an image with a conventional optical microscope, may easily perform image examination according to the digital pathology.

10 40 30 40 In addition, the user terminalaccording to an embodiment may output a region of the pathology slide imagethat has been observed by the user, and the other regions (i.e., unobserved regions), to be distinguished from each other. Accordingly, the usermay intuitively check whether there is an unobserved region in the pathology slide image.

10 40 30 In addition, the user terminalaccording to an embodiment determines whether a region of interest is included in the regions of the pathology slide imagethat have been observed by the user. Accordingly, the usermay intuitively check whether region of interests have been completely observed.

10 2 14 FIGS.A to Hereinafter, a method, performed by the user terminal, of outputting a pathology slide image will be described in detail with reference to.

10 40 40 10 20 For convenience of description, throughout the present specification, it is described that the user terminaldetermines the area of the guide to be output on the pathology slide imageand the region included in the guide, and outputs the pathology slide imageon which the guide is overlaid, but the present disclosure is not limited thereto. For example, at least some of the operations performed by the user terminalmay be performed by the server.

10 20 20 40 10 10 40 20 20 1 11 FIGS.to In other words, at least some of the operations of the user terminaldescribed with reference tomay be performed by the server. For example, the servermay determine the area of the guide to be output on the pathology slide imageand the region included in the guide, and transmit, to the user terminal, a result of the determining. In addition, the user terminalmay output the pathology slide imageon which the guide is overlaid according to information (e.g., information about the areas of the pathology slide image and the guide or information about the region included the guide) transmitted from the server. However, the operation of the serveris not limited to the above-described operation.

2 FIG.A 10 is a configuration diagram illustrating an example of the user terminalaccording to an embodiment.

2 FIG.A 2 FIG.A 2 FIG.A 2 FIG.A 100 110 120 130 140 100 110 120 130 140 Referring to, a user terminalincludes a processor, a memory, an input/output interface, and a communication module. For convenience of description,illustrates only components related to the present disclosure. Accordingly, other general-purpose components other than the components illustrated inmay be further included in the user terminal. In addition, it is apparent to those of skill in the art related to the present disclosure that the processor, the memory, the input/output interface, and the communication moduleillustrated inmay be implemented as independent devices.

110 120 20 110 100 The processormay process commands of a computer program by performing basic arithmetic, logic, and input/output operations. Here, the command may be provided from the memoryor an external device (e.g., the server). In addition, the processormay control the overall operation of other components included in the user terminal.

110 110 In particular, the processormay determine the area of the guide to be output on the pathology slide image, and the region of the pathology slide image included in the guide, based on a user input related to a method of outputting at least one region included in the pathology slide image. In addition, the processormay control a display device to output the pathology slide image on which the guide is overlaid, based on the determined area and region.

110 In addition, the processormay identify regions of the pathology slide image that have been observed by the user according to a certain reference, and control the display device to output the observed regions and the other regions to be distinguished from each other, according to a result of the identifying.

110 110 110 In addition, the processormay set at least one region of interest in the pathology slide image, and identify regions in the pathology slide image that have been observed by the user according to a certain reference. Then, the processormay determine whether the regions observed by the user include the region of interest. Then, when a user input related to termination of observation of the pathology slide image is received, the processormay generate a notification signal for asking the user whether to additionally observe the pathology slide image based on a result of the determining.

110 110 110 110 The processormay be implemented as an array of a plurality of logic gates, or may be implemented as a combination of a general-purpose microprocessor and a memory storing a program executable by the microprocessor. For example, the processormay include a general-purpose processor, a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor, a digital signal processor (DSP), a controller, a microcontroller, a state machine, and the like. In some environments, the processormay include an application-specific integrated circuit (ASIC), a programmable logic device (PLD), a field-programmable gate array (FPGA), and the like. For example, processormay also refer to a combination of processing devices, such as a combination of a DSP and a microprocessor, a combination of a plurality of microprocessors, a combination of one or more microprocessors in combination with a DSP core, or any other combination of such components.

110 3 14 FIGS.to Detailed examples in which the processoraccording to an embodiment operates will be described with reference to.

120 120 120 110 3 14 FIGS.to The memorymay include any non-transitory computer-readable recording medium. For example, the memorymay include a permanent mass storage device, such as random-access memory (RAM), read-only memory (ROM), a disk drive, a solid-state drive (SSD), or flash memory. As another example, the permanent mass storage device, such as ROM, an SSD, flash memory, or a disk drive, may be a permanent storage device separate from the memory. Also, the memorymay store an operating system (OS) and at least one piece of program code (e.g., code for the processorto perform an operation to be described below with reference to).

120 100 120 140 120 110 140 3 14 FIGS.to These software components may be loaded from a computer-readable recording medium separate from the memory. The separate computer-readable recording medium may be a recording medium that may be directly connected to the user terminal, and may include, for example, a computer-readable recording medium, such as a floppy drive, a disk, a tape, a digital video disc (DVD)/compact disc ROM (CD-ROM) drive, or a memory card. Alternatively, the software components may be loaded into the memorythrough the communication module, instead of a computer-readable recording medium. For example, at least one program may be loaded to the memorybased on a computer program (e.g., a computer program for the processorto perform an operation to be described below with reference to) installed by files provided by developers or a file distribution system that provides an installation file of an application, through the communication module.

130 100 100 130 110 130 110 2 FIG.A The input/output interfacemay be a unit for an interface with a device (e.g., a keyboard or a mouse) for input or output that may be connected to the user terminalor included in the user terminal. Althoughillustrates that the input/output interfaceis an element implemented separately from the processor, the present disclosure is not limited thereto, and the input/output interfacemay be implemented to be included in the processor.

140 20 100 140 100 110 20 140 The communication modulemay provide a configuration or function for the serverand the user terminalto communicate with each other through a network. In addition, the communication modulemay provide a configuration or function for the user terminalto communicate with another external device. For example, a control signal, a command, data, and the like provided under control by the processormay be transmitted to the serverand/or an external device through the communication moduleand a network.

2 FIG.A 100 100 Meanwhile, although not illustrated in, the user terminalmay further include a display device. Alternatively, the user terminalmay be connected to an independent display device in a wired or wireless communication manner to transmit and receive data to and from each other.

2 FIG.B is a configuration diagram illustrating an example of a server according to an embodiment.

2 FIG.B 2 FIG.B 2 FIG.B 2 FIG.B 200 210 220 230 200 210 220 230 Referring to, a serverincludes a processor, a memory, and a communication module. For convenience of description,illustrates only components related to the present disclosure. Accordingly, other general-purpose components other than the components illustrated inmay be further included in the server. In addition, it is apparent to those of skill in the art related to the present disclosure that the processor, the memory, and the communication moduleillustrated inmay be implemented as independent devices.

210 210 10 10 The processormay determine the area of a guide to be output on a pathology slide image and a region of ​​the pathology slide image included in the guide, based on a user input related to a method of outputting at least one region included in the pathology slide image. In addition, the processormay transmit, to the user terminal, relevant information such that the pathology slide image on which the guide is overlaid is output on the user terminal.

210 In addition, the processormay identify, from the pathology slide image, regions that have been observed by the user according to a certain reference.

210 210 210 In addition, the processormay set at least one region of interest in the pathology slide image, and identify regions in the pathology slide image that have been observed by the user according to a certain reference. Then, the processormay determine whether the regions observed by the user include the region of interest. Then, when a user input related to termination of observation of the pathology slide image is received, the processormay generate a notification signal for asking the user whether to additionally observe the pathology slide image based on a result of the determining.

110 210 10 20 2 FIG.A In other words, the operation of the processordescribed above with reference tomay be performed by the processor. In this case, the user terminalmay output, through a display device, information transmitted from the server.

210 110 2 FIG.A Meanwhile, an example of an implementation of the processoris the same as that of the processordescribed above with reference to, and thus, detailed descriptions thereof will be omitted.

210 220 210 220 Various pieces of data, such as a pathology slide image or data generated according to an operation of the processor, may be stored in the memory. Also, an OS and at least one program (e.g., a program required for the processorto operate) may be stored in the memory.

220 220 2 FIG.A Meanwhile, an example of an implementation of the memoryis the same as that of the memorydescribed above with reference to, and thus, detailed descriptions thereof will be omitted.

230 200 100 230 200 210 100 230 The communication modulemay provide a configuration or function for the serverand the user terminalto communicate with each other through a network. In addition, the communication modulemay provide a configuration or function for the serverto communicate with another external device. For example, a control signal, a command, data, and the like provided under control by the processormay be transmitted to the user terminaland/or an external device through the communication moduleand a network.

3 FIG. is a flowchart illustrating an example of a method of outputting a pathology slide image according to an embodiment.

3 FIG. 1 2 FIGS.andA 1 2 FIGS.andA 3 FIG. 10 100 110 10 100 110 Referring to, the method of outputting a pathology slide image includes operations that are processed, in a time-series manner, by the user terminaloror the processorillustrated in. Therefore, the descriptions provided above with respect to the user terminaloror the processorillustrated in, which are even omitted below, may also be applied to the method of outputting a pathology slide image of.

310 10 100 In operation, the user terminalorreceives a user input related to a method of outputting at least one region included in the pathology slide image.

40 10 100 10 100 For example, the pathology slide imagemay be output on the display device of the user terminalor, and then, the user terminalormay receive a user input. The user input may be received through a separate image output on the display device (e.g., an image configured to receive a user input), through a movement of a mouse wheel, or through an input of a keyboard shortcut preset by the user, but the present disclosure is not limited thereto.

40 30 30 40 The user input may be a manipulation on a region in the pathology slide imagethat the userdesires to closely observe. For example, the usermay enlarge or reduce a partial region of the pathology slide imagethrough the user input.

320 110 40 40 In operation, the processordetermines, based on the user input, the area of a guide to be output on the pathology slide imageand a region of the pathology slide imageincluded in the guide.

110 40 110 40 For example, the processormay increase or decrease the area of the guide in a state in which the region of the pathology slide imageincluded in the guide is fixed, based on a magnification specified according to the user input. As another example, the processormay enlarge or reduce the region of the pathology slide imageincluded in the guide in a state in which the area of the guide is fixed, based on a magnification specified according to the user input.

30 30 40 30 Here, the guide may limit a region that the userdesires to closely observe. In a case in which image examination is performed by using an optical microscope according to a related-art technique, the userobserves the image through an eyepiece. Thus, the guide may have a shape that appears on the pathology slide imageto be the same as the region observed by the userthrough the eyepiece.

3 FIG. 110 40 110 40 40 Meanwhile, although not illustrated in, the processormay calculate a scan magnification and a length per pixel of the pathology slide image. Accordingly, the processormay calculate the area of the guide to be displayed on the pathology slide imageand the region of the pathology slide imageincluded in the guide, according to the user input.

330 110 110 In operation, the processoroutputs the pathology slide image on which the guide is overlaid based on the determined area and region. For example, the processormay control the display device to output the pathology slide image on which the guide is overlaid.

110 40 40 320 The processormay overlay the guide on the pathology slide image by modifying (e.g., enlarging or reducing) the pathology slide imageaccording to the area of the guide and the region of the pathology slide imageincluded in the guide, which are determined in operation.

4 5 FIGS.to Hereinafter, examples in which a pathology slide image on which a guide is overlaid is output will be described with reference to.

4 FIG.A 4 FIG.B andare diagrams for describing an example in which a pathology slide image on which a guide is overlaid is output, according to an embodiment.

4 FIG.A 410 410 30 30 410 Referring to, a guideis overlaid on a pathology slide image. The guidemay be designated by the useras a mark for distinguishing a region that the userdesired to closely observe, from the other regions. For example, the guidemay have a circular shape, but is not limited thereto.

410 410 410 30 410 410 30 410 There may be various examples of overlaying the guideon the pathology slide image. For example, the guidemay be initially displayed at a preset position of the pathology slide image (e.g., the central position of the pathology slide image or the position of a region of interest), and then, the position of the guidemay be changed or fixed according to a manipulation by the user. As another example, the guidemay be initially displayed along a mark (e.g., a mouse cursor) on the pathology slide image, and then the position of the guidemay be fixed according to a manipulation by the user. However, examples of displaying the guideon the pathology slide image are not limited to those described above, and any method that may help the user 30 observe the pathology slide image may be applied without limitation.

431 410 410 431 30 Meanwhile, a magnificationmay be displayed in a region adjacent to the guide. Here, the term 'magnification' refers to an actual magnification that may be implemented with an optical microscope. Accordingly, through the guideand the magnificationdisplayed on the pathology slide image, the usermay have a similar experience to observing an image by using an optical microscope.

110 110 110 4 FIG.A In response to a user input received while the pathology slide image is output on the display device, the processoroutput the guide overlaid on the pathology slide image. For example, when a user input for increasing or decreasing the magnification is received while the pathology slide image illustrated inis output on the display device, the processordetermines the area of the guide and a region included in the guide, which correspond to the user input. Then, the processoroverlays the guide on the pathology slide image, based on the determined area of the guide and the determined region included in the guide.

110 For example, the processormay increase or decrease the area of the guide in a state in which the region included in the guide is fixed, based on a magnification specified according to the user input.

4 FIG.A 4 FIG.B 4 FIG. 4 FIG. x x x x x x 110 410 420 420 100 410 110 420 410 420 In the example ofand, it is assumed that the user input is for increasing the magnification to be between 40magnification and 100magnification. In this case, the processormay determine that the area of the guideis larger than the area of a guidein a state in which the size of the pathology slide image output on the display device is fixed. That is, the size of the guidecorresponding tomagnification is smaller than the size of the guidecorresponding to 40magnification. In other words, in a state in which the size of the pathology slide image output on the display device is fixed, the processoradjusts the size of the guidesuch that the actual area of a region observed with an optical microscope at 40magnification (i.e., the area of the region ofof) and the actual area of a region observed with the optical microscope at 100magnification (i.e., the area of the region ofof) are equal to each other.

110 420 420 420 110 432 420 432 432 30 420 The processoroverlays the guideon the pathology slide image based on the area of the guide, and outputs the guide. Also, the processormay display a magnificationcorresponding to the user input, together with an output image (i.e., an image on which the guideis overlaid). Here, the magnificationrefers to an actual magnification of the optical microscope. That is, the magnificationrefers to a magnification set in the optical microscope when the userdesires to observe a region in the guidethrough the optical microscope.

4 FIG.A 4 FIG.B 431 432 30 431 432 110 410 420 431 432 30 Meanwhile, although not illustrated inand, an image for setting the magnificationormay be output on the display device together with the pathology slide image or before the pathology slide image is output. For example, the usermay specify the magnificationorthrough the image, and the processoroutputs the guideoron the pathology slide image in response to a user input for specifying the magnification. In addition, the image for setting the magnificationormay be variously configured such that the usermay select a desired magnification.

5 FIG.A 5 FIG.B andare diagrams for describing another example in which a pathology slide image on which a guide is overlaid is output, according to an embodiment.

5 FIG.A 5 FIG.B 4 FIG.A 4 FIG.B x x In the example ofand, it is assumed that a user input is for increasing the magnification to be between 40magnification and 100magnification, as in the example ofand.

5 FIG.A 110 110 When a user input for increasing or decreasing the magnification is received while the pathology slide image illustrated inis output on the display device, the processordetermines the area of the guide and a region included in the guide, which correspond to the user input. Then, the processoroverlays the guide on the pathology slide image, based on the determined area of the guide and the determined region included in the guide, and outputs the guide.

110 For example, the processormay enlarge or reduce the region included in the guide while the area of the guide is fixed, based on a magnification specified according to the user input.

40 100 110 510 520 510 520 520 100 510 40 110 520 40 510 100 520 x x x x x x When a user input for increasing the magnification frommagnification tomagnification is received, the processormay determine that the actual area of the region included in a guideis smaller than the actual area of the region included in a guidein a state in which the on-screen areas of the guideand the guideare fixed to be the same value. That is, the actual area of the region included in the guidecorresponding tomagnification is smaller than the actual area of the region included in the guidecorresponding tomagnification. In other words, the processoradjusts the actual area of the region included in the guidesuch that the size of the guide corresponding to the optical microscope atmagnification (i.e., the on-screen area of the guide) and the size of the guide corresponding to the optical microscope atmagnification (i.e., the on-screen area of the guide) are equal to each other.

110 531 532 510 520 30 531 532 531 532 30 510 520 In addition, the processordisplays the magnificationorin a region adjacent to the guideor, such that the usermay intuitively recognize the magnification of the currently displayed guide. Here, the magnificationorrefers to an actual magnification of the optical microscope. That is, the magnificationorrefers to a magnification set in the optical microscope when the userdesires to observe a region in the guideorthrough the optical microscope.

5 FIG.A 5 FIG.B 4 FIG.A 4 FIG.B 531 532 Meanwhile, although not illustrated inand, an image for setting the magnificationormay be output on the display device as described above with reference toand.

3 5 FIGS.to 30 As described above with reference to, even the user, who is accustomed to examining an image by using an existing optical microscope, may easily perform image examination according to the digital pathology.

4 5 FIGS.to Meanwhile, an example in which a user input for increasing the magnification is received is described above with reference to. The same method may also be applied to a case in which a user input for decreasing the magnification is received.

6 FIG. is a flowchart illustrating another example of a method of outputting a pathology slide image according to an embodiment.

6 FIG. 3 FIG. 610 630 310 330 610 630 Referring to, operationstoare the same as operationstoof. Therefore, detailed descriptions of operationstowill be omitted below.

640 10 100 40 110 40 In operation, the user terminaloroutputs an enlarged or reduced image of at least a portion of the pathology slide image, based on an additionally received user input. For example, the processormay control the display device to output the enlarged or reduced image of the at least a portion of the pathology slide image.

40 40 30 40 40 40 40 110 7 FIG. Here, the additional user input may be an input for enlarging or reducing the pathology slide imagewithout adjusting the magnification. For example, the pathology slide imagemay be enlarged or reduced by various additional user inputs, for example, by the userzooming in or out on the pathology slide imageby using a mouse wheel, or manipulating a UI (e.g., a button or a scroll bar) output on the display device. At this time, the guide overlaid on the pathology slide imagemay be enlarged or reduced together with the pathology slide image, or may be fixed regardless of the pathology slide imagebeing enlarged or reduced. Hereinafter, an example in which the processoroutputs an image based on an additional user input will be described with reference to.

7 FIG.A 7 FIG.B andare diagrams for describing another example in which a pathology slide image on which a guide is overlaid is output, according to an embodiment.

7 FIG.A 710 40 110 x Referring to, it is assumed that the pathology slide image on which a guidecorresponding tomagnification is displayed is output on the display device. Thereafter, according to a received user input, the processormay enlarge or reduce the pathology slide image.

7 FIG.B 7 FIG.B 7 FIG.A 7 FIG.B 720 710 720 x For example, referring to, a guideillustrated inhas the same magnification (i.e., 40magnification) as that of the guideillustrated in. That is, according to a user input for enlarging the pathology slide image, the pathology slide image in which a partial region is enlarged as illustrated inis output, and the size of the guidemay also be increased at the same time.

7 FIG.A 7 FIG.B Meanwhile, an example in which a user input for enlarging a pathology slide image is received is described above with reference toand. The same method may also be applied to a case in which a user input for reducing a pathology slide image is received.

8 FIG. is a flowchart illustrating another example of a method of outputting a pathology slide image according to an embodiment.

8 FIG. 3 FIG. 810 830 310 330 810 830 Referring to, operationstoare the same as operationstoof. Therefore, detailed descriptions of operationstowill be omitted below.

840 110 40 In operation, the processoridentifies, from the pathology slide image, a region that has been observed by the user according to a certain reference.

110 40 Here, the certain reference includes that a magnification specified according to a user input is greater than or equal to a threshold magnification. That is, the processormay determine which region in the pathology slide imageis observed by the user at a magnification greater than or equal to the threshold magnification.

30 30 For example, the threshold magnification may be a magnification of a level at which the usermay recognize the morphology of a cell based on an image. The threshold magnification may be preset or may be adjusted by the user.

850 10 100 110 In operation, the user terminaloroutput the observed region and the other regions to be distinguished from each other, according to a result of the identifying. For example, the processormay control the display device to output the observed region to be distinguished from the other regions.

40 For example, a thumbnail image may be overlaid on a portion of the pathology slide imageand output, and the thumbnail image may be an image expressed such that the observed region according to a certain reference and the other regions are distinguished from each other.

9 10 FIGS.and Hereinafter, an example in which an observed region according to a certain reference and the other regions are output to be distinguished from each other will be described with reference to.

9 FIG.A 9 FIG.C toare diagrams for describing an example in which an observed region according to a certain reference and the other regions are distinguished from each other, according to an embodiment.

9 FIG.A illustrates an original pathology slide image (i.e., an image without an observed region according to a certain reference).

3 7 FIGS.to 30 30 As described above with reference to, the usermay observe at least a portion of the original pathology slide image while enlarging or reducing it. In particular, the usermay observe the image while enlarging or reducing the image according to a desired magnification.

110 30 30 110 110 The processormay check whether the magnification of a region observed by the useris greater than or equal to a threshold magnification. For example, when information about a magnification specified by the useris received as a user input, the processormay determine whether the magnification according to the user input is greater than or equal to the threshold magnification by comparing the magnification with the threshold magnification. Then, the processormay control the display device to output the region observed with a magnification greater than or equal to the magnification threshold, to be distinguished from the other regions.

9 FIG.B 910 30 110 910 110 920 930 920 930 Referring to, a regioncurrently being observed by the useris displayed. The processordetermines whether a magnification corresponding to the regionis greater than or equal to the threshold magnification. In this way, the processormay identify a region, which is observed at a magnification greater than or equal to the threshold magnification, and other regions, and control the display device to output an image in which the regionand the regionsare distinguished from each other.

9 FIG.B 9 FIG.C 920 930 930 920 930 Referring toand, the regionmay be displayed to be brighter than the other regions, and thus distinguished from the other regions. However, the method of distinguishing the regionfrom the other regionsis not limited to the above-described method.

10 FIG. is a diagram for describing an example in which a thumbnail image is output, according to an embodiment.

10 FIG. 9 FIG. 10 FIG. 1000 1000 1000 illustrates an example in which a thumbnail imageis output to overlap a portion of a pathology slide image. Here, the thumbnail imagemay be the image described above with reference to. However, the region in which the thumbnail imageis output and the size thereof are not limited to the example illustrated in.

11 FIG. is a flowchart illustrating another example of a method of outputting a pathology slide image according to an embodiment.

11 FIG. 11 FIG. 3 10 FIGS.to 3 10 FIGS.to Although not illustrated in, an example to be described below with reference tomay be combined with at least some of the examples described above with reference toin parallel or in a time-series manner. Therefore, detailed descriptions of the contents provided above with reference towill be omitted below.

1110 110 In operation, the processorsets at least one region of interest in the pathology slide image.

40 30 40 40 Here, the region of interest refers to a region in the pathology slide imagethat needs to be observed by the user. For example, the region of interest may be, but is not limited to, an important region that needs to be observed for pathological examination on a disease. For example, in a case in which a diagnosis of lung cancer is required, a region of the pathology slide imagein which a lung cancer cell is present may be a region of interest, and in a case of examining mitosis of breast cancer, a region of the pathology slide imagein which mitosis is expected to be high may be a region of interest.

110 40 110 40 110 40 The processoranalyzes the pathology slide imageto set at least one region of interest. For example, the processormay detect a region of interest by analyzing the pathology slide imageby using a predetermined image processing technique. As another example, the processormay detect a region of interest in the pathology slide imageby using a machine learning model. In this case, the machine learning model may be trained to detect a region of interest in reference pathology slide images by using training data including a plurality of reference pathology slide images and a plurality of pieces of reference label information.

110 40 110 40 30 40 110 30 For example, the processormay analyze the entire pathology slide imageto set at least one region of interest. As another example, the processormay analyze a portion of the pathology slide imageto set at least one region of interest. In detail, the usermay specify in advance a portion of the pathology slide imagein which regions of interest are to be set, and the processormay analyze only the portion specified by the userto select at least one region of interest.

1120 110 In operation, the processoridentifies, from the pathology slide image, a region that has been observed by the user according to a certain reference.

110 40 Here, the certain reference includes that a magnification specified according to a user input is greater than or equal to a threshold magnification. That is, the processormay determine which region in the pathology slide imageis observed by the user at a magnification greater than or equal to the threshold magnification.

30 30 For example, the threshold magnification may be a magnification of a level at which the usermay recognize the morphology of a cell based on an image. The threshold magnification may be preset or may be adjusted by the user.

1130 110 In operation, the processordetermines whether the region observed by the user includes at least one region of interest.

110 1110 110 40 110 30 40 In other words, the processormay determine whether a region of interest is included in the region observed by the user. In operation, the processormay identify in advance the region of interest in the pathology slide image. Accordingly, the processormay determine whether the region of interest is included in the region observed by the userin the pathology slide image.

110 12 FIG. Hereinafter, an example in which the processordetermines whether a region of interest is included in a region observed by the user will be described with reference to.

12 FIG. is a diagram for describing an example in which a processor according to an embodiment operates.

12 FIG. 8 FIG. 110 1210 30 1220 1210 1120 110 1210 1220 1210 1220 1230 30 Referring to, the processormay distinguish, in a pathology slide image, a regionthat has been observed by the userfrom other regions. Here, the regionmay be a region that has been observed to meet a certain reference as described above with reference to operation. Also, a method, performed by the processor, of distinguishing the regionfrom the other regionsis the same as described above with reference to. That is, the regionand the other regionsmay be determined based on a regioncurrently being observed by the user.

110 1240 1210 1240 1110 110 1230 1210 1220 1240 1210 The processordetermines whether regions of interestare included in the region. For example, the fact that the regions of interestmay be detected in advance according to a certain reference is as described above with reference to operation. The processormay distinguish, based on the region, the regionfrom the other regions, and determine whether the regions of interestare included in the region.

110 30 1240 1210 13 14 FIGS.and In addition, the processormay provide a notification to the useraccording to whether the regions of interestare included in the region, which will be described below with reference to.

13 FIG. is a flowchart illustrating another example of a method of outputting a pathology slide image according to an embodiment.

13 FIG. 11 FIG. 1310 1320 1110 1120 1310 1320 Referring to, operationsandare the same as operationsandof. Therefore, detailed descriptions of operationsandwill be omitted below.

1330 10 100 In operation, the user terminalorreceives a user input related to termination of observation of the pathology slide image.

30 40 30 40 40 1330 40 1330 The user input includes various attempts by the userto terminate the observation of the pathology slide imagecurrently output on the display device. For example, the user input may include an attempt by the userto replace the currently output pathology slide imagewith another image or to terminate a program for output and observation of the pathology slide image. Alternatively, the user input in operationmay include a case in which no user input is received for a preset time period. For example, a case in which no user input is received for the preset time period after the pathology slide imageis output may correspond to reception of the user input in operation.

1340 110 In operation, the processordetermines whether at least one region of interest is included in the region observed by the user.

40 110 1120 1130 110 Here, the region observed by the user includes a region of the pathology slide imageobserved by the user according to a certain reference, and the certain reference includes that a magnification specified according to a user input is greater than or equal to a threshold magnification. An example in which the processordetermines whether a region of interest is included in a region observed by the user is as described above with reference to operationsand. In a case in which a plurality of regions of interest are present, the processormay determine whether all of the plurality of regions of interest are included in the region observed by the user.

110 110 40 110 Meanwhile, an example in which the processordetermines whether a region of interest is included in a region observed by the user is not limited to the above-described example. For example, in a case in which the ratio of the region observed by the user to the entire region of interest is less than a preset ratio, the processormay determine that the region of interest is not included in the region observed by the user. In a case of examining mitosis of breast cancer, a region expected to have high mitosis in the pathology slide imagemay be set as a region of interest. Here, in a case in which the region observed by the user does not include a preset proportion (e.g., 50 %) or more of the region of interest, the processormay determine that the region observed by the user does not include the region of interest.

1350 In a case in which the region of interest is not included in the region observed by the user, operationis performed, and otherwise, the process is terminated.

1350 110 40 In operation, the processorgenerates a notification signal for asking the user whether to additionally observe the pathology slide image.

1330 1340 40 110 30 40 As described above with reference to operationsand, the observation of the currently output pathology slide imagemay be terminated in a state in which the region of interest has not been observed. In this case, the processormay generate and output a notification signal to help the userdetermine whether to proceed with additional observation of the pathology slide image.

110 40 30 30 110 30 30 110 Meanwhile, regardless of whether the entire region of interest has been observed, the processormay determine whether to generate the notification signal according to whether a predefined condition is satisfied. For example, in a case in which at least a portion of the region of interest of the pathology slide imageis included in the region previously observed by the user, and even the region of interest is present in a region not observed by the user, the processormay not generate the notification signal. As another example, regardless of whether the region previously observed by the userincludes the region of interest, in a case in which the region of interest is present in the region not observed by the user, the processormay generates the notification signal.

For example, the notification signal may be an image output on the display device, a sound output through a speaker, or a vibration output through a vibration motor. However, examples of the notification signal are not limited to those described above.

14 FIG. is a diagram for describing an example in which a notification signal is generated, according to an embodiment.

14 FIG. 1420 1410 1430 110 1440 illustrates a case in which regions of interestare not included in a regionthat has been observed by the user. In this situation, when a user inputrelated to termination of observation of a pathology slide image is received, the processorgenerates and outputs a notification signal.

1440 1440 1420 1440 1440 14 FIG. Meanwhile, an example in which the notification signalis output is not limited to that illustrated in. In detail, in a case in which the notification signalis to notify the user that at least a portion of the regions of interesthas not been observed, the notification signalmay be of any type, such as an image, a sound, or a vibration. Also, in a case in which the notification signalis an image, the image may include any message.

30 1440 110 1420 30 When an attempt by the userto additionally observe the pathology slide image is received after the notification signalis output, the processormay output the regions of intereston the display device to help the userobserve the pathology slide image.

30 30 40 30 40 As described above, even the user, who is accustomed to examining an image with a conventional optical microscope, may easily perform image examination according to the digital pathology. In addition, the usermay intuitively check whether there is an unobserved region in the pathology slide image. Also, the usermay intuitively check whether the region of interest included in the pathology slide imagehas been completely observed.

Meanwhile, the above-described method may be written as a computer-executable program, and may be implemented in a general-purpose digital computer that executes the program by using a computer-readable recording medium. In addition, the structure of the data used in the above-described method may be recorded in a computer-readable recording medium through various means. The computer-readable recording medium may include a magnetic storage medium (e.g., ROM, RAM, a universal serial bus (USB) storage device, a floppy disk, or a hard disk), and an optical recording medium (e.g., a CD-ROM or a DVD), but is not limited thereto.

It will be understood by those of skill in the art that the present disclosure may be implemented in a modified form without departing from the intrinsic characteristics of the descriptions provided above. Therefore, the methods disclosed herein are to be considered in a descriptive sense only, and not for purposes of limitation, and the scope of the present disclosure is defined not by the above descriptions, but by the claims and their equivalents, and all variations within the scope of the claims and their equivalents are to be construed as being included in the present disclosure.

It should be understood that embodiments described herein should be considered in a descriptive sense only and not for purposes of limitation. Descriptions of features or aspects within each embodiment should typically be considered as available for other similar features or aspects in other embodiments. While one or more embodiments have been described with reference to the figures, it will be understood by those of ordinary skill in the art that various changes in form and details may be made therein without departing from the spirit and scope as defined by the following claims.