Information processing apparatus, information processing method, and information processing program for supporting interpretation of images

This application claims priority from Japanese Application No. 2022-150756, filed on Sep. 21, 2022, the entire disclosure of which is incorporated herein by reference.

The present disclosure relates to an information processing apparatus, an information processing method, and an information processing program.

In the related art, image diagnosis is performed using medical images obtained by imaging apparatuses such as computed tomography (CT) apparatuses and magnetic resonance imaging (MRI) apparatuses. In addition, medical images are analyzed via computer aided detection/diagnosis (CAD) using a discriminator in which learning is performed by deep learning or the like, and regions of interest including structures, lesions, and the like included in the medical images are detected and/or diagnosed. The medical images and analysis results via CAD are transmitted to a terminal of a healthcare professional such as a radiologist who interprets the medical images. The healthcare professional such as a radiologist interprets the medical image by referring to the medical image and analysis result using his or her own terminal and creates an interpretation report.

In addition, various methods for supporting the interpretation of medical images have been proposed. For example, JP2019-153250A discloses a technique for creating an interpretation report based on a keyword input by a radiologist and an analysis result of a medical image. In the technique disclosed in JP2019-153250A, a sentence to be included in the interpretation report is created by using a recurrent neural network trained to generate a sentence from input characters.

Further, for example, JP2000-308619A discloses a technique for, in a case of correcting the shape of a region of interest in a medical image, arranging multi-node line segments or point sequences at equal intervals and dragging one of them to follow and move other adjacent multi-node line segments or point sequences in accordance with a predetermined tension.

In the related art, in a case where a region of interest is detected from a medical image via CAD, there is a case that the region of interest may not be detected correctly due to inclusion of an extra peripheral range or omission of detection of a peripheral portion. In the case of using the detected region of interest (for example, in the case of highlighting the region of interest in the medical image, in the case of generating comments on findings regarding the region of interest, and the like), a user is required to manually correct the region of interest to the correct one. However, a technique that can reduce the time and effort in this case is desired. In the technique disclosed in JP2000-308619A, another point adjacent to the dragged point moves in accordance with a predetermined tension, and thus it may be difficult to perform correction accurately in accordance with a correct region of interest.

The present disclosure provides an information processing apparatus, an information processing method, and an information processing program capable of supporting interpretation of images.

According to a first aspect of the present disclosure, there is provided an information processing apparatus comprising at least one processor, in which the processor is configured to: acquire an image; display, on a display, a figure indicating a first region of interest included in the image in a superimposed manner on the image; receive a correction instruction for at least a part of the figure; and specify a second region of interest that at least partially overlaps with the first region of interest based on an image feature of the image and the correction instruction.

According to a second aspect of the present disclosure, in the above first aspect, the processor may be configured to display, on the display, a figure indicating the second region of interest in a superimposed manner on the image.

According to a third aspect of the present disclosure, in the above first aspect or second aspect, the processor may be configured to receive, as the correction instruction, correction of at least one point of points forming the figure.

According to a fourth aspect of the present disclosure, in the above third aspect, the processor may be configured to specify, as the second region of interest, a region of interest in which at least one point forming the figure after correction is located within a predetermined range from an outer edge of the region of interest among the regions of interest included in the image.

According to a fifth aspect of the present disclosure, in any one of the above first to fourth aspects, the processor may be configured to receive, as the correction instruction, an instruction in a language representing a change in a shape of the figure.

According to a sixth aspect of the present disclosure, in any one of the above first to fifth aspects, the processor may be configured to specify the second region of interest by using a first learning model trained in advance to receive the image, the first region of interest, and the correction instruction as inputs and output the second region of interest.

According to a seventh aspect of the present disclosure, in any one of the above first to fifth aspects, the processor may be configured to: generate a feature map of the image by using a second learning model trained in advance to receive an image as an input and output a feature map of the input image; and specify the second region of interest by using a third learning model trained in advance to receive the feature map, the first region of interest, and the correction instruction as inputs and output the second region of interest.

According to an eighth aspect of the present disclosure, in the above seventh aspect, the processor may be configured to specify the first region of interest based on the feature map.

According to a ninth aspect of the present disclosure, in any one of the above first to eighth aspects, the figure may be at least one of a bounding box, a mask, or a mesh.

According to a tenth aspect of the present disclosure, in any one of the above first to ninth aspects, the image may be a medical image, and the region of interest may be at least one of a region of a structure included in the medical image or a region of a lesion included in the medical image.

According to an eleventh aspect of the present disclosure, there is provided an information processing method comprising: acquiring an image; displaying, on a display, a figure indicating a first region of interest included in the image in a superimposed manner on the image; receiving a correction instruction for at least a part of the figure; and specifying a second region of interest that at least partially overlaps with the first region of interest based on an image feature of the image and the correction instruction.

According to a twelfth aspect of the present disclosure, there is provided an information processing program for causing a computer to execute a process comprising: acquiring an image; displaying, on a display, a figure indicating a first region of interest included in the image in a superimposed manner on the image; receiving a correction instruction for at least a part of the figure; and specifying a second region of interest that at least partially overlaps with the first region of interest based on an image feature of the image and the correction instruction.

With the information processing apparatus, the information processing method, and the information processing program according to the aspects of the present disclosure, it is possible to support the interpretation of images.

Hereinafter, an embodiment of the present disclosure will be described with reference to the drawings. First, a configuration of an information processing systemto which an information processing apparatusof the present disclosure is applied will be described.is a diagram showing a schematic configuration of the information processing system. The information processing systemshown inperforms imaging of an examination target part of a subject and storing of a medical image acquired by the imaging based on an examination order from a doctor in a medical department using a known ordering system. In addition, the information processing systemperforms an interpretation work of a medical image and creation of an interpretation report by a radiologist and viewing of the interpretation report by a doctor of a medical department that is a request source.

As shown in, the information processing systemincludes an imaging apparatus, an interpretation work station (WS)that is an interpretation terminal, a medical care WS, an image server, an image database (DB), a report server, and a report DB. The imaging apparatus, the interpretation WS, the medical care WS, the image server, the image DB, the report server, and the report DBare connected to each other via a wired or wireless networkin a communicable state.

Each apparatus is a computer on which an application program for causing each apparatus to function as a component of the information processing systemis installed. The application program may be recorded on, for example, a recording medium, such as a digital versatile disc read only memory (DVD-ROM) or a compact disc read only memory (CD-ROM), and distributed, and be installed on the computer from the recording medium. In addition, the application program may be stored in, for example, a storage apparatus of a server computer connected to the networkor in a network storage in a state in which it can be accessed from the outside, and be downloaded and installed on the computer in response to a request.

The imaging apparatusis an apparatus (modality) that generates a medical image showing a diagnosis target part of the subject by imaging the diagnosis target part. Examples of the imaging apparatusinclude a simple X-ray imaging apparatus, a computed tomography (CT) apparatus, a magnetic resonance imaging (MRI) apparatus, a positron emission tomography (PET) apparatus, an ultrasound diagnostic apparatus, an endoscope, a fundus camera, and the like. The medical image generated by the imaging apparatusis transmitted to the image serverand is saved in the image DB.

is a diagram schematically showing an example of a medical image acquired by the imaging apparatus. A medical image T shown inis, for example, a CT image consisting of a plurality of tomographic images Tto Tm (m is 2 or more) representing tomographic planes from the head to the lumbar region of one subject (human body).

is a diagram schematically showing an example of one tomographic image Tx out of the plurality of tomographic images Tto Tm. The tomographic image Tx shown inrepresents a tomographic plane including a lung. Each of the tomographic images Tto Tm may include a region SA of a structure showing various organs and viscera of the human body (for example, lungs, livers, and the like), various tissues constituting various organs and viscera (for example, blood vessels, nerves, muscles, and the like), and the like. In addition, each tomographic image may include a region AA of lesions such as, for example, nodules, tumors, injuries, defects, and inflammation. In the tomographic image Tx shown in, the lung region is the region SA of the structure, and the nodule region is the region AA of a lesion. A single tomographic image may include regions SA of a plurality of structures and/or regions AA of lesions. Hereinafter, at least one of the region SA of the structure included in the medical image or the region AA of the lesion included in the medical image will be referred to as a “region of interest”.

The interpretation WSis a computer used by, for example, a healthcare professional such as a radiologist of a radiology department to interpret a medical image and to create an interpretation report, and encompasses an information processing apparatusaccording to the present embodiment. In the interpretation WS, a viewing request for a medical image to the image server, various image processing for the medical image received from the image server, display of the medical image, and input reception of a sentence regarding the medical image are performed. In the interpretation WS, an analysis process for medical images, support for creating an interpretation report based on the analysis result, a registration request and a viewing request for the interpretation report to the report server, and display of the interpretation report received from the report serverare performed. The above processes are performed by the interpretation WSexecuting software programs for respective processes.

The medical care WSis a computer used by, for example, a healthcare professional such as a doctor in a medical department to observe a medical image in detail, view an interpretation report, create an electronic medical record, and the like, and includes a processing apparatus, a display apparatus such as a display, and an input apparatus such as a keyboard and a mouse. In the medical care WS, a viewing request for the medical image to the image server, display of the medical image received from the image server, a viewing request for the interpretation report to the report server, and display of the interpretation report received from the report serverare performed. The above processes are performed by the medical care WSexecuting software programs for respective processes.

The image serveris a general-purpose computer on which a software program that provides a function of a database management system (DBMS) is installed. The image serveris connected to the image DB. The connection form between the image serverand the image DBis not particularly limited, and may be a form connected by a data bus, or a form connected to each other via a network such as a network attached storage (NAS) and a storage area network (SAN).

The image DBis realized by, for example, a storage medium such as a hard disk drive (HDD), a solid state drive (SSD), and a flash memory. In the image DB, the medical image acquired by the imaging apparatusand accessory information attached to the medical image are registered in association with each other.

The accessory information may include, for example, identification information such as an image identification (ID) for identifying a medical image, a tomographic ID assigned to each tomographic image included in the medical image, a subject ID for identifying a subject, and an examination ID for identifying an examination. In addition, the accessory information may include, for example, information related to imaging such as an imaging method, an imaging condition, an imaging purpose, and an imaging date and time related to imaging of a medical image. The “imaging method” and “imaging condition” are, for example, a type of the imaging apparatus, an imaging part, an imaging protocol, an imaging sequence, an imaging method, the presence or absence of use of a contrast medium, a slice thickness in tomographic imaging, and the like. In addition, the accessory information may include information related to the subject such as the name, date of birth, age, and gender of the subject.

In a case where the image serverreceives a request to register a medical image from the imaging apparatus, the image serverprepares the medical image in a format for a database and registers the medical image in the image DB. In addition, in a case where the viewing request from the interpretation WSand the medical care WSis received, the image serversearches for a medical image registered in the image DBand transmits the searched for medical image to the interpretation WSand to the medical care WSthat are viewing request sources.

The report serveris a general-purpose computer on which a software program that provides a function of a database management system is installed. The report serveris connected to the report DB. The connection form between the report serverand the report DBis not particularly limited, and may be a form connected by a data bus or a form connected via a network such as a NAS and a SAN.

The report DBis realized by, for example, a storage medium such as an HDD, an SSD, and a flash memory. In the report DB, an interpretation report created in the interpretation WSis registered. In addition, the report DBmay store findings information regarding the medical image. Findings information includes, for example, information obtained by the interpretation WSthrough image analysis of a medical image using a computer aided detection/diagnosis (CAD) technology, an artificial intelligence (AI) technology, or the like, and information or the like input by a user after interpreting a medical image.

The findings information indicates, for example, various findings such as a type (name), a property, a position, a measurement value such as a size, and an estimated disease name of a region of interest included in the medical image. Examples of types (names) include the types of structures such as “lung” and “liver”, and the types of lesions such as “nodule” and “tumor”. The property mainly means the features of the lesion. For example, in the case of a lung nodule, findings indicating absorption values such as “solid type” and “frosted glass type”, margin shapes such as “clear/unclear”, “smooth/irregular”, “spicula”, “lobulation”, and “serration”, and an overall shape such as “round shape” and “irregular shape” can be mentioned. In addition, for example, there are findings regarding the relationship with surrounding tissues such as “pleural contact” and “pleural invagination”, and the presence or absence of contrast, washout, and the like.

The position means an anatomical position, a position in a medical image, and a relative positional relationship with other regions of interest such as “inside”, “margin”, and “periphery”. The anatomical position may be indicated by an organ name such as “lung” and “liver”, and may be expressed in terms of subdivided lungs such as “right lung”, “upper lobe”, and apical segment (“S1”). The measurement value is a value that can be quantitatively measured from a medical image, and is, for example, at least one of a size or a signal value of a region of interest. The size is represented by, for example, a major axis, a minor axis, an area, a volume, or the like of a region of interest. The signal value is represented by, for example, a pixel value in a region of interest, a CT value in units of HU, or the like. The estimated disease name is an evaluation result estimated based on the lesion, and, for example, the disease name such as “cancer” and “inflammation” and the evaluation result such as “negative/positive”, “benign/malignant”, and “mild/severe” regarding disease names and properties can be mentioned.

Further, in a case where the report serverreceives a request to register the interpretation report from the interpretation WS, the report serverprepares the interpretation report in a format for a database and registers the interpretation report in the report DB. Further, in a case where the report serverreceives the viewing request for the interpretation report from the interpretation WSand the medical care WS, the report serversearches for the interpretation report registered in the report DB, and transmits the searched for interpretation report to the interpretation WSand to the medical care WSthat are viewing request sources.

The networkis, for example, a network such as a local area network (LAN) and a wide area network (WAN). The imaging apparatus, the interpretation WS, the medical care WS, the image server, the image DB, the report server, and the report DBincluded in the information processing systemmay be disposed in the same medical institution, or may be disposed in different medical institutions or the like. Further, the number of each apparatus of the imaging apparatus, the interpretation WS, the medical care WS, the image server, the image DB, the report server, and the report DBis not limited to the number shown in, and each apparatus may be composed of a plurality of apparatuses having the same functions.

Incidentally, in a case where a region of interest is detected from a medical image via CAD, there is a case that the region of interest may not be detected correctly due to inclusion of an extra peripheral range or omission of detection of a peripheral portion. In the case of using the detected region of interest (for example, in the case of highlighting the region of interest in the medical image, in the case of generating comments on findings regarding the region of interest, and the like), a user is required to manually correct the region of interest to the correct one.

Therefore, the information processing apparatusaccording to the present embodiment has a function of reducing the time and effort of the correction and supporting the interpretation of the medical image by providing assistance in the case of correcting the region of interest detected from the medical image. The information processing apparatuswill be described below. As described above, the information processing apparatusis encompassed in the interpretation WS.

First, with reference to, an example of a hardware configuration of the information processing apparatusaccording to the present embodiment will be described. As shown in, the information processing apparatusincludes a central processing unit (CPU), a non-volatile storage unit, and a memoryas a temporary storage area. Further, the information processing apparatusincludes a displaysuch as a liquid crystal display, an input unitsuch as a keyboard and a mouse, and a network interface (I/F). The network I/Fis connected to the networkand performs wired and/or wireless communication. The CPU, the storage unit, the memory, the display, the input unit, and the network I/Fare connected to each other via a bussuch as a system bus and a control bus so that various types of information can be exchanged.

The storage unitis realized by, for example, a storage medium such as an HDD, an SSD, and a flash memory. An information processing programin the information processing apparatusis stored in the storage unit. The CPUreads out the information processing programfrom the storage unit, loads the read-out program into the memory, and executes the loaded information processing program. The CPUis an example of a processor of the present disclosure. As the information processing apparatus, for example, a personal computer, a server computer, a smartphone, a tablet terminal, a wearable terminal, or the like can be appropriately applied.

Next, with reference to, an example of a functional configuration of the information processing apparatusaccording to the present embodiment will be described. As shown in, the information processing apparatusincludes an acquisition unit, a specifying unit, a reception unit, and a controller. As the CPUexecutes the information processing program, the CPUfunctions as respective functional units of the acquisition unit, the specifying unit, the reception unit, and the controller.

The acquisition unitacquires a medical image obtained by imaging a subject from the image server.

The specifying unitspecifies a first region of interest Aincluded in the medical image acquired by the acquisition unit. As a method for specifying the first region of interest A, a known method using a CAD technology, an AI technology, or the like can be appropriately applied. For example, the specifying unitmay specify a first region of interest Afrom a medical image by using a learning model such as a convolutional neural network (CNN) that has been trained to receive the medical image as an input and specify and output a region of interest included in the medical image.

In addition, the specifying unitmay generate a comment on findings regarding the specified first region of interest A. As a method for generating a comment on findings, a known method using a CAD technology, an AI technology, or the like can be appropriately applied. For example, the specifying unitmay generate findings information of the first region of interest Aby using a learning model such as a CNN that has been trained in advance to receive the region of interest specified from the medical image as an input and output the findings information of the region of interest. After that, the specifying unitmay generate a comment on findings including the generated findings information. For example, the specifying unitmay generate a comment on findings by using a method using machine learning such as the recurrent neural network described in JP2019-153250A. Further, for example, the specifying unitmay generate a comment on findings by embedding findings information in a predetermined template.

The controllerperforms control to display, on the display, a figure indicating the first region of interest Aspecified by the specifying unitin a superimposed manner on the medical image acquired by the acquisition unit.shows an example of a screen Ddisplayed on the displayby the controller. The screen Dincludes a medical image T, in which the first region of interest Ais emphasized by being surrounded by a bounding box B. The bounding box Bis an example of a figure indicating a first region of interest of the present disclosure.

Note thatshows an example in which the bounding box Bis a rectangle, but the present disclosure is not limited thereto, and the bounding box Bmay be a polygon other than the rectangle. In addition, the figure indicating the first region of interest Ais not limited to the bounding box B, and for example, at least one of a mask that changes display methods of colors and the like between the first region of interest Aand other regions or a mesh that expresses a geometric shape of the first region of interest Amay be used.

In addition, the controllermay perform control to display, on the display, a comment on findings regarding the first region of interest Agenerated by the specifying unit. The screen Dofincludes a comment on findingsregarding the first region of interest A.

Here, it is assumed that the user desires to correct the region of interest (correct region of interest) of interest in the medical image Tof the screen Dfrom the first region of interest Ato a second region of interest A. In this case, the user selects a correction buttonon the screen Dvia the input unit. In a case where the correction buttonis selected, the controllertransitions to a screen Dshown in. The screen Dis a screen for receiving the correction of the figure indicating the first region of interest A.