A measurement method includes a generation step, a region detection step, an image display step, a point input step, and a measurement step. Three-dimensional image data including three-dimensional coordinates of two or more points on a subject are generated in the generation step. Two or more feature regions on the subject are detected on the basis of the three-dimensional image data in the region detection step. An image of the two or more feature regions is displayed on a display in the image display step. A point on the subject is accepted through an input device in the point input step. A size of the subject is measured on the basis of three-dimensional coordinates of two or more points on the subject.
Legal claims defining the scope of protection, as filed with the USPTO.
calculating three-dimensional coordinates of two or more points on a subject having a three-dimensional shape on the basis of a two-dimensional image of the subject, wherein the subject includes a surface defining a normal line having a direction perpendicular to the surface; generating three-dimensional image data including the three-dimensional coordinates of the two or more points; extracting, on the basis of the three-dimensional coordinates of the two or more points, shape features including a first shape feature in the three-dimensional shape of the subject and a second shape feature different from the first shape feature in the three-dimensional shape of the subject, the shape features being extracted on the basis of three-dimensional distances between the three-dimensional coordinates of the two or more points and a change in a direction of the normal line on the surface of a three-dimensional shape formed by the three-dimensional coordinates of the two or more points on the subject having the three-dimensional shape; detecting a first feature region on the three-dimensional shape of the subject having the first shape feature and a second feature region on the three-dimensional shape of the subject having the second shape feature; assigning each of the two or more points to one of the first feature region or the second feature region on the basis of the shape features; controlling a display to display an image of the first feature region and the second feature region so that the first feature region and the second feature region are visually distinguished from each other; selecting one of the first feature region and the second feature region as a selected region; accepting a point on the subject through an input device after displaying the image of the first feature region and the second feature region; generating point information indicating the accepted point; invalidating the accepted point information when a point outside the selected region is accepted as the point on the subject accepted through the input device; generating new point information indicating a point in the selected region; and measuring a size of the subject on the basis of three-dimensional coordinates of at least two points of the two or more points on the subject including the new point in the selected region and other than the invalid point information. . A measurement method executed by one or more processors, the measurement method comprising:
claim 1 selecting one of the first feature region and the second feature region as a selected region; and wherein the size of the subject is measured on the basis of the three-dimensional coordinates of the two or more points including the point indicated by the valid point information. validating the point information only when a point in the selected region is accepted as the point on the subject, . The measurement method according to, further comprising:
claim 2 wherein a feature region that is included in the first feature region and the second feature region and includes a first point on the subject is selected as the selected region when the first point is accepted as the point on the subject, wherein the size of the subject is measured on the basis of the three-dimensional coordinates of the two or more points including the first point and a second point in the selected region when the second point is accepted as the point on the subject, and wherein the second point is different from the first point. . The measurement method according to,
claim 2 wherein, when a position on the image of the first feature region and the second feature region is accepted through the input device, a feature region that is included in the first feature region and the second feature region and corresponds to the accepted position is selected as the selected region. . The measurement method according to,
claim 1 selecting one of the first feature region and the second feature region as a selected region; controlling the display to display a mark on one of the image of the three-dimensional image data and the two-dimensional image; and calculating a position at which the mark is displayed on the basis of information accepted through the input device, wherein the position at which the mark is displayed is restricted to a position in a region corresponding to the selected region, and wherein a point corresponding to the position of the mark is the point on the subject accepted through the input device. controlling the display to display one of an image of the three-dimensional image data and the two-dimensional image with the image of the first feature region and the second feature region; . The measurement method according to, further comprising:
claim 1 selecting one of the first feature region and the second feature region as a selected region, controlling the display to display a mark on the image of the first feature region and the second feature region; and calculating a position at which the mark is displayed on the basis of information accepted through the input device, wherein the position at which the mark is displayed is restricted to a position in the selected region, and wherein a point corresponding to the position of the mark is accepted as the point on the subject accepted through the input device. . The measurement method according to, further comprising:
claim 1 selecting one of the first feature region and the second feature region as a selected region; controlling the display to display a mark on one of the image of the three-dimensional image data and the two-dimensional image; and calculating a position at which the mark is displayed on the basis of information accepted through the input device, wherein the size of the subject is measured on the basis of the three-dimensional coordinates of the two or more points including the point indicated by the point information when the mark is displayed in a region corresponding to the selected region and the point corresponding to the position of the mark is accepted as the point on the subject accepted through the input device, and wherein a position at which the mark is displayed is changed to a position in the region corresponding to the selected region when the mark is displayed outside the region corresponding to the selected region and the point corresponding to the position of the mark is accepted as the point on the subject accepted through the input device. controlling the display to display one of an image of the three-dimensional image data and the two-dimensional image with the image of the first feature region and the second feature region; . The measurement method according to, further comprising:
claim 1 selecting one of the first feature region and the second feature region as a selected region; controlling the display to display a mark on the image of the first region and the second feature region; and calculating a position at which the mark is displayed on the basis of information accepted through the input device, wherein the size of the subject is measured on the basis of the three-dimensional coordinates of the two or more points including the point indicated by the point information when the mark is displayed in the selected region and the point corresponding to the position of the mark is accepted, and wherein a position at which the mark is displayed is changed to a position in the selected region when the mark is displayed outside the selected region and the point corresponding to the position of the mark is accepted. . The measurement method according to, further comprising:
claim 1 wherein the selected region is displayed in a first color and a feature region that is included in the first feature region and the second feature region and is different from the selected region is displayed in a second color different from the first color. selecting one of the first feature region and the second feature region as a selected region, . The measurement method according to, further comprising:
claim 1 wherein only the selected region is displayed. selecting one of the first feature region and the second feature region as a selected region, . The measurement method according to, further comprising:
claim 1 selecting one of the first feature region and the second feature region as a selected region, wherein the selected region is highlighted when displaying the selected region. . The measurement method according to, further comprising:
claim 1 detecting at least one of an edge and a step on the surface of the subject on the basis of the three-dimensional coordinates of the two or more points and the change in the direction of the normal line on the surface of the subject, wherein extracting the shape features comprises extracting the shape features on the basis of the three-dimensional distances between the two or more points and the at least one of the edge and the step detected on the basis of the change in the direction of the normal line on the surface of the subject. . The measurement method according to, further comprising:
calculating three-dimensional coordinates of two or more points on a subject having a three-dimensional shape on the basis of a two-dimensional image of the subject, wherein the subject includes a surface defining a normal line having a direction perpendicular to the surface; generating three-dimensional image data including the three-dimensional coordinates of the two or more points; extracting, on the basis of the three-dimensional coordinates of the two or more points, a shape feature in the three-dimensional shape of the subject, the shape features being extracted on the basis of three-dimensional distances between the three-dimensional coordinates of the two or more points and a change in a direction of the normal line on the surface of a three-dimensional shape formed by the three-dimensional coordinates of the two or more points on the subject having the three-dimensional shape; detecting a feature region on the three-dimensional shape of the subject having the shape feature; assigning at least one of the two or more points to the feature region on the basis of the shape feature; controlling a display to display one of an image of the three-dimensional image data, the two-dimensional image, and an image of the feature region; selecting one of the first feature region and the second feature region as a selected region; accepting a point on the subject through an input device after displaying the one of the image of the three-dimensional image data, the two-dimensional image, and the image of the feature region; generating point information indicating the accepted point; invalidating the generated point information when a point outside the selected region is accepted as the point on the subject accepted through the input device; generating new point information indicating a new point in the selected region; and wherein the point information is controlled in the first control so that the point information indicates a position in the feature region or a position on an edge of the feature region, wherein a mark is displayed on one of the image of the three-dimensional image data, the two-dimensional image, and the image of the feature region in the second control, wherein a position of the mark is restricted to a position in a region corresponding to the feature region, and wherein the point corresponding to the position of the mark is accepted in the accepting the point on the subject through the input device; and executing one of a first control and a second control, measuring a size of the subject on the basis of three-dimensional coordinates of at least two points of the two or more points on the subject including the new point in the selected region and other than the invalid point information. . A measurement method executed by one or more processors, the measurement method comprising:
calculate three-dimensional coordinates of two or more points on a subject having a three-dimensional shape on the basis of a two-dimensional image of the subject, wherein the subject includes a surface defining a normal line having a direction perpendicular to the surface; generate three-dimensional image data including the three-dimensional coordinates of the two or more points; extract, on the basis of the three-dimensional coordinates of the two or more points, shape features including a first shape feature in the three-dimensional shape of the subject and a second shape feature different from the first shape feature in the three-dimensional shape of the subject, the shape features being extracted on the basis of three-dimensional distances between the three-dimensional coordinates of the two or more points and a change in a direction of the normal line on the surface of a three-dimensional shape formed by the three-dimensional coordinates of the two or more points on the subject having the three-dimensional shape; detect a first feature region on the three-dimensional shape of the subject having the first shape feature and a second feature region on the three-dimensional shape of the subject having the second shape feature; assign each of the two or more points to one of the first feature region or the second feature region on the basis of the shape features; control a display to display an image of the first feature region and the second feature region so that the first feature region and the second feature region are visually distinguished from each other; select one of the first feature region and the second feature region as a selected region; accept a point on the subject through an input device after displaying the image of the first feature region and the second feature region; generate point information indicating the accepted point; and invalidate the generated point information when a point outside the selected region is accepted as the point on the subject accepted through the input device; generate new point information indicating a new point in the selected region; and measure a size of the subject on the basis of three-dimensional coordinates of at least two points of the two or more points on the subject including the new point in the selected region and other than the invalid point information. one or more processors configured to: . A measurement device comprising:
calculating three-dimensional coordinates of two or more points on a subject having a three-dimensional shape on the basis of a two-dimensional image of the subject, wherein the subject includes a surface defining a normal line having a direction perpendicular to the surface; generating three-dimensional image data including the three-dimensional coordinates of the two or more points; extracting, on the basis of the three-dimensional coordinates of the two or more points, shape features including a first shape feature in the three-dimensional shape of the subject and a second shape feature different from the first shape feature in the three-dimensional shape of the subject, the shape features being extracted on the basis of three-dimensional distances between the three-dimensional coordinates of the two or more points and a change in a direction of the normal line on the surface of a three-dimensional shape formed by the three-dimensional coordinates of the two or more points on the subject having the three-dimensional shape; detecting a first feature region on the three-dimensional shape of the subject having the first shape feature on the subject and a second feature region on the three-dimensional shape of the subject having the second shape feature; assigning each of the two or more points to one of the first feature region or the second feature region on the basis of the shape features; controlling a display to display an image of the first feature region and the second feature region so that the first feature region and the second feature region are visually distinguished from each other; selecting one of the first feature region and the second feature region as a selected region; accepting a point on the subject through an input device after displaying the image of the first feature region and the second feature region; generating point information indicating the accepted point; invalidating the accepted point information when a point outside the selected region is accepted as the point on the subject accepted through the input device; generating new point information indicating a point in the selected region; and measuring a size of the subject on the basis of three-dimensional coordinates of at least two points of the two or more points on the subject including the new point in the selected region and other then the invalid point information. . A non-transitory computer-readable recording medium storing a program causing a computer to at least execute:
Complete technical specification and implementation details from the patent document.
The present application is a continuation application based on International Patent Application No. PCT/JP2020/010021 filed on Mar. 9, 2020, the content of which is incorporated herein by reference.
The present invention relates to a measurement method, a measurement device, and a recording medium.
Industrial endoscope devices have been used for observation and inspection of internal damage, corrosion, and the like of boilers, pipes, aircraft engines, and the like. In such an endoscope device, multiple types of optical adapters for observing and inspecting various objects to be observed are prepared. Optical adapters are attached to the distal ends of endoscopes and are exchangeable. In an inspection using such an endoscope device, there is a desire to quantitatively measure the size of a defect, damage, or the like of a subject. To meet such a desire, there is an endoscope device provided with a three-dimensional measurement function.
For example, an endoscope device has a function of measuring geometric sizes of a subject on the basis of information of a point designated on an image by a user. For example, in a distance-between-two-points measurement, the three-dimensional distance between two measurement points designated by a user is measured. In a line-based measurement, a reference line is set on the basis of two reference points designated by a user, and the three-dimensional distance between a measurement point designated by a user and the reference line is measured. In a plane-based measurement, a reference plane is set on the basis of three or more reference points designated by a user, and the three-dimensional distance between a measurement point designated by a user and the reference plane is measured.
A user designates a point on an image displayed on a display. The accuracy of three-dimensional measurement depends on the position of a point designated by a user. For example, there is a case in which two objects overlap each other and it is difficult to recognize the border between the two objects. Even when a user determines that a correct point has been designated, the designated position may be actually on a different object than that on which the point exists. Since an incorrect point is designated, an endoscope device is unable to acquire a correct measurement result.
Japanese Unexamined Patent Application, First Publication No. 2010-127964 discloses a technique used when a user moves a cursor to a suitable position for measurement in a two-dimensional image. In this technique, movement of a cursor is controlled on the basis of the number of features of an image. In a first example, a cursor moves to a position at which the number of features of an image is the greatest. In a second example, the moving speed of a cursor changes on the basis of the number of features of an image. In a third example, the moving direction of a cursor changes on the basis of the number of features of an image.
According to a first aspect of the present invention, a measurement method executed by a processor includes a generation step, a region detection step, an image display step, a point input step, and a measurement step. In the generation step, three-dimensional coordinates of two or more points on a subject are calculated on the basis of a two-dimensional image of the subject and three-dimensional image data including the three-dimensional coordinates of the two or more points are generated. In the region detection step, two or more feature regions on the subject are detected on the basis of the three-dimensional image data. The three-dimensional shape of the subject has a common feature in each of the two or more feature regions. In the image display step, an image of the two or more feature regions is displayed on a display so that the two or more feature regions are visually distinguished from each other. In the point input step, a point on the subject is accepted through an input device after the image display step is executed and point information indicating the accepted point is generated. In the measurement step, the size of the subject is measured on the basis of three-dimensional coordinates of two or more points on the subject including the point indicated by the point information.
According to a second aspect of the present invention, in the first aspect, the measurement method may further include a region selection step and an information control step. In the region selection step, one of the two or more feature regions may be selected as a selected region. In the information control step, the point information may be validated only when a point in the selected region is accepted in the point input step. In the measurement step, the size of the subject may be measured on the basis of the three-dimensional coordinates of the two or more points including the point indicated by the valid point information.
According to a third aspect of the present invention, in the first aspect, one of an image of the three-dimensional image data and the two-dimensional image may be displayed on the display in the image display step. The measurement method may further include a region selection step, a mark display step, and a position calculation step. In the region selection step, one of the two or more feature regions may be selected as a selected region. In the mark display step, a mark may be displayed on one of the image of the three-dimensional image data and the two-dimensional image. In the position calculation step, the position at which the mark is displayed may be calculated on the basis of information accepted through the input device. The position at which the mark may be displayed is restricted to a position in a region corresponding to the selected region. The point corresponding to the position of the mark may be accepted in the point input step.
According to a fourth aspect of the present invention, in the first aspect, the measurement method may further include a region selection step, a mark display step, and a position calculation step. In the region selection step, one of the two or more feature regions may be selected as a selected region. In the mark display step, a mark may be displayed on the image of the two or more feature regions. In the position calculation step, the position at which the mark is displayed may be calculated on the basis of information accepted through the input device. The position at which the mark is displayed may be restricted to a position in the selected region. The point corresponding to the position of the mark may be accepted in the point input step.
According to a fifth aspect of the present invention, in the first aspect, the measurement method may further include a region selection step and an information control step. In the region selection step, one of the two or more feature regions may be selected as a selected region. In the information control step, the accepted point information may be invalidated when a point outside the selected region is accepted in the point input step and new point information indicating a point in the selected region may be generated. In the measurement step, the size of the subject may be measured on the basis of the three-dimensional coordinates of the two or more points indicated by the point information other than the invalid point information.
According to a sixth aspect of the present invention, in the first aspect, one of an image of the three-dimensional image data and the two-dimensional image may be displayed on the display in the image display step. The measurement method may further include a region selection step, a mark display step, and a position calculation step. In the region selection step, one of the two or more feature regions may be selected as a selected region. In the mark display step, a mark may be displayed on one of the image of the three-dimensional image data and the two-dimensional image. In the position calculation step, the position at which the mark is displayed may be calculated on the basis of information accepted through the input device. In the measurement step, the size of the subject may be measured on the basis of the three-dimensional coordinates of the two or more points including the point indicated by the point information when the mark is displayed in a region corresponding to the selected region and the point corresponding to the position of the mark is accepted in the point input step. The position at which the mark is displayed may be changed to a position in the region corresponding to the selected region when the mark is displayed outside the region corresponding to the selected region and the point corresponding to the position of the mark is accepted in the point input step.
According to a seventh aspect of the present invention, in the first aspect, the measurement method may further include a region selection step, a mark display step, and a position calculation step. In the region selection step, one of the two or more feature regions may be selected as a selected region. In the mark display step, a mark may be displayed on the image of the two or more feature regions. In the position calculation step, the position at which the mark is displayed may be calculated on the basis of information accepted through the input device. In the measurement step, the size of the subject may be measured on the basis of the three-dimensional coordinates of the two or more points including the point indicated by the point information when the mark is displayed in the selected region and the point corresponding to the position of the mark is accepted in the point input step. The position at which the mark is displayed may be changed to a position in the selected region when the mark is displayed outside the selected region and the point corresponding to the position of the mark is accepted in the point input step.
According to an eighth aspect of the present invention, in the second aspect, a feature region that is included in the two or more feature regions and includes a first point on the subject may be selected in the region selection step when the first point is accepted in the point input step. The size of the subject may be measured on the basis of the three-dimensional coordinates of the two or more points including the first point and a second point in the selected region when the second point is accepted in the point input step. The second point may be different from the first point.
According to a ninth aspect of the present invention, in the second aspect, when a position on the image of the two or more feature regions is accepted through the input device, a feature region that is included in the two or more feature regions and corresponds to the accepted position may be selected in the region selection step.
According to a tenth aspect of the present invention, in the first aspect, the measurement method may further include a region selection step of selecting one of the two or more feature regions as a selected region. The selected region may be displayed in a first color in the image display step and a feature region that is included in the two or more feature regions and is different from the selected region may be displayed in a second color different from the first color in the image display step.
According to an eleventh aspect of the present invention, in the first aspect, the measurement method may further include a region selection step of selecting one of the two or more feature regions as a selected region. In the image display step, only the selected region may be displayed.
According to a twelfth aspect of the present invention, in the first aspect, the measurement method may further include a region selection step of selecting one of the two or more feature regions as a selected region. In the image display step, the selected region may be highlighted.
According to a thirteenth aspect of the present invention, a measurement method executed by a processor includes a generation step, a region detection step, an image display step, a point input step, a control step, and a measurement step. In the generation step, three-dimensional coordinates of two or more points on a subject are calculated on the basis of a two-dimensional image of the subject and three-dimensional image data including the three-dimensional coordinates of the two or more points are generated. In the region detection step, one or more feature regions on the subject are detected on the basis of the three-dimensional image data. The three-dimensional shape of the subject has a common feature in each of the one or more feature regions. In the image display step, one of an image of the three-dimensional image data, the two-dimensional image, and an image of the one or more feature regions is displayed on a display. In the point input step, a point on the subject is accepted through an input device after the image display step is executed and point information indicating the accepted point is generated. In the control step, one of first control and second control is executed. The point information is controlled in the first control so that the point information indicates a point in one of the one or more feature regions. A mark is displayed on one of the image of the three-dimensional image data, the two-dimensional image, and the image of the one or more feature regions in the second control. The position of the mark is restricted to a position in a region corresponding to one of the one or more feature regions. In the point input step, the point corresponding to the position of the mark is accepted. In the measurement step, the size of the subject is measured on the basis of three-dimensional coordinates of two or more points on the subject including the point indicated by the point information.
According to a fourteenth aspect of the present invention, a measurement device includes a processor. The processor calculates three-dimensional coordinates of two or more points on a subject on the basis of a two-dimensional image of the subject. The processor generates three-dimensional image data including the three-dimensional coordinates of the two or more points. The processor detects two or more feature regions on the subject on the basis of the three-dimensional image data. The three-dimensional shape of the subject has a common feature in each of the two or more feature regions. The processor displays an image of the two or more feature regions on a display so that the two or more feature regions are visually distinguished from each other. The processor accepts a point on the subject through an input device after the image of the two or more feature regions is displayed. The processor generates point information indicating the accepted point. The processor measures the size of the subject on the basis of three-dimensional coordinates of two or more points on the subject including the point indicated by the point information.
According to a fifteenth aspect of the present invention, a non-transitory computer-readable recording medium saves a program causing a computer to execute a generation step, a region detection step, an image display step, a point input step, and a measurement step. In the generation step, three-dimensional coordinates of two or more points on a subject are calculated on the basis of a two-dimensional image of the subject and three-dimensional image data including the three-dimensional coordinates of the two or more points are generated. In the region detection step, two or more feature regions are detected on the subject on the basis of the three-dimensional image data. The three-dimensional shape of the subject has a common feature in each of the two or more feature regions. In the image display step, an image of the two or more feature regions is displayed on a display so that the two or more feature regions are visually distinguished from each other. In the point input step, a point on the subject is accepted through an input device after the image display step is executed and point information indicating the accepted point is generated. In the measurement step, the size of the subject is measured on the basis of three-dimensional coordinates of two or more points on the subject including the point indicated by the point information.
Hereinafter, embodiments of the present invention will be described with reference to the drawings.
1 FIG. 1 FIG. 7 7 70 71 72 73 74 shows a configuration of a measurement deviceaccording to a first embodiment of the present invention. The measurement deviceshown inincludes a generation unit, a region detection unit, a display control unit, a point input unit, and a measurement unit.
70 71 72 73 74 The generation unitcalculates three-dimensional coordinates of two or more points on a subject on the basis of a two-dimensional image of the subject and generates point-cloud data (three-dimensional image data) including the three-dimensional coordinates of the two or more points (generation step). The region detection unitdetects two or more feature regions on the subject on the basis of the point-cloud data (region detection step). The three-dimensional shape of the subject has a common feature in each of the two or more feature regions. The display control unitdisplays an image of the two or more feature regions on a display so that the two or more feature regions are visually distinguished from each other (image display step). The point input unitaccepts a point on the subject through an input device and generates point information indicating the accepted point after the image of the two or more feature regions is displayed (point input step). The measurement unitmeasures the size of the subject on the basis of three-dimensional coordinates of two or more points on the subject including the point indicated by the point information (measurement step).
1 FIG. 1 FIG. 1 FIG. Each unit shown inmay be constituted by at least one of a processor and a logic circuit. For example, the processor is at least one of a central processing unit (CPU), a digital signal processor (DSP), and a graphics-processing unit (GPU). For example, the logic circuit is at least one of an application-specific integrated circuit (ASIC) and a field-programmable gate array (FPGA). Each unit shown inmay include one or a plurality of processors. Each unit shown inmay include one or a plurality of logic circuits.
7 70 71 72 73 74 70 71 72 73 74 A computer of the measurement devicemay read a program and execute the read program. The program includes commands defining the operations of the generation unit, the region detection unit, the display control unit, the point input unit, and the measurement unit. In other words, the functions of the generation unit, the region detection unit, the display control unit, the point input unit, and the measurement unitmay be realized by software.
7 The program described above may be recorded on a computer-readable recording medium. The program may be transmitted from a computer storing the program to the measurement devicethrough a transmission medium or transmission waves in a transmission medium. The “transmission medium” transmitting the program is a medium having a function of transmitting information. The medium having the function of transmitting information includes a network (communication network) such as the Internet and a communication circuit line (communication line) such as a telephone line. The program described above may realize some of the functions described above. In addition, the program described above may be a differential file (differential program). The functions described above may be realized by a combination of a program that has already been recorded in a computer and a differential program.
2 FIG. 2 FIG. Three-dimensional measurement in the first embodiment will be described by using.shows a procedure of the three-dimensional measurement.
70 1 1 The generation unitcalculates three-dimensional coordinates (3D coordinates) of two or more points on a subject on the basis of a two-dimensional image (2D image) of the subject and generates point-cloud data including the 3D coordinates of the two or more points (Step S). Step Scorresponds to the generation step.
70 70 In a case in which the 2D image of the subject is a stereo image, one 2D image includes an image of the subject seen from a first viewpoint and an image of the subject seen from a second viewpoint different from the first viewpoint. The generation unitcalculates 3D coordinates corresponding to each pixel of the 2D image. The generation unitgenerates the point-cloud data including the 3D coordinates of the two or more points on the subject. The 3D coordinates of each of the two or more points are associated with a point on the 2D image in the point-cloud data. Specifically, the 3D coordinates in the point-cloud data are associated with a pixel on the 2D image. For example, the point-cloud data include the 3D coordinates and position information of the pixel on the 2D image.
70 70 70 The generation unitmay calculate 3D coordinates of two or more points on a subject by using two or more images and by applying structure-from-motion (SfM). The generation unitmay calculate 3D coordinates of two or more points on a subject by using two or more 2D images of the subject on which two or more stripe patterns having different spatial phases are projected and by applying a phase-shift method. The generation unitmay calculate 3D coordinates of two or more points on a subject by using one 2D image of the subject on which patterned light having randomly disposed bright and dark parts is emitted. A method of generating the point-cloud data is not limited to the above-described methods.
70 70 70 The generation unitmay generate a three-dimensional image (3D image) for displaying the point-cloud data on a display. The 3D image is an image of a three-dimensional shape (3D shape) indicated by the point-cloud data. The 3D image includes color data of each pixel. Each pixel of the 3D image is associated with the 3D coordinates. The generation unitmay generate a 3D image corresponding to each of the two or more different viewpoints. The generation unitmay generate an image of the feature regions. The image of the feature regions includes color data of each pixel. Each pixel of the image of the feature regions is associated with the 3D coordinates. Each pixel of the image of the feature regions is associated with a pixel of the 2D image or a pixel of the 3D image.
1 71 2 2 After Step S, the region detection unitdetects two or more feature regions on the subject on the basis of the point-cloud data (Step S). Step Scorresponds to the region detection step.
71 71 The region detection unitextracts features of the 3D shape of the subject by using the point-cloud data. The region detection unitassigns each point corresponding to the 3D coordinates in the point-cloud data to one of the two or more feature regions in accordance with the extracted features. The 3D shape of the subject has a common feature in one feature region. The feature of the 3D shape of the subject is different between two or more different feature regions. Only one point may be assigned to one feature region. When two or more points are assigned to one feature region, the two or more points meet a common condition indicating a feature of the 3D shape of the subject. The condition met by a point in a first feature region and the condition met by a point in a second feature region different from the first feature region are different from each other.
71 Segmentation is known as a simple method of classifying each point corresponding to the point-cloud data. For example, the region detection unitcan use Euclidean cluster extraction in segmentation. This is a function installed in a point cloud library (PCL) that is open source software.
71 The region detection unitdetermines a point within a predetermined distance of each point as a near-point by using this function. One point and its near-point are on the same object. For example, in a case in which the subject includes a first object and a second object apart from each other, each point corresponding to the point-cloud data is classified into any one of a point on the first object and a point on the second object. Each of the first object and the second object is a feature region (segment). In this case, two or more points in one feature region have a feature that the points are on the same object.
71 71 71 71 71 71 71 The region detection unitmay use a watershed algorithm, deep learning, or the like for segmentation. The region detection unitmay calculate a normal line perpendicular to the surface of the subject on the basis of the point-cloud data and may detect an edge or a step of the subject as a feature region on the basis of the change in the direction of the normal line. For example, the region detection unitmay detect a first feature region constituted by an edge or a step and may detect a second feature region constituted by a part other than the edge or the step. The region detection unitmay detect an edge of the subject by performing image processing on the 2D image of the subject. The region detection unitmay detect a feature region corresponding to the edge on the 2D image of the subject from the 3D shape of the subject indicated by the point-cloud data. The region detection unitmay detect a feature region on the basis of the brightness or the color of the 2D image of the subject or the 3D image of the point-cloud data. The region detection unitmay execute matching processing on a stereo image of the subject and may detect a feature region on the basis of the correlation value obtained in the matching processing.
3 FIG. 70 70 1 shows an example of the 3D image of the point-cloud data. The generation unitconverts the point-cloud data into mesh data and adds texture to the mesh data. The generation unitgenerates a 3D image Gby executing this processing.
4 FIG. 70 2 71 1 2 3 4 2 shows an example of an image of the feature regions. The generation unitgenerates an image Gof the feature regions on the basis of the information of the two or more feature regions detected by the region detection unit. A feature region R, a feature region R, a feature region R, and a feature region Rare displayed on the image G.
2 72 3 3 After Step S, the display control unitdisplays an image of the two or more feature regions on the display so that the two or more feature regions are visually distinguished from each other (Step S). Step Scorresponds to the image display step.
72 71 72 72 72 The display control unitoutputs the image of the two or more feature regions detected by the region detection unitto the display and causes the display to display the image. For example, the display control unitdisplays the image of the feature regions and the 2D image of the subject on the display. The display control unitmay display the image of the feature regions and the 3D image of the point-cloud data on the display. The display control unitmay display only the image of the feature regions on the display without displaying the 2D image of the subject and the 3D image of the point-cloud data on the display.
72 72 The image of the feature regions may be superimposed on the 2D image of the subject. The display control unitmay superimpose data of each pixel in the image of the feature regions on data of a pixel in the 2D image of the subject associated with a pixel in the image of the feature regions. The display control unitmay display the 2D image of the subject on which the image of the feature regions is superimposed on the display. In the 2D image, the unevenness of the surface of the subject and the positions of the feature regions are visible.
72 72 The image of the feature regions may be superimposed on the 3D image of the point-cloud data. The display control unitmay superimpose data of each pixel in the image of the feature regions on data of a pixel in the 3D image of the point-cloud data associated with a pixel in the image of the feature regions. The display control unitmay display the 3D image of the point-cloud data on which the image of the feature regions is superimposed on the display. In the 3D image, the unevenness of the surface of the subject and the positions of the feature regions are visible.
For example, the two or more feature regions are displayed in different colors in the image. Lines showing the borders between the two or more feature regions may be superimposed on the image. A character, a symbol, a mark, or the like showing each feature region may be displayed on each feature region. A user can visually distinguish the two or more feature regions on the image displayed on the display.
3 73 4 4 After Step S, the point input unitaccepts a point on the subject through an input device and generates point information indicating the accepted point (Step S). Step Scorresponds to the point input step.
73 73 A user inputs information of a point on the subject into an input device by operating the input device. The input device is a user interface. For example, the input device includes at least one of a button, a switch, a key, a mouse, a joystick, a touch pad, a track ball, and a touch panel. The input device outputs the information of a point input by a user. The point input unitaccepts a point on the subject by acquiring the information output from the input device. The point input unitgenerates point information indicating the position of a point in at least one of the 2D image of the subject, the 3D image of the point-cloud data, and the image of the feature regions.
73 73 73 73 73 In a case in which the point input unitaccepts a point on the 2D image of the subject, the point information includes the coordinates of a pixel corresponding to the point. The coordinates of the pixel are associated with the 3D coordinates in the point-cloud data. In a case in which the point input unitaccepts a point on the 3D image of the point-cloud data, the point information includes the 3D coordinates of the point. In a case in which the point input unitaccepts a point on the image of the feature regions, the point information includes the coordinates of a pixel corresponding to the point. The coordinates of the pixel are associated with the 3D coordinates in the point-cloud data. In a case in which the point input unitaccepts a point on the 2D image of the subject on which the image of the feature regions is superimposed, the point information includes the coordinates of a pixel corresponding to the point. The coordinates of the pixel are associated with the 3D coordinates in the point-cloud data. In a case in which the point input unitaccepts a point on the 3D image of the point-cloud data on which the image of the feature regions is superimposed, the point information includes the 3D coordinates of the point.
73 73 73 73 For example, in a case in which the distance-between-two-points measurement is executed, the point input unitgenerates point information indicating the position of each of two measurement points. In a case in which the line-based measurement is executed, the point input unitgenerates point information indicating the position of each of one measurement point and two reference points defining a reference line. In a case in which the plane-based measurement is executed, the point input unitgenerates point information indicating the position of each of one measurement point and three or more reference points defining a reference plane. In a case in which area measurement or perimeter measurement is executed, the point input unitgenerates point information indicating the position of each of three or more measurement points. In a case in which at least one of the line-based measurement, the plane-based measurement, the area measurement, and the perimeter measurement is executed, the point-cloud data include 3D coordinates of three or more points on the subject.
73 73 The point input unitmay accept only some of the two or more points used for measurement. For example, one or more points may be detected in the 2D image of the subject or the 3D image of the point-cloud data through image processing such as feature-point detection, and point information of the detected points may be generated. In this example, the one or more points accepted by the point input unitand the one or more points detected through image processing are used for measurement.
73 73 For example, in a case in which the line-based measurement is executed, two reference points may be detected through image processing and the point input unitmay accept one measurement point. Alternatively, the point input unitmay accept two reference points, and one measurement point may be detected through image processing. For example, one measurement point may be detected through image processing so that the three-dimensional distance (3D distance) between a reference line and the measurement point becomes the greatest.
73 73 For example, in a case in which the plane-based measurement is executed, three reference points may be detected through image processing and the point input unitmay accept one measurement point. Alternatively, the point input unitmay accept three reference points, and one measurement point may be detected through image processing. For example, one measurement point may be detected through image processing so that the 3D distance between a reference plane and the measurement point becomes the greatest.
73 73 73 The point accepted by the point input unitdoes not need to be one point corresponding to one pixel on the screen. The point accepted by the point input unitmay include a region having an arbitrary size. The point accepted by the point input unitmay include a region that can be designated in units of sub-pixels.
5 FIG. 4 FIG. 5 FIG. 4 2 2 72 2 73 73 shows an example of an image displayed on the display in Step S. The same image Gas that shown inis displayed on the display. For example, a cursor not shown inis displayed on the image G. A user inputs information indicating the position of the cursor into the input device. The display control unitdisplays the cursor at a position on the image Gcorresponding to the information. When a user performs a predetermined operation, a point corresponding to the position of the cursor is input into the input device. At this time, the point input unitaccepts the point. The cursor does not need to be displayed. In a case in which the input device and the display are constituted as a touch panel, a user inputs information of a point by touching the screen of the display. At this time, the point input unitaccepts the point.
5 FIG. 5 FIG. 5 FIG. 73 1 2 1 2 1 2 2 73 2 In the example shown in, a user inputs information of two points into the input device. The point input unitaccepts a point Pand a point Pshown in. The point Pand the point Pare in a feature region RE Marks indicating the point Pand the point Pare displayed on the image G. The mark indicating the point accepted by the point input unitdoes not need to be displayed. In the example shown in, four feature regions in the image Gare visually recognizable. Therefore, a user can easily designate a point in a feature region intended by the user.
6 FIG. 4 FIG. 6 FIG. 4 2 3 3 72 3 shows another example of an image displayed on the display in Step S. The same image Gas that shown inand a 2D image Gof the subject are displayed on the display. For example, a cursor not shown inis displayed on the 2D image G. A user inputs information indicating the position of the cursor into the input device. The display control unitdisplays the cursor at a position on the 2D image Gcorresponding to the information.
73 73 73 3 When a user performs a predetermined operation, a point corresponding to the position of the cursor is input into the input device. At this time, the point input unitaccepts the point. The cursor does not need to be displayed. In a case in which the input device and the display are constituted as a touch panel, a user inputs information of a point by touching the screen of the display. At this time, the point input unitaccepts the point. The point input unitidentifies a point in the point-cloud data associated with the point accepted on the 2D image Gand generates point information of the point.
6 FIG. 6 FIG. 6 FIG. 73 3 4 3 4 3 73 3 2 72 73 2 In the example shown in, a user inputs information of two points into the input device. The point input unitaccepts a point Pand a point Pshown in. Marks indicating the point Pand the point Pare displayed on the 2D image G. The mark indicating the point accepted by the point input unitdoes not need to be displayed. In the example shown in, a user designates a point on the 2D image Gby referring to the image G. Therefore, a user can easily designate a point on a region corresponding to a feature region intended by the user. The display control unitmay display a point corresponding to the point accepted by the point input uniton the image G.
6 FIG. 2 3 2 3 2 3 2 3 2 3 In the example shown in, the image Gand the 2D image Gare arranged in the horizontal direction of the display. The disposition of the image Gand the 2D image Gis not limited to this example. For example, the image Gand the 2D image Gmay be arranged in the vertical direction of the display. At this time, the image Gand the 2D image Gmay rotate by 90 degrees. Part of the image Gand part of the 2D image Gmay overlap each other.
4 74 5 5 5 After Step S, the measurement unitmeasures the size of the subject on the basis of the 3D coordinates of two or more points on the subject including the point indicated by the point information (Step S). Step Scorresponds to the measurement step. When Step Sis executed, the three-dimensional measurement is completed.
74 74 74 74 The measurement unitacquires the 3D coordinates of each of the two or more points on the subject. When the point information includes coordinates of a pixel in the 2D image of the subject, the measurement unitacquires 3D coordinates associated with the coordinates of the pixel from the point-cloud data. When the point information includes 3D coordinates of a point in the point-cloud data, the measurement unitacquires the 3D coordinates from the point information. When the point information includes coordinates of a pixel in the image of the feature regions, the measurement unitacquires 3D coordinates associated with the coordinates of the pixel from the point-cloud data.
74 74 74 74 74 74 74 The measurement unitmeasures the size of the subject on the basis of the 3D coordinates of the two or more points on the subject. For example, in the distance-between-two-points measurement, the measurement unitmeasures the 3D distance between the two measurement points indicated by the point information. In the line-based measurement, the measurement unitcalculates a reference line on the basis of the two reference points indicated by the point information and measures the 3D distance between the reference line and one measurement point indicated by the point information. In the plane-based measurement, the measurement unitcalculates a reference plane on the basis of the three reference points indicated by the point information and measures the 3D distance between the reference plane and one measurement point indicated by the point information. In the area measurement, the measurement unitcalculates the area of a region surrounded by the three or more measurement points indicated by the point information. In the perimeter measurement, the measurement unitcalculates the perimeter of a region surrounded by the three or more measurement points indicated by the point information. Measurement executed by the measurement unitis not limited to the above-described examples.
5 FIG. 6 FIG. In the example shown in, the image of the feature regions is displayed on the display, and a point on the image is input into the input device. In the example shown in, the image of the feature regions and the 2D image of the subject are displayed on the display, and a point on the 2D image of the subject is input into the input device. A combination of an image displayed on the display and an image used for inputting a point is not limited to these examples.
The 2D image of the subject may be displayed on the display, and a point on the 2D image may be input into the input device. The 3D image of the point-cloud data may be displayed on the display, and a point on the 3D image may be input into the input device.
The image of the feature regions and the 2D image of the subject may be displayed on the display, and a point on the image of the feature regions may be input into the input device. The image of the feature regions and the 3D image of the point-cloud data may be displayed on the display, and a point on the 3D image may be input into the input device. The image of the feature regions and the 3D image of the point-cloud data may be displayed on the display, and a point on the image of the feature regions may be input into the input device.
The 2D image of the subject on which the image of the feature regions is superimposed may be displayed on the display, and a point on the 2D image may be input into the input device. The 3D image of the point-cloud data on which the image of the feature regions is superimposed may be displayed on the display, and a point on the 3D image may be input into the input device.
The 3D image of the point-cloud data on which the image of the feature regions is superimposed and the 2D image of the subject may be displayed on the display, and a point on the 2D image may be input into the input device. The 3D image of the point-cloud data on which the image of the feature regions is superimposed and the 2D image of the subject may be displayed on the display, and a point on the 3D image may be input into the input device.
The 2D image of the subject on which the image of the feature regions is superimposed and the 3D image of the point-cloud data may be displayed on the display, and a point on the 3D image may be input into the input device. The 2D image of the subject on which the image of the feature regions is superimposed and the 3D image of the point-cloud data may be displayed on the display, and a point on the 2D image may be input into the input device.
7 7 In the first embodiment, the measurement devicedisplays the image of the two or more feature regions on the display. A user can easily designate a point in a feature region intended by the user by referring to the image displayed on the display. Therefore, the measurement devicecan support correct designation of a point on the subject.
7 FIG. 7 FIG. 1 FIG. 7 7 70 71 72 73 74 75 a a a a A second embodiment of the present invention will be described.shows a configuration of a measurement deviceaccording to the second embodiment. The measurement deviceshown inincludes a generation unit, a region detection unit, a display control unit, a point input unit, a measurement unit, and an information control unit. The same configuration as that shown inwill not be described.
7 FIG. 7 FIG. 7 FIG. Each unit shown inmay be constituted by at least one of a processor and a logic circuit. Each unit shown inmay include one or a plurality of processors. Each unit shown inmay include one or a plurality of logic circuits.
7 7 a 1 FIG. A computer of the measurement devicemay read a program and execute the read program. The program is realized as with the program executed by the computer of the measurement deviceshown in.
71 71 71 71 a a a The region detection unitdetects one or more feature regions on a subject on the basis of the point-cloud data (region detection step). The region detection unitin the first embodiment detects two or more feature regions, but the region detection unitin the second embodiment may detect one feature region. For example, in a case in which the subject is constituted by only one object, the region detection unitdetects one feature region. The 3D shape of the subject has a common feature in one feature region. The feature of the 3D shape of the subject is different between two or more different feature regions. Only one point may be assigned to one feature region. When two or more points are assigned to one feature region, the two or more points meet a common condition indicating a feature of the 3D shape of the subject. The condition met by a point in a first feature region and the condition met by a point in a second feature region different from the first feature region are different from each other.
72 a The display control unitdisplays one of a 3D image of the point-cloud data, a 2D image of the subject, and an image of one or more feature regions on the display (image display step). The 3D image of the point-cloud data is an image of the 3D shape indicated by the point-cloud data.
75 75 73 75 75 73 72 72 75 a a The information control unitexecutes one of first control and second control (control step). The information control unitcontrols point information in the first control so that the point information indicates a point in one of the one or more feature regions. The point information indicates a point accepted by the point input unit. The information control unitdisplays a mark on one of the 3D image of the point-cloud data, the 2D image of the subject, and the image of the one or more feature regions in the second control. The mark is an icon, a pointer, or the like. For example, the mark is a cursor. The information control unitrestricts the position of the mark to a position in a region corresponding to one of the one or more feature regions. The point input unitaccepts a point corresponding to the position of the mark. The display control unitmay execute the second control. In this case, the display control unithas the function of the information control unit.
8 FIG. 9 FIG. 8 FIG. 2 FIG. Three-dimensional measurement in the second embodiment will be described by usingand.shows a procedure of the three-dimensional measurement. The same processing as that shown inwill not be described.
1 71 2 2 71 a a a a After Step S, the region detection unitdetects one or more feature regions on a subject on the basis of the point-cloud data (Step S). Step Scorresponds to the region detection step. The region detection unitcan detect a feature region by using the same method as that shown in the first embodiment.
2 72 3 3 a a a a After Step S, the display control unitdisplays one of the 3D image of the point-cloud data, the 2D image of the subject, and the image of the one or more feature regions on the display (Step S). Step Scorresponds to the image display step.
72 72 72 a a a For example, the display control unitdisplays the 3D image of the point-cloud data on the display without displaying the 2D image of the subject and the image of the feature regions. The display control unitmay display the 2D image of the subject on the display without displaying the 3D image of the point-cloud data and the image of the feature regions. The display control unitmay display the image of the feature regions on the display without displaying the 3D image of the point-cloud data and the 2D image of the subject.
3 72 72 72 72 a a a a a The image displayed on the display in Step Shas only to be one of the 3D image of the point-cloud data, the 2D image of the subject, and the image of the feature regions. However, the display control unitmay display two or more of the 3D image of the point-cloud data, the 2D image of the subject, and the image of the feature regions on the display. For example, the display control unitmay display the 3D image of the point-cloud data and the image of the feature regions on the display. The display control unitmay display the 2D image of the subject and the image of the feature regions on the display. The display control unitmay display the 3D image of the point-cloud data on which the image of the feature regions is superimposed on the display or may display the 2D image of the subject on which the image of the feature regions is superimposed on the display.
3 73 4 4 75 75 4 6 6 a After Step S, the point input unitaccepts a point on the subject through the input device and generates point information indicating the accepted point in Step S. After Step S, the information control unitexecutes the first control. In other words, the information control unitcontrols the point information so that the point information generated in Step Sindicates a point in one of the one or more feature regions (Step S). Step Scorresponds to the control step.
6 5 For example, only the point information indicating a point in one feature region becomes valid, and the 3D coordinates of the point are used for measurement. Alternatively, the point information indicating a point outside one feature region is updated to point information indicating a point in the feature region. Specific examples of the first control will be described in fourth to sixth embodiments. After Step S, Step Sis executed.
9 FIG. 2 FIG. 8 FIG. shows another procedure of the three-dimensional measurement. The same processing as that shown inandwill not be described.
3 75 75 3 3 3 75 7 7 a a a a After Step S, the information control unitexecutes the second control. In other words, the information control unitdisplays a mark on one of the 3D image of the point-cloud data, the 2D image of the subject, and the image of the one or more feature regions. When the 3D image of the point-cloud data is displayed on the display in Step S, the mark is displayed on the 3D image. When the 2D image of the subject is displayed on the display in Step S, the mark is displayed on the 2D image. When the image of the feature regions is displayed on the display in Step S, the mark is displayed on the image. The information control unitrestricts the position of the mark to a position in a region corresponding to one of the one or more feature regions (Step S). Step Scorresponds to the control step.
72 7 a For example, the mark moves only in a region corresponding to one feature region. Alternatively, when the mark moves to the outside of a region corresponding to one feature region, the position of the mark is forcibly changed to a position in the feature region. Specific examples of the second control will be described in fifth to seventh embodiments. The display control unitmay execute Step S.
7 73 4 After Step S, the point input unitaccepts a point corresponding to the position of the mark in Step S.
8 FIG. 9 FIG. 2 1 2 73 4 75 7 a a The order of processing in the three-dimensional measurement is not limited to that shown inand. For example, the 3D image of the point-cloud data may be displayed on the display before Step Sis executed. The 2D image of the subject may be displayed on the display before Step Sor Step Sis executed. After the point input unitaccepts the point in Step S, the information control unitmay execute the second control on the basis of the point in Step S.
7 7 a a In the second embodiment, the measurement deviceexecutes one of the first control and the second control. In the first control, the point information is controlled so that the point information indicates a position in one of the one or more feature regions. In the second control, the position of the mark on the image is restricted to a position in a region corresponding to one of the one or more feature regions. The 3D coordinates of a point in one feature region are used for measurement. Therefore, the measurement devicecan support correct designation of a point on the subject.
A third embodiment of the present invention will be described. Hereinafter, an example in which the measurement device is an endoscope device will be described. The measurement device has only to be a device having a measurement function and is not limited to an endoscope device. The measurement device may be built-in equipment mounted on a specific device or a system. The measurement device may operate in a cloud environment. A subject is an industrial product.
10 FIG. 11 FIG. 1 1 1 2 shows an external appearance of an endoscope deviceaccording to the third embodiment.shows an internal configuration of the endoscope device. The endoscope deviceimages a subject and generates an image. In order to observe various subjects, an inspector can perform replacement of an optical adaptor mounted at a distal end of an insertion unit, selection of a built-in video-processing program, and addition of a video-processing program.
1 2 3 4 5 10 FIG. The endoscope deviceshown inincludes the insertion unit, a main body unit, an operation unit, and a display unit.
2 2 20 2 3 20 2 20 2 3 2 4 1 5 2 The insertion unitis inserted into the inside of a subject. The insertion unithas a long and thin bendable tube shape from the distal endto a base end portion. The insertion unitimages a subject and outputs an imaging signal to the main body unit. An optical adapter is mounted on the distal endof the insertion unit. For example, a single-eye optical adapter is mounted on the distal endof the insertion unit. The main body unitis a control device including a housing unit that houses the insertion unit. The operation unitaccepts an operation for the endoscope devicefrom a user. The display unitincludes a display screen and displays an image of a subject acquired by the insertion unit, an operation menu, and the like on the display screen.
4 4 5 5 4 5 The operation unitis a user interface (input device). For example, the operation unitis at least one of a button, a switch, a key, a mouse, a joystick, a touch pad, a track ball, and a touch panel. The display unitis a monitor (display) such as a liquid crystal display (LCD). The display unitmay be a touch panel. In such a case, the operation unitand the display unitare integrated.
3 8 9 10 8 2 28 20 2 28 28 9 28 28 9 9 28 9 11 FIG. The main body unitshown inincludes an endoscope unit, a camera control unit (CCU), and a control device. The endoscope unitincludes a light source device and a bending device not shown in the drawing. The light source device supplies illumination light that is necessary for observation. The bending device bends a bending mechanism built in the insertion unit. An imaging deviceis built in the distal endof the insertion unit. The imaging deviceis an image sensor. The imaging devicephoto-electrically converts an optical image of a subject formed by an optical adaptor and generates an imaging signal. The CCUdrives the imaging device. The imaging signal output from the imaging deviceis input into the CCU. The CCUperforms preprocessing including amplification, noise elimination, and the like for the imaging signal acquired by the imaging device. The CCUconverts the processed imaging signal into a video signal such as an NTSC signal.
10 12 13 14 15 16 17 18 The control deviceincludes a video-signal-processing circuit, a read-only memory (ROM), a random-access memory (RAM), a card interface, an external device interface, a control interface, and a central processing unit (CPU).
12 9 12 12 9 18 12 5 12 9 18 The video-signal-processing circuitperforms predetermined video processing on the video signal output from the CCU. For example, the video-signal-processing circuitperforms video processing related to improvement of visibility. For example, the video processing is color reproduction, gray scale correction, noise suppression, contour enhancement, and the like. The video-signal-processing circuitcombines the video signal output from the CCUand a graphic image signal generated by the CPU. The graphic image signal includes an image of the operation screen, measurement information, and the like. The measurement information includes a 3D image of the point-cloud data, an image of feature regions, a measurement result, or the like. The video-signal-processing circuitoutputs a combined video signal to the display unit. In addition, the video-signal-processing circuitoutputs image data on the basis of the video signal output from the CCUto the CPU.
13 18 1 14 18 1 18 1 13 The ROMis a nonvolatile recording medium on which a program for the CPUto control the operation of the endoscope deviceis recorded. The RAMis a volatile recording medium that temporarily stores information used by the CPUfor controlling the endoscope device. The CPUcontrols the operation of the endoscope deviceon the basis of the program recorded on the ROM.
42 15 15 42 10 15 1 42 A memory cardthat is a removable recording medium is connected to the card interface. The card interfaceinputs control-processing information, image information, and the like stored on the memory cardinto the control device. In addition, the card interfacerecords control-processing information, image information, and the like generated by the endoscope deviceon the memory card.
16 41 16 16 41 41 41 41 1 An external device such as a USB device is connected to the external device interface. For example, a personal computer (PC)is connected to the external device interface. The external device interfacetransmits information to the PCand receives information from the PC. In this way, the monitor of the PCcan display information. In addition, by inputting an instruction into the PC, a user can perform an operation related to control of the endoscope device.
17 4 8 9 17 18 4 17 8 17 28 9 The control interfaceperforms communication with the operation unit, the endoscope unit, and the CCUfor operation control. The control interfacenotifies the CPUof an instruction input into the operation unitby a user. The control interfaceoutputs control signals used for controlling the light source device and the bending device to the endoscope unit. The control interfaceoutputs a control signal used for controlling the imaging deviceto the CCU.
18 1 41 41 1 1 1 41 1 A program executed by the CPUmay be recorded on a computer-readable recording medium. The program recorded on this recording medium may be read and executed by a computer other than the endoscope device. For example, the program may be read and executed by the PC. The PCmay control the endoscope deviceby transmitting control information used for controlling the endoscope deviceto the endoscope devicein accordance with the program. Alternatively, the PCmay acquire a video signal from the endoscope deviceand may process the acquired video signal.
1 28 18 28 28 28 18 12 As described above, the endoscope deviceincludes the imaging deviceand the CPU. The imaging deviceimages a subject and generates an imaging signal. The imaging signal includes an image of the subject. Accordingly, the imaging deviceacquires the image of the subject generated by capturing the image of the subject. The image acquired by the imaging deviceis input into the CPUvia the video-signal-processing circuit.
28 41 1 The imaging devicehas a function of an image acquisition unit that acquires an image of a subject. The image acquisition unit may be an image input device. For example, in a case in which the PCoperates as a measurement device, the image acquisition unit is a communication interface (communicator) that performs communication with the endoscope device. The image acquisition unit may be a wireless communicator. The image acquisition unit may be a reading circuit that reads an image from a recording medium on which the image is recorded.
12 FIG. 12 FIG. 18 18 180 181 182 183 184 185 186 187 18 shows a functional configuration of the CPU. The CPUhas functional units including a control unit, a generation unit, a region detection unit, a display control unit, a cursor calculation unit, a point input unit, a region selection unit, and a measurement unit. At least one of the blocks shown inmay be constituted by a different circuit from the CPU.
12 FIG. 12 FIG. 12 FIG. Each unit shown inmay be constituted by at least one of a processor and a logic circuit. Each unit shown inmay include one or a plurality of processors. Each unit shown inmay include one or a plurality of logic circuits.
180 12 12 FIG. The control unitacquires a 2D image (image data) of a subject from the video-signal-processing circuitand controls processing executed by each unit shown in.
181 70 181 181 181 181 5 1 FIG. The generation unithas the same function as that of the generation unitshown in. The generation unitcalculates 3D coordinates of two or more points on a subject on the basis of a 2D image of the subject and generates point-cloud data including the 3D coordinates of the two or more points (generation step). The generation unitcan generate the point-cloud data by using the same method as that shown in the first embodiment. In addition, the generation unitgenerates an image of feature regions. The generation unitmay generate a 3D image for displaying the point-cloud data on the display unit.
182 71 182 182 1 FIG. The region detection unithas the same function as that of the region detection unitshown in. The region detection unitdetects two or more feature regions on the subject on the basis of the point-cloud data (region detection step). The region detection unitcan detect feature regions by using the same method as that shown in the first embodiment.
183 28 181 5 183 5 183 5 183 5 183 5 The display control unitdisplays the 2D image of the subject acquired by the imaging deviceand the image of the two or more feature regions generated by the generation uniton the display unit. The display control unitdisplays the image of the two or more feature regions on the display unitso that the two or more feature regions are visually distinguished from each other (image display step). The display control unitmay display the 3D image of the point-cloud data on the display unit. The display control unitmay display the 2D image of the subject on which the image of the feature regions is superimposed on the display unit. The display control unitmay display the 3D image of the point-cloud data on which the image of the feature regions is superimposed on the display unit.
183 12 183 12 12 5 5 12 183 5 12 5 12 For example, the display control unitcontrols processing executed by the video-signal-processing circuit. The display control unitcauses the video signal processed by the video-signal-processing circuitto be output from the video-signal-processing circuitto the display unit. The video signal includes color data of each pixel of the 2D image of the subject. The display unitdisplays the 2D image of the subject on the basis of the video signal output from the video-signal-processing circuit. Alternatively, the display control unitoutputs a video signal for displaying the image of the feature regions to the display unitvia the video-signal-processing circuit. The video signal includes color data of each pixel of the image of the feature regions. The display unitdisplays the image of the feature regions on the basis of the video signal output from the video-signal-processing circuit.
183 183 12 12 9 18 12 5 5 Specifically, the display control unitgenerates a graphic image signal for displaying the image of the feature regions. The display control unitoutputs the generated graphic image signal to the video-signal-processing circuit. The video-signal-processing circuitcombines the video signal output from the CCUand the graphic image signal output from the CPU. The video-signal-processing circuitoutputs the combined video signal to the display unit. The display unitdisplays the 2D image of the subject and the image of the feature regions.
183 5 12 5 12 The display control unitmay output a 3D video signal to the display unitvia the video-signal-processing circuit. The 3D video signal includes color data of each pixel of the 3D image of the point-cloud data. The display unitmay display the 3D image of the point-cloud data on the basis of the 3D video signal output from the video-signal-processing circuit.
183 5 183 The display control unitdisplays various kinds of information on the display unit. In other words, the display control unitdisplays various kinds of information on an image. Various kinds of information include a cursor, a measurement result, or the like. The cursor is a mark used by a user to designate a specific point on an image.
183 183 12 12 9 18 12 5 5 For example, the display control unitgenerates a graphic image signal of various kinds of information. The display control unitoutputs the generated graphic image signal to the video-signal-processing circuit. The video-signal-processing circuitcombines the video signal output from the CCUand the graphic image signal output from the CPU. In this way, various kinds of information are superimposed on an image. The video-signal-processing circuitoutputs the combined video signal to the display unit. The display unitdisplays an image on which various kinds of information are superimposed.
4 4 4 4 4 17 17 18 184 4 183 184 A user inputs position information of a cursor into the operation unitby operating the operation unit. The operation unitoutputs the position information input into the operation unitby a user. The position information input into the operation unitis input into the control interfacethat is an input unit. The position information is output from the control interfaceto the CPU. The cursor calculation unitcalculates a position on an image on the basis of the position information input into the operation unit. The display control unitdisplays a cursor at the position calculated by the cursor calculation unit.
185 73 185 4 185 5 4 4 5 185 185 1 FIG. The point input unithas the same function as that of the point input unitshown in. The point input unitaccepts a position on a subject through the operation unit. For example, a user moves the cursor to an intended position on the image and performs a predetermined operation. At this time, the point input unitaccepts a point corresponding to the position. In a case in which the display unitand the operation unitare constituted as a touch panel, a user inputs a point on the image into the operation unitby touching the screen of the display unit. The point input unitaccepts the point. The point input unitgenerates point information indicating the accepted point (point input step).
185 185 185 185 185 In a case in which the point input unitaccepts a point on the 2D image of the subject, the point information includes coordinates of a pixel corresponding to the point. The coordinates of the pixel are associated with the 3D coordinates in the point-cloud data. In a case in which the point input unitaccepts a point on the image of the feature regions, the point information includes coordinates of a pixel corresponding to the point. The coordinates of the pixel are associated with the 3D coordinates in the point-cloud data. In a case in which the point input unitaccepts a point on the 3D image of the point-cloud data, the point information includes the 3D coordinates of the point. In a case in which the point input unitaccepts a point on the 2D image of the subject on which the image of the feature regions is superimposed, the point information includes coordinates of a pixel corresponding to the point. The coordinates of the pixel are associated with the 3D coordinates in the point-cloud data. In a case in which the point input unitaccepts a point on the 3D image of the point-cloud data on which the image of the feature regions is superimposed, the point information includes the 3D coordinates of the point.
185 The point input unitaccepts a reference point and a measurement point. The reference point indicates a reference position for calculating a reference line in the line-based measurement or a reference plane in the plane-based measurement. The measurement point indicates a position at which the size of the subject is measured.
185 185 185 185 185 185 185 185 185 185 185 14 When the point input unitaccepts a point on the 2D image of the subject, the point input unitsets the accepted point to the 2D image. When the point input unitaccepts a point on the 3D image of the point-cloud data, the point input unitsets the accepted point to the 3D image. When the point input unitaccepts a point on the image of the feature regions, the point input unitsets the accepted point to the image. When the point input unitaccepts a point on the 2D image of the subject on which the image of the feature regions is superimposed, the point input unitsets the accepted point to the 2D image. When the point input unitaccepts a point on the 3D image of the point-cloud data on which the image of the feature regions is superimposed, the point input unitsets the accepted point to the 3D image. The position information of the point set by the point input unitis held on the RAM. The point is set by associating the point with a specific image.
186 182 4 186 186 186 The region selection unitselects one of the two or more feature regions detected by the region detection unitas a selected region (region selection step). For example, a user moves the cursor to an intended position on the 2D image of the subject or the 3D image of the point-cloud data by operating the operation unit. The position is in a region corresponding to a feature region in which a user intends to designate a point. When the cursor is displayed at the position, a user performs a predetermined operation. At this time, the region selection unitcalculates a position on the image of the feature regions corresponding to the position at which the cursor is displayed. Each pixel of the 2D image and each pixel of the 3D image are associated with a pixel of the image of the feature regions. Therefore, the region selection unitcan calculate a position on the image of the feature regions corresponding to the position on the 2D image or the 3D image. The region selection unitselects a feature region including the calculated position.
4 186 A user may move the cursor to an intended position on the image of the feature regions by operating the operation unit. The position is in a feature region in which a user intends to designate a point. When the cursor is displayed at the position, a user performs a predetermined operation. At this time, the region selection unitselects the feature region including the position.
5 4 4 5 In a case in which the display unitand the operation unitare constituted as a touch panel, a user may input a position on an image into the operation unitby touching the screen of the display unit.
186 186 186 186 186 186 The region selection unitmay select a feature region that meets a condition set in advance. For example, the region selection unitmay select a feature region having the largest area. The region selection unitmay calculate the brightness of an image for each feature region and may select a feature region of which the brightness is in a predetermined range. The region selection unitmay calculate the contrast for each feature region and may select a feature region of which the contrast is the highest. The region selection unitmay select a feature region on the basis of the measurement mode. For example, in a case in which the measurement mode is the line-based measurement, the region selection unitmay select a feature region including an edge of the subject. A method of selecting a feature region is not limited to the above-described methods.
187 74 187 187 187 187 187 1 FIG. The measurement unithas the same function as that of the measurement unitshown in. The measurement unitmeasures the size of the subject on the basis of the 3D coordinates of two or more points on the subject including the point indicated by the point information (measurement step). Specifically, the measurement unitacquires 3D coordinates of each of the two or more points on the subject. When the point information includes coordinates of a pixel in the 2D image of the subject, the measurement unitacquires 3D coordinates associated with the coordinates of the pixel from the point-cloud data. When the point information includes 3D coordinates of a point in the point-cloud data, the measurement unitacquires the 3D coordinates from the point information. When the point information includes coordinates of a pixel in the image of the feature regions, the measurement unitacquires the 3D coordinates associated with the coordinates of the pixel from the point-cloud data.
187 187 187 187 185 The measurement unitexecutes measurement by using the acquired 3D coordinates. The measurement unitexecutes at least one of the distance-between-two-points measurement, the line-based measurement, the plane-based measurement, the area measurement, and the perimeter measurement. Measurement executed by the measurement unitis not limited to these. The measurement unitmay execute measurement by using 3D coordinates of one or more points accepted by the point input unitand 3D coordinates of one or more points detected in image processing.
13 FIG. 13 FIG. Three-dimensional measurement in the third embodiment will be described by using.shows a procedure of the three-dimensional measurement.
181 101 101 The generation unitcalculates 3D coordinates of two or more points on a subject on the basis of the 2D image of the subject and generates point-cloud data including the 3D coordinates of the two or more points (Step S). Step Scorresponds to the generation step.
101 182 102 102 182 182 103 102 103 After Step S, the region detection unitextracts a feature of the 3D shape of the subject by using the point-cloud data (Step S). After Step S, the region detection unitassigns each point corresponding to the 3D coordinates in the point-cloud data to each of two or more feature regions in accordance with the extracted feature. In this way, the region detection unitdivides the 3D shape of the subject into the two or more feature regions (Step S). Step Sand Step Scorrespond to the region detection step.
103 183 5 104 104 After Step S, the display control unitdisplays the 2D image of the subject and the image of the feature regions on the display unit(Step S). Step Scorresponds to the image display step.
14 FIG. 5 104 11 12 5 11 12 13 14 12 11 12 13 14 5 shows an example of an image displayed on the display unitin Step S. A 2D image Gof a subject and an image Gof feature regions are displayed on the display unit. A feature region R, a feature region R, a feature region R, and a feature region Rare displayed on the image G. For example, the feature region R, the feature region R, the feature region R, and the feature region Rare displayed in different colors in the image. A user can visually distinguish the two or more feature regions on the image displayed on the display unitfrom each other.
11 12 13 14 12 Lines showing the borders between the feature region R, the feature region R, the feature region R, and the feature region Rmay be superimposed on the image G. A character, a symbol, a mark, or the like showing each feature region may be displayed on each feature region.
14 FIG. 11 12 5 11 12 11 12 5 11 12 11 12 In the example shown in, the 2D image Gand the image Gare arranged in the horizontal direction of the display unit. The disposition of the 2D image Gand the image Gis not limited to this example. For example, the 2D image Gand the image Gmay be arranged in the vertical direction of the display unit. At this time, the 2D image Gand the image Gmay rotate by 90 degrees. Part of the 2D image Gand part of the image Gmay overlap each other.
104 183 105 4 184 4 183 184 After Step S, the display control unitdisplays a cursor on the 2D image of the subject (Step S). A user can move the cursor on the 2D image by operating the operation unit. The cursor calculation unitcalculates a position on the 2D image on the basis of the position information input into the operation unit. The display control unitdisplays a cursor at the position calculated by the cursor calculation unit, thus updating the cursor.
15 FIG. 14 FIG. 14 FIG. 5 105 11 12 5 11 11 11 11 11 shows an example of an image displayed on the display unitin Step S. The same 2D image Gas that shown inand the same image Gas that shown inare displayed on the display unit. A cursor Cis displayed on the 2D image G. For example, the cursor Cis first displayed at a predetermined position on the 2D image G. The predetermined position is the center or the like of the 2D image G.
105 186 182 106 106 After Step S, the region selection unitselects one of the two or more feature regions detected by the region detection unitas a selected region (Step S). Step Scorresponds to the region selection step.
186 186 For example, the region selection unitselects a feature region corresponding to a region including the position of the cursor. Alternatively, the region selection unitselects a feature region that meets a predetermined condition.
16 FIG. 14 FIG. 14 FIG. 15 FIG. 5 106 11 12 5 11 11 11 11 186 12 11 186 shows an example of an image displayed on the display unitin Step S. The same 2D image Gas that shown inand the same image Gas that shown inare displayed on the display unit. The same cursor Cas that shown inis displayed on the 2D image G. A user moves the cursor Con the 2D image G. When a user performs a predetermined operation, the region selection unitcalculates a position on the image Gcorresponding to the position of the cursor C. The region selection unitselects a feature region including the calculated position.
16 FIG. 11 11 11 186 11 In the example shown in, the cursor Cis at a position on the 2D image Gcorresponding to a position in the feature region Rwhen a user performs a predetermined operation. Therefore, the region selection unitselects the feature region Ras a selected region.
183 11 106 106 11 183 12 13 14 16 FIG. The display control unithighlights the feature region R, which is the selected region, in Step S. Step Scorresponds to the image display step. In the example shown in, the feature region Ris surrounded by a thick line. The line is included in the graphic image signal generated by the display control unit. The feature region R, the feature region R, and the feature region Rare not highlighted.
11 11 11 A color or a pattern that draws a user's attention may be added to the feature region R. A character, a symbol, a mark, or the like showing the selected region may be displayed on the feature region R. A method of highlighting the selected region is not limited to the above-described methods. Since the selected region is highlighted, a user can easily confirm a region on the 2D image Gon which a point should be designated.
183 11 106 12 13 14 11 The display control unitmay display only the feature region R, which is the selected region, in Step S. In this case, the feature region R, the feature region R, and the feature region Rare not displayed. Since only the selected region is displayed, a user can easily confirm a region on the 2D image Gon which a point should be designated.
183 106 106 11 12 13 14 12 13 14 11 The display control unitmay display the selected region in a first color in Step Sand may display a feature region different from the selected region in a second color different from the first color in Step S. For example, the feature region Ris displayed in a color such as red, and the feature region R, the feature region R, and the feature region Rare displayed in a color such as gray. The feature region R, the feature region R, and the feature region Rare displayed in the same color. Since the selected region and each of the other feature regions are displayed in different colors, a user can easily confirm a region on the 2D image Gon which a point should be designated.
106 184 4 107 107 183 184 108 After Step S, the cursor calculation unitcalculates a position on the 2D image of the subject on the basis of the position information input into the operation unit(Step S). After Step S, the display control unitdisplays the cursor at the position calculated by the cursor calculation unit, thus updating the cursor (Step S).
108 185 4 109 109 After Step S, the point input unitaccepts one point on the 2D image of the subject through the operation unitand generates point information indicating the accepted point (Step S). Step Scorresponds to the point input step.
17 FIG. 14 FIG. 14 FIG. 15 FIG. 5 109 11 12 5 11 11 11 11 185 shows an example of an image displayed on the display unitin Step S. The same 2D image Gas that shown inand the same image Gas that shown inare displayed on the display unit. The same cursor Cas that shown inis displayed on the 2D image G. A user moves the cursor Cto an intended position on the 2D image Gand performs a predetermined operation. At this time, the point input unitaccepts a point corresponding to the position.
17 FIG. 17 FIG. 185 11 11 11 185 11 11 12 183 185 12 In the example shown in, the point input unitaccepts a point P. A mark showing the point Pis displayed on the 2D image G. The mark indicating the point accepted by the point input unitdoes not need to be displayed. In the example shown in, a user designates a point on the 2D image Gcorresponding to a position in the feature region Rby referring to the image G. Therefore, a user can easily designate a point on a region corresponding to a feature region intended by the user. The display control unitmay display a point corresponding to the point accepted by the point input uniton the image G.
185 4 4 185 185 After the point input unitaccepts a point, the point may be corrected. For example, a user may determine that the point designated by the user is wrong and may input an instruction to correct the point into the operation unit. In this case, a user inputs a new point into the operation unit. The point input unitaccepts the instruction and the new point. The point input unitupdates the point, which is directed to be corrected, with the new point.
109 180 185 110 After Step S, the control unitdetermines whether or not the point input unithas accepted a necessary number of points (Step S). Two measurement points are necessary for the distance-between-two-points measurement. Three points including two reference points and one measurement point are necessary for the line-based measurement. Four or more points including three or more reference points and one measurement point are necessary for the plane-based measurement. Three or more measurement points are necessary for the area measurement and the perimeter measurement.
180 185 110 107 108 109 185 When the control unitdetermines that the point input unithas not accepted the necessary number of points in Step S, Step Sis executed. Thereafter, Step Sand Step Sare executed again. Therefore, the point input unitaccepts two or more points.
18 FIG. 14 FIG. 14 FIG. 15 FIG. 18 FIG. 5 109 11 12 5 11 11 185 11 11 11 12 13 185 11 109 185 12 109 13 109 shows an example of an image displayed on the display unitafter Step Sis executed twice or more. The same 2D image Gas that shown inand the same image Gas that shown inare displayed on the display unit. The same cursor Cas that shown inis displayed on the 2D image G. In, an example of the line-based measurement is shown, and the point input unitaccepts three points on the 2D image G. After the point Pis designated, a user moves the cursor Cand sequentially designates a point Pand a point P. The point input unitaccepts the point Pin Step Sfor the first time. Thereafter, the point input unitaccepts the point Pin Step Sfor the second time and accepts the point Pin Step Sfor the third time.
185 13 180 185 110 185 13 180 185 110 Before the point input unitaccepts the point P, the control unitdetermines that the point input unithas not accepted the necessary number of points in Step S. After the point input unitaccepts the point P, the control unitdetermines that the point input unithas accepted the necessary number of points in Step S.
180 185 110 187 111 111 111 When the control unitdetermines that the point input unithas accepted the necessary number of points in Step S, the measurement unitmeasures the size of the subject on the basis of the 3D coordinates of two or more points on the subject including the point indicated by the point information (Step S). Step Scorresponds to the measurement step. When Step Sis executed, the three-dimensional measurement is completed.
111 183 5 183 After Step Sis executed, the display control unitmay display a measurement result on the display unit. The measurement result is included in the graphic image signal generated by the display control unit.
19 FIG. 19 FIG. 14 FIG. 15 FIG. 19 FIG. 19 FIG. 5 104 12 5 11 12 12 4 106 186 106 11 11 185 12 11 14 15 16 185 14 15 16 shows another example of an image displayed on the display unit. In the example shown in, the 2D image of the subject is not displayed in Step S. The same image Gas that shown inis displayed on the display unit. The same cursor Cas that shown inis displayed on the image G. When a position on the image Gis accepted through the operation unitin Step S, the region selection unitselects a feature region corresponding to the position in Step S. In, an example in which the feature region Ris selected is shown, and the feature region Ris highlighted. In, an example of the line-based measurement is shown, and the point input unitaccepts three points on the image G. A user moves the cursor Cand sequentially designates a point P, a point P, and a point P. The point input unitaccepts the point P, the point P, and the point P.
13 18 FIGS.to 19 FIG. 5 5 5 In the examples shown in, the 2D image of the subject and the image of the feature regions are displayed on the display unit, and the 2D image is used for inputting a point. In the example shown in, the image of the feature regions is displayed on the display unit, and the image is used for inputting a point. A combination of an image displayed on the display unitand an image used for inputting a point is not limited to these examples.
5 4 5 4 The 2D image of the subject may be displayed on the display unit, and a point on the 2D image may be input into the operation unit. The 3D image of the point-cloud data may be displayed on the display unit, and a point on the 3D image may be input into the operation unit.
5 4 5 4 5 4 The image of the feature regions and the 2D image of the subject may be displayed on the display unit, and a point on the image of the feature regions may be input into the operation unit. The image of the feature regions and the 3D image of the point-cloud data may be displayed on the display unit, and a point on the 3D image may be input into the operation unit. The image of the feature regions and the 3D image of the point-cloud data may be displayed on the display unit, and a point on the image of the feature regions may be input into the operation unit.
5 4 5 4 The 2D image of the subject on which the image of the feature regions is superimposed may be displayed on the display unit, and a point on the 2D image may be input into the operation unit. The 3D image of the point-cloud data on which the image of the feature regions is superimposed may be displayed on the display unit, and a point on the 3D image may be input into the operation unit.
5 4 5 4 The 3D image of the point-cloud data on which the image of the feature regions is superimposed and the 2D image of the subject may be displayed on the display unit, and a point on the 2D image may be input into the operation unit. The 3D image of the point-cloud data on which the image of the feature regions is superimposed and the 2D image of the subject may be displayed on the display unit, and a point on the 3D image may be input into the operation unit.
5 4 5 4 The 2D image of the subject on which the image of the feature regions is superimposed and the 3D image of the point-cloud data may be displayed on the display unit, and a point on the 3D image may be input into the operation unit. The 2D image of the subject on which the image of the feature regions is superimposed and the 3D image of the point-cloud data may be displayed on the display unit, and a point on the 2D image may be input into the operation unit.
5 4 4 5 105 107 108 18 184 In a case in which the display unitand the operation unitare constituted as a touch panel, a user can input a point into the operation unitby touching the screen of the display unit. Therefore, the cursor does not need to be displayed. Accordingly, Step S, Step S, and Step Sdo not need to be executed. The CPUdoes not need to have the function of the cursor calculation unit.
16 FIG. 106 18 186 The feature regions shown indo not need to be highlighted. Accordingly, Step Sdoes not need to be executed. The CPUdoes not need to have the function of the region selection unit.
13 FIG. 5 101 102 103 5 102 103 The order of processing in the three-dimensional measurement is not limited to that shown in. For example, the 2D image of the subject may be displayed on the display unitbefore Step S, Step S, or Step Sis executed. In a case in which the 3D image of the point-cloud data is used instead of the 2D image of the subject, the 3D image may be displayed on the display unitbefore Step Sor Step Sis executed.
1 5 5 1 In the third embodiment, the endoscope devicedisplays the image of the two or more feature regions on the display unit. A user can easily designate a point in a feature region intended by the user by referring to the image displayed on the display unit. Therefore, the endoscope devicecan support correct designation of a point on the subject.
A fourth embodiment of the present invention will be described. In the fourth embodiment, only a point in one selected region is valid, and the 3D coordinates of the valid point are used for measurement.
18 18 18 18 180 181 182 183 184 185 186 187 188 18 12 FIG. 20 FIG. 20 FIG. 20 FIG. 12 FIG. a a a a In the fourth embodiment, the CPUshown inis changed to a CPUshown in.shows a functional configuration of the CPU. The CPUhas functional units including a control unit, a generation unit, a region detection unit, a display control unit, a cursor calculation unit, a point input unit, a region selection unit, the measurement unit, and an information control unit. At least one of the blocks shown inmay be constituted by a different circuit from the CPU. The same configuration as that shown inwill not be described.
20 FIG. 20 FIG. 20 FIG. Each unit shown inmay be constituted by at least one of a processor and a logic circuit. Each unit shown inmay include one or a plurality of processors. Each unit shown inmay include one or a plurality of logic circuits.
182 182 182 182 182 The region detection unitdetects one or more feature regions on a subject on the basis of the point-cloud data. The region detection unitin the third embodiment detects two or more feature regions, but the region detection unitin the fourth embodiment may detect one feature region. In a case in which the region detection unitdetects one feature region, the region detection unitassigns all the points in the point-cloud data to the feature region.
188 188 185 The information control unitexecutes the first control. The information control unitcontrols the point information generated by the point input unitso that the point information indicates a point in one of the one or more feature regions.
186 182 182 186 182 186 185 188 187 The region selection unitselects one of the one or more feature regions detected by the region detection unitas a selected region (region selection step). When the region detection unitdetects two or more feature regions, the region selection unitselects one of the two or more feature regions as a selected region. When the region detection unitdetects only one feature region, the region selection unitselects the feature region. Only when the point input unitaccepts a point in the selected region, the information control unitvalidates the point information indicating the point (information control step). The measurement unitmeasures the size of the subject on the basis of the 3D coordinates of two or more points including the point indicated by the valid point information.
187 187 187 The measurement unituses the 3D coordinates of one or more points indicated by the valid point information. It is not necessary that the point information of all the two or more points used by the measurement unitbe valid. The point information of some of the two or more points used by the measurement unitmay be invalid. For example, in a case in which the plane-based measurement is executed, only the point information of three reference points defining a reference plane may be valid, and the point information of one measurement point may be invalid. Alternatively, the point information of one measurement point may be valid regardless of the position of the measurement point.
186 182 185 188 185 The region selection unitmay select one of the two or more feature regions detected by the region detection unitas a first selected region and may select one of the one or more feature regions other than the first selected region as a second selected region. Only when the point input unitaccepts a point in the first selected region or the second selected region, the information control unitmay validate the point information indicating the point. For example, in a case in which the plane-based measurement is executed, the point input unitmay accept three reference points in the first selected region and may accept one measurement point in the second selected region.
21 FIG. 21 FIG. 13 FIG. Three-dimensional measurement in the fourth embodiment will be described by using.shows a procedure of the three-dimensional measurement. The same processing as that shown inwill not be described.
105 185 4 188 121 121 After Step S, the point input unitaccepts one point on the 2D image of the subject through the operation unitand generates point information indicating the accepted point. The information control unitvalidates the point information (Step S). Step Scorresponds to the point input step.
188 188 For example, the information control unitgenerates information indicating that the point information is valid and adds the generated information to the point information. The information control unitmay manage a table including the point information and the information indicating that the point information is valid.
121 186 121 186 122 122 After Step S, the region selection unitdetects a position on the image of the feature regions corresponding to the point accepted in Step S. The region selection unitselects a feature region including the position as a selected region (Step S). Step Scorresponds to the region selection step.
186 186 122 107 Each pixel of the 2D image of the subject is associated with a pixel of the image of the feature regions. Therefore, the region selection unitcan calculate a position on the image of the feature regions corresponding to a position on the 2D image of the subject. The region selection unitselects a feature region including the calculated position. After Step S, Step Sis executed.
109 188 109 188 188 123 After Step S, the information control unitcalculates a position on the image of the feature regions corresponding to the point accepted in Step S. The information control unitcan calculate a position on the image of the feature regions corresponding to a position on the 2D image of the subject. The information control unitdetermines whether or not the calculated position is in the selected region (Step S). Points on the borders between the selected region and other feature regions are dealt as points in the selected region. Points on the borders between the selected region and other feature regions may be dealt as points outside the selected region.
188 123 107 188 183 5 1 When the information control unitdetermines that the point is not in the selected region in Step S, Step Sis executed. In such a case, the point information of the point is invalid. The information control unitmay discard the point information of the point. The display control unitmay display information indicating that the accepted point is invalid on the display unit. In this way, the endoscope devicecan encourage a user to designate a valid point.
188 123 188 124 124 121 124 110 When the information control unitdetermines that the point is in the selected region in Step S, the information control unitmakes the point information of the point valid (Step S). Step Scorresponds to the information control step. A method of validating the point information is similar to that in Step S. After Step S, Step Sis executed.
187 1 The measurement unitmeasures the size of the subject on the basis of the 3D coordinates of only the point indicated by the valid point information. The point indicated by the valid point information is in the selected region. Since the point in the selected region is used for measurement, the endoscope devicecan avoid using a point that is not intended by a user.
22 FIG. 14 FIG. 14 FIG. 15 FIG. 22 FIG. 5 11 12 5 11 11 11 122 shows an example of an image displayed on the display unit. The same 2D image Gas that shown inand the same image Gas that shown inare displayed on the display unit. The same cursor Cas that shown inis displayed on the 2D image G.shows a state after the feature region Ris selected in Step S.
11 11 21 185 21 21 12 188 12 21 11 123 21 187 21 A user moves the cursor Cand designates a point on the 2D image G. For example, a user designates a point P, and the point input unitaccepts the point P. The point Pcorresponds to a point in the feature region R. The information control unitdetermines that a point on the image Gcorresponding to the point Pis not in the feature region Rin Step S. Therefore, the point information of the point Pis invalid, and the measurement unitdoes not use the 3D coordinates of the point Pfor measurement.
22 185 22 22 11 188 12 22 11 123 188 22 124 187 22 Thereafter, a user designates a point P, and the point input unitaccepts the point P. The point Pcorresponds to a point in the feature region R. The information control unitdetermines that a point on the image Gcorresponding to the point Pis in the feature region Rin Step S. Therefore, the information control unitvalidates the point information of the point Pin Step S. The measurement unituses the 3D coordinates of the point Pfor measurement.
23 FIG. 23 FIG. 14 FIG. 15 FIG. 23 FIG. 5 104 12 5 11 12 12 4 121 186 122 11 122 shows another example of an image displayed on the display unit. In the example shown in, the 2D image of the subject is not displayed in Step S. The same image Gas that shown inis displayed on the display unit. The same cursor Cas that shown inis displayed on the image G. When a position on the image Gis accepted through the operation unitin Step S, the region selection unitselects a feature region corresponding to the position in Step S.shows a state after the feature region Ris selected in Step S.
11 12 23 185 23 23 12 188 23 11 123 23 187 23 A user moves the cursor Cand designates a point on the image G. For example, a user designates a point P, and the point input unitaccepts the point P. The point Pis in the feature region R. The information control unitdetermines that the point Pis not in the feature region Rin Step S. Therefore, the point information of the point Pis invalid, and the measurement unitdoes not use the 3D coordinates of the point Pfor measurement.
24 185 24 24 11 188 24 11 123 188 24 124 187 24 Thereafter, a user designates a point P, and the point input unitaccepts the point P. The point Pis in the feature region R. The information control unitdetermines that the point Pis in the feature region Rin Step S. Therefore, the information control unitvalidates the point information of the point Pin Step S. The measurement unituses the 3D coordinates of the point Pfor measurement.
185 11 12 13 14 188 11 123 187 There is a case in which the point input unitaccepts a point in a region (background) that is not any of the feature region R, the feature region R, the feature region R, and the feature region R. In such a case, the information control unitdetermines that the point is not in the feature region Rin Step S. Therefore, the point information of the point is invalid, and the measurement unitdoes not use the 3D coordinates of the point for measurement.
21 FIG. 185 121 186 122 185 109 187 111 185 1 In the three-dimensional measurement shown in, when the point input unitaccepts a first point on the subject in Step S, the region selection unitselects a feature region including the first point as a selected region in Step S. When the point input unitaccepts a second point in the selected region in Step S, the measurement unitmeasures the size of the subject on the basis of the 3D coordinates of two or more points including the first point and the second point in Step S. The second point is different from the first point. In a case in which the point input unitaccepts three or more points, the three or more points include the first point and the second point described above. Since two or more points in the selected region are used for measurement, the endoscope devicecan avoid using a point that is not intended by a user.
22 FIG. 23 FIG. 5 5 5 In the example shown in, the 2D image of the subject and the image of the feature regions are displayed on the display unit, and the 2D image is used for inputting a point. In the example shown in, the image of the feature regions is displayed on the display unit, and the image is used for inputting a point. A combination of an image displayed on the display unitand an image used for inputting a point is not limited to these examples. The combination is similar to that in the third embodiment. In the fourth embodiment, the image of the feature regions does not need to be displayed.
21 FIG. 187 121 187 In the three-dimensional measurement shown in, the measurement unituses the point accepted in Step Sfor measurement. The point may be used only for selecting a feature region. The measurement unitdoes not need to use the point for measurement.
183 122 183 5 122 The display control unitmay highlight the selected region in Step S. Alternatively, the display control unitmay display only the selected region on the display unitin Step S.
5 4 105 107 108 18 184 In a case in which the display unitand the operation unitare constituted as a touch panel, the cursor does not need to be displayed. Accordingly, Step S, Step S, and Step Sdo not need to be executed. The CPUdoes not need to have the function of the cursor calculation unit.
21 FIG. 5 101 102 103 5 102 103 The order of processing in the three-dimensional measurement is not limited to that shown in. For example, the 2D image of the subject may be displayed on the display unitbefore Step S, Step S, or Step Sis executed. In a case in which the 3D image of the point-cloud data is used instead of the 2D image of the subject, the 3D image may be displayed on the display unitbefore Step Sor Step Sis executed.
1 1 1 In the fourth embodiment, the endoscope deviceexecutes the first control. In the first control, the point information is controlled so that the point information indicates a point in the selected region. The endoscope deviceuses the 3D coordinates of the point in the selected region for measurement. Therefore, the endoscope devicecan support correct designation of a point on the subject. A user can easily designate a point on the border of the selected region.
1 18 12 FIG. A fifth embodiment of the present invention will be described. The endoscope deviceaccording to the fifth embodiment includes the CPUshown in. In the fifth embodiment, the position of a mark on an image is restricted to a position in one feature region.
183 184 183 184 The display control unitand the cursor calculation unitexecute the second control. The display control unitdisplays a mark on an image. The cursor calculation unitrestricts the position of the mark to a position in a region corresponding to one of one or more feature regions.
183 5 186 182 182 186 182 186 183 184 4 185 The display control unitdisplays one of the 3D image of the point-cloud data and the 2D image of the subject on the display unit(image display step). The region selection unitselects one of the one or more feature regions detected by the region detection unitas a selected region (region selection step). In a case in which the region detection unitdetects two or more feature regions, the region selection unitselects one of the two or more feature regions as a selected region. In a case in which the region detection unitdetects only one feature region, the region selection unitselects the feature region. The display control unitdisplays a mark on one of the 3D image of the point-cloud data and the 2D image of the subject (mark display step). The cursor calculation unitcalculates a position at which the mark is displayed on the basis of the information accepted through the operation unit(position calculation step). The position at which the mark is displayed is restricted to a position in a region corresponding to the selected region. The point input unitaccepts a point corresponding to the position of the mark (point input step).
183 183 5 The display control unitmay display the mark on the image of the one or more feature regions (mark display step). In a case in which the mark is displayed on the image of the feature regions, the display control unitdoes not need to display the 3D image of the point-cloud data and the 2D image of the subject on the display unit.
186 182 185 185 The region selection unitmay select one of the two or more feature regions detected by the region detection unitas a first selected region and may select one of the one or more feature regions other than the first selected region as a second selected region. The position at which the mark is displayed may be restricted to a position in a region corresponding to the first selected region or the second selected region. For example, in a case in which the plane-based measurement is executed, the position of the mark may be restricted to a position in a region corresponding to the first selected region, and the point input unitmay accept three reference points in the first selected region. Thereafter, the position of the mark may be restricted to a position in a region corresponding to the second selected region, and the point input unitmay accept one measurement point in the second selected region.
24 FIG. 24 FIG. 13 FIG. 21 FIG. Three-dimensional measurement in the fifth embodiment will be described by using.shows a procedure of the three-dimensional measurement. The same processing as that shown inandwill not be described.
183 5 104 105 185 4 131 131 The display control unitdisplays the 2D image of the subject and the image of the feature regions on the display unitin Step S. After Step S, the point input unitaccepts one point on the 2D image of the subject through the operation unitand generates point information indicating the accepted point (Step S). Step Scorresponds to the point input step.
131 186 122 184 4 107 107 After Step S, the region selection unitselects one feature region as a selected region in Step S. The cursor calculation unitcalculates a position on the 2D image of the subject on the basis of the position information input into the operation unitin Step S. Step Scorresponds to the position calculation step.
107 184 107 184 184 132 After Step S, the cursor calculation unitcalculates a position on the image of the feature regions corresponding to the position calculated in Step S. Each pixel of the 2D image of the subject is associated with a pixel of the image of the feature regions. Therefore, the cursor calculation unitcan calculate a position on the image of the feature regions corresponding to a position on the 2D image of the subject. The cursor calculation unitdetermines whether or not the calculated position is in the selected region (Step S). Points on the borders between the selected region and other feature regions are dealt as points in the selected region. Points on the borders between the selected region and other feature regions may be dealt as points outside the selected region.
184 132 107 184 132 183 108 108 When the cursor calculation unitdetermines that the position is not in the selected region in Step S, Step Sis executed. When the cursor calculation unitdetermines that the position is in the selected region in Step S, the display control unitdisplays a cursor at the position in Step S. Step Scorresponds to the mark display step. The cursor is displayed in a region of the 2D image corresponding to the selected region. A user can move the cursor only in the region.
185 109 187 111 1 The point input unitaccepts a point in the region of the 2D image corresponding to the selected region in Step S. The measurement unitmeasures the size of the subject on the basis of the 3D coordinates of only a point in the region of the 2D image corresponding to the selected region in Step S. Since the point in the selected region is used for measurement, the endoscope devicecan avoid using a point that is not intended by a user.
25 FIG. 14 FIG. 14 FIG. 15 FIG. 25 FIG. 5 11 12 5 11 11 11 122 shows an example of an image displayed on the display unit. The same 2D image Gas that shown inand the same image Gas that shown inare displayed on the display unit. The same cursor Cas that shown inis displayed on the 2D image G.shows a state after the feature region Ris selected in Step S.
11 11 11 11 11 184 12 11 11 132 108 11 11 12 11 11 11 31 185 31 31 11 187 31 A user moves the cursor Cand designates a point on the 2D image G. When a user tries to move the cursor Coutside a region on the 2D image Gcorresponding to the feature region R, the cursor calculation unitdetermines that the position on the image Gcorresponding to the position of the cursor Cis not in the feature region Rin Step S. Therefore, Step Sis not executed. For example, a user cannot move the cursor Cto a position on the 2D image Gcorresponding to a position in the feature region R. On the other hand, a user can move the cursor Cto a position on the 2D image Gcorresponding to a position in the feature region R. For example, a user designates a point P, and the point input unitaccepts the point P. The point Pcorresponds to a point in the feature region R. The measurement unituses the 3D coordinates of the point Pfor measurement.
26 FIG. 26 FIG. 14 FIG. 15 FIG. 26 FIG. 5 104 12 5 11 12 12 4 131 186 122 11 122 shows another example of an image displayed on the display unit. In the example shown in, the 2D image of the subject is not displayed in Step S. The same image Gas that shown inis displayed on the display unit. The same cursor Cas that shown inis displayed on the image G. When a position on the image Gis accepted through the operation unitin Step S, the region selection unitselects a feature region corresponding to the position in Step S.shows a state after the feature region Ris selected in Step S.
11 12 11 11 184 11 11 132 108 11 12 11 11 32 185 32 32 11 187 32 A user moves the cursor Cand designates a point on the image G. When a user tries to move the cursor Coutside the feature region R, the cursor calculation unitdetermines that the position of the cursor Cis not in the feature region Rin Step S. Therefore, Step Sis not executed. For example, a user cannot move the cursor Cto a position in the feature region R. On the other hand, a user can move the cursor Cto a position in the feature region R. For example, a user designates a point P, and the point input unitaccepts the point P. The point Pis in the feature region R. The measurement unituses the 3D coordinates of the point Pfor measurement.
24 FIG. 185 131 186 122 185 109 187 111 185 1 In the three-dimensional measurement shown in, when the point input unitaccepts a first point on the subject in Step S, the region selection unitselects a feature region including the first point as a selected region in Step S. When the point input unitaccepts a second point in the selected region in Step S, the measurement unitmeasures the size of the subject on the basis of the 3D coordinates of two or more points including the first point and the second point in Step S. The second point is different from the first point. In a case in which the point input unitaccepts three or more points, the three or more points include the first point and the second point described above. Since two or more points in the selected region are used for measurement, the endoscope devicecan avoid using a point that is not intended by a user.
25 FIG. 26 FIG. 5 5 5 In the example shown in, the 2D image of the subject and the image of the feature regions are displayed on the display unit, and the 2D image is used for inputting a point. In the example shown in, the image of the feature regions is displayed on the display unit, and the image is used for inputting a point. A combination of an image displayed on the display unitand an image used for inputting a point is not limited to these examples. The combination is similar to that in the third embodiment. In the fifth embodiment, the image of the feature regions does not need to be displayed.
24 FIG. 187 131 187 In the three-dimensional measurement shown in, the measurement unituses the point accepted in Step Sfor measurement. The point may be used only for selecting a feature region. The measurement unitdoes not need to use the point for measurement.
183 122 183 5 122 The display control unitmay highlight the selected region in Step S. Alternatively, the display control unitmay display only the selected region on the display unitin Step S.
24 FIG. 5 101 102 103 5 102 103 The order of processing in the three-dimensional measurement is not limited to that shown in. For example, the 2D image of the subject may be displayed on the display unitbefore Step S, Step S, or Step Sis executed. In a case in which the 3D image of the point-cloud data is used instead of the 2D image of the subject, the 3D image may be displayed on the display unitbefore Step Sor Step Sis executed.
1 1 1 In the fifth embodiment, the endoscope deviceexecutes the second control. In the second control, a mark is displayed on an image, and the position of the mark is restricted to a position in a region corresponding to the selected region. The endoscope deviceuses the 3D coordinates of the point in the selected region for measurement. Therefore, the endoscope devicecan support correct designation of a point on the subject. A user can easily designate a point on the border of the selected region.
1 18 a 20 FIG. A sixth embodiment of the present invention will be described. The endoscope deviceaccording to the sixth embodiment includes the CPUshown in. In the sixth embodiment, in a case in which a point outside a feature region is accepted, the point is changed to a point in the feature region.
188 188 185 The information control unitexecutes the first control. The information control unitcontrols the point information generated by the point input unitso that the point information indicates a point in one of the one or more feature regions.
186 182 182 186 182 186 185 188 188 187 The region selection unitselects one of the one or more feature regions detected by the region detection unitas a selected region (region selection step). When the region detection unitdetects two or more feature regions, the region selection unitselects one of the two or more feature regions as a selected region. When the region detection unitdetects only one feature region, the region selection unitselects the feature region. When the point input unitaccepts a point outside the selected region, the information control unitinvalidates the point information indicating the point. Furthermore, the information control unitgenerates new point information indicating a point in the selected region (information control step). The measurement unitmeasures the size of the subject on the basis of the 3D coordinates of two or more points indicated by the point information other than the invalid point information (measurement step).
187 187 187 The measurement unitdoes not use the 3D coordinates of one or more points indicated by the invalid point information for measurement. The measurement unituses the 3D coordinates of one or more points indicated by the point information that is not invalid for measurement. The point information of some of the two or more points used by the measurement unitdoes not need to be a target in the above-described processing. For example, in a case in which the plane-based measurement is executed, the point information of three reference points defining a reference plane may be a target in the above-described processing, and the point information of one measurement point does not need to be a target in the above-described processing.
186 182 185 188 185 The region selection unitmay select one of the two or more feature regions detected by the region detection unitas a first selected region and may select one of the one or more feature regions other than the first selected region as a second selected region. Only when the point input unitaccepts a point in a region other than the first selected region and the second selected region, the information control unitmay invalidate the point information indicating the point. For example, in a case in which the plane-based measurement is executed, the point input unitmay accept three reference points in the first selected region and may accept one measurement point in the second selected region.
27 FIG. 27 FIG. 13 FIG. 21 FIG. 24 FIG. Three-dimensional measurement in the sixth embodiment will be described by using.shows a procedure of the three-dimensional measurement. The same processing as that shown in,, andwill not be described.
183 5 104 185 131 186 122 The display control unitdisplays the 2D image of the subject and the image of the feature regions on the display unitin Step S. The point input unitaccepts a point on the 2D image of the subject in Step S. The region selection unitselects one feature region as a selected region in Step S.
185 109 188 123 188 123 The point input unitaccepts a point on the 2D image of the subject in Step S. The information control unitcalculates a position on the image of the feature regions corresponding to the point in Step S. The information control unitdetermines whether or not the calculated position is in the selected region in Step S.
188 123 188 141 141 When the information control unitdetermines that the position is not in the selected region in Step S, the information control unitinvalidates the point information of the point (Step S). Step Scorresponds to the information control step.
188 188 188 For example, the information control unitgenerates information indicating that the point information is invalid and adds the generated information to the point information. The information control unitmay manage a table including the point information and the information indicating that the point information is invalid. The invalid point information indicates that use of the point indicated by the point information for measurement is prohibited. The information control unitmay invalidate the point information by discarding the point information.
141 188 142 142 After Step S, the information control unitgenerates new point information indicating a point in the selected region (Step S). Step Scorresponds to the information control step.
188 123 188 123 188 188 123 142 110 The information control unitcalculates a position in the selected region on the basis of the position calculated in Step S. For example, the information control unitcalculates the closest position to that calculated in Step S. The information control unitgenerates point information indicating the position. The information control unitmay calculate a position of a new point on the basis of the distance between the position calculated in Step Sand a position in the selected region, the brightness at a position in the selected region, the contrast at a position in the selected region, a correlation value obtained in matching processing, and the like. After Step S, Step Sis executed.
187 111 1 The measurement unitmeasures the size of the subject on the basis of the 3D coordinates of only a point indicated by the point information that is not invalid in Step S. The point indicated by the point information that is not invalid is in the selected region. Since the point in the selected region is used for measurement, the endoscope devicecan avoid using a point that is not intended by a user.
28 FIG. 14 FIG. 14 FIG. 15 FIG. 28 FIG. 5 11 12 5 11 11 11 122 shows an example of an image displayed on the display unit. The same 2D image Gas that shown inand the same image Gas that shown inare displayed on the display unit. The same cursor Cas that shown inis displayed on the 2D image G.shows a state after the feature region Ris selected in Step S.
11 11 41 185 41 41 12 188 12 41 11 123 188 41 141 188 42 11 142 42 41 42 41 11 187 42 41 28 FIG. A user moves the cursor Cand designates a point on the 2D image G. For example, a user designates a point P, and the point input unitaccepts the point P. The point Pcorresponds to a point in the feature region R. The information control unitdetermines that a point on the image Gcorresponding to the point Pis not in the feature region Rin Step S. Therefore, the information control unitinvalidates the point information of the point Pin Step S. Furthermore, the information control unitgenerates new point information indicating a point Pin a region corresponding to the feature region Rin Step S. At this time, a mark of the point Pmay be displayed without displaying a mark of the point P. In the example shown in, the point Pis the closest to the point Pamong points in the region corresponding to the feature region R. The measurement unituses the 3D coordinates of the point Pfor measurement without using the 3D coordinates of the point Pfor measurement.
29 FIG. 29 FIG. 14 FIG. 15 FIG. 29 FIG. 5 104 12 5 11 12 12 4 131 186 122 11 122 shows another example of an image displayed on the display unit. In the example shown in, the 2D image of the subject is not displayed in Step S. The same image Gas that shown inis displayed on the display unit. The same cursor Cas that shown inis displayed on the image G. When a position on the image Gis accepted through the operation unitin Step S, the region selection unitselects a feature region corresponding to the position in Step S.shows a state after the feature region Ris selected in Step S.
11 12 43 185 43 43 12 188 43 11 123 188 43 141 188 44 11 142 44 43 44 43 11 187 44 43 29 FIG. A user moves the cursor Cand designates a point on the image G. For example, a user designates a point P, and the point input unitaccepts the point P. The point Pis in the feature region R. The information control unitdetermines that the point Pis not in the feature region Rin Step S. Therefore, the information control unitinvalidates the point information of the point Pin Step S. Furthermore, the information control unitgenerates new point information indicating a point Pin the feature region Rin Step S. At this time, a mark of the point Pmay be displayed without displaying a mark of the point P. In the example shown in, the point Pis the closest to the point Pamong points in the feature region R. The measurement unituses the 3D coordinates of the point Pfor measurement without using the 3D coordinates of the point Pfor measurement.
185 11 12 13 14 188 11 123 187 There is a case in which the point input unitaccepts a point in a region (background) that is not any of the feature region R, the feature region R, the feature region R, and the feature region R. In such a case, the information control unitdetermines that the point is not in the feature region Rin Step S. Therefore, the point information of the point is invalid, and the measurement unitdoes not use the 3D coordinates of the point for measurement.
27 FIG. 185 131 186 122 185 109 187 111 185 1 In the three-dimensional measurement shown in, when the point input unitaccepts a first point on the subject in Step S, the region selection unitselects a feature region including the first point as a selected region in Step S. When the point input unitaccepts a second point in the selected region in Step S, the measurement unitmeasures the size of the subject on the basis of the 3D coordinates of two or more points including the first point and the second point in Step S. The second point is different from the first point. In a case in which the point input unitaccepts three or more points, the three or more points include the first point and the second point described above. Since two or more points in the selected region are used for measurement, the endoscope devicecan avoid using a point that is not intended by a user.
28 FIG. 29 FIG. 5 5 5 In the example shown in, the 2D image of the subject and the image of the feature regions are displayed on the display unit, and the 2D image is used for inputting a point. In the example shown in, the image of the feature regions is displayed on the display unit, and the image is used for inputting a point. A combination of an image displayed on the display unitand an image used for inputting a point is not limited to these examples. The combination is similar to that in the third embodiment. In the sixth embodiment, the image of the feature regions does not need to be displayed.
27 FIG. 187 131 187 In the three-dimensional measurement shown in, the measurement unituses the point accepted in Step Sfor measurement. The point may be used only for selecting a feature region. The measurement unitdoes not need to use the point for measurement.
183 122 183 5 122 The display control unitmay highlight the selected region in Step S. Alternatively, the display control unitmay display only the selected region on the display unitin Step S.
5 4 105 107 108 18 184 In a case in which the display unitand the operation unitare constituted as a touch panel, the cursor does not need to be displayed. Accordingly, Step S, Step S, and Step Sdo not need to be executed. The CPUdoes not need to have the function of the cursor calculation unit.
27 FIG. 5 101 102 103 5 102 103 The order of processing in the three-dimensional measurement is not limited to that shown in. For example, the 2D image of the subject may be displayed on the display unitbefore Step S, Step S, or Step Sis executed. In a case in which the 3D image of the point-cloud data is used instead of the 2D image of the subject, the 3D image may be displayed on the display unitbefore Step Sor Step Sis executed.
1 1 1 In the sixth embodiment, the endoscope deviceexecutes the first control. In the first control, the point information is controlled so that the point information indicates a point in the selected region. The endoscope deviceuses the 3D coordinates of the point in the selected region for measurement. Therefore, the endoscope devicecan support correct designation of a point on the subject. A user can easily designate a point on the border of the selected region.
1 18 a 20 FIG. A seventh embodiment of the present invention will be described. The endoscope deviceaccording to the seventh embodiment includes the CPUshown in. In the seventh embodiment, in a case in which a mark on an image is outside one feature region and a point at the position of the mark is accepted, the position of the mark is changed to a position in the feature region.
183 184 183 184 The display control unitand the cursor calculation unitexecute the second control. The display control unitdisplays a mark on an image. The cursor calculation unitrestricts the position of the mark to a position in a region corresponding to one of one or more feature regions.
183 5 186 182 182 186 182 186 183 184 4 The display control unitdisplays one of the 3D image of the point-cloud data and the 2D image of the subject on the display unit(image display step). The region selection unitselects one of the one or more feature regions detected by the region detection unitas a selected region (region selection step). In a case in which the region detection unitdetects two or more feature regions, the region selection unitselects one of the two or more feature regions as a selected region. In a case in which the region detection unitdetects only one feature region, the region selection unitselects the feature region. The display control unitdisplays a mark on one of the 3D image of the point-cloud data and the 2D image of the subject (mark display step). The cursor calculation unitcalculates a position at which the mark is displayed on the basis of the information accepted through the operation unit(position calculation step).
185 187 185 When the mark is displayed in a region corresponding to the selected region and the point input unitaccepts a point corresponding to the position of the mark, the measurement unitcalculates the size of the subject on the basis of the 3D coordinates of two or more points including the point indicated by the point information (measurement step). When the mark is displayed outside the region corresponding to the selected region and the point input unitaccepts a point corresponding to the position of the mark, the mark is displayed in the region corresponding to the selected region.
183 185 187 185 183 5 The display control unitmay display the mark on the image of the one or more feature regions (mark display step). When the mark is displayed in the selected region and the point input unitaccepts a point corresponding to the position of the mark, the measurement unitmay measure the size of the subject on the basis of the 3D coordinates of two or more points including the point indicated by the point information (measurement step). When the mark is displayed outside the selected region and the point input unitaccepts a point corresponding to the position of the mark, the mark may be displayed in the selected region. In a case in which the mark is displayed on the image of the feature regions, the display control unitdoes not need to display the 3D image of the point-cloud data and the 2D image of the subject on the display unit.
186 182 185 185 The region selection unitmay select one of the two or more feature regions detected by the region detection unitas a first selected region and may select one of the one or more feature regions other than the first selected region as a second selected region. The position at which the mark is displayed may be restricted to a position in a region corresponding to the first selected region or the second selected region. For example, in a case in which the plane-based measurement is executed, the position of the mark may be restricted to a position in a region corresponding to the first selected region and the point input unitmay accept three reference points in the first selected region. Thereafter, the position of the mark may be restricted to a position in a region corresponding to the second selected region and the point input unitmay accept one measurement point in the second selected region.
30 FIG. 30 FIG. 13 FIG. 21 FIG. 24 FIG. 27 FIG. Three-dimensional measurement in the seventh embodiment will be described by using.shows a procedure of the three-dimensional measurement. The same processing as that shown in,,, andwill not be described.
183 5 104 185 131 186 122 The display control unitdisplays the 2D image of the subject and the image of the feature regions on the display unitin Step S. The point input unitaccepts a point on the 2D image of the subject in Step S. The region selection unitselects one feature region as a selected region in Step S.
185 109 188 123 188 123 The point input unitaccepts a point on the 2D image of the subject in Step S. The information control unitcalculates a position on the image of the feature regions corresponding to the point in Step S. The information control unitdetermines whether or not the calculated position is in the selected region in Step S.
188 123 188 141 When the information control unitdetermines that the position is not in the selected region in Step S, the information control unitinvalidates the point information of the point in Step S.
141 184 151 151 After Step S, the cursor calculation unitcalculates a new position of the cursor on the 2D image of the subject (Step S). Step Scorresponds to the position calculation step.
184 123 184 123 184 123 151 108 The cursor calculation unitcalculates a position in the selected region on the basis of the position calculated in Step S. For example, the cursor calculation unitcalculates the closest position to that calculated in Step S. The cursor calculation unitmay calculate a new position of the cursor on the basis of the distance between the position calculated in Step Sand a position in the selected region, the brightness at a position in the selected region, the contrast at a position in the selected region, a correlation value obtained in matching processing, and the like. After Step S, Step Sis executed.
183 184 108 108 The display control unitdisplays the cursor at the position calculated by the cursor calculation unitin Step S, thus updating the cursor. When Step Sis executed, the cursor moves from a position outside a region of the 2D image corresponding to the selected region to a position in the region of the 2D image corresponding to the selected region.
187 111 1 The measurement unitmeasures the size of the subject on the basis of the 3D coordinates of only a point indicated by the point information that is not invalid in Step S. The point indicated by the point information that is not invalid is in the selected region. Since the point in the selected region is used for measurement, the endoscope devicecan avoid using a point that is not intended by a user.
31 FIG. 14 FIG. 14 FIG. 15 FIG. 31 FIG. 5 11 12 5 11 11 11 122 shows an example of an image displayed on the display unit. The same 2D image Gas that shown inand the same image Gas that shown inare displayed on the display unit. The same cursor Cas that shown inis displayed on the 2D image G.shows a state after the feature region Ris selected in Step S.
11 11 51 185 51 51 12 188 12 51 11 123 188 51 141 184 11 151 11 183 11 151 187 51 A user moves the cursor Cand designates a point on the 2D image G. For example, a user designates a point P, and the point input unitaccepts the point P. The point Pcorresponds to a point in the feature region R. The information control unitdetermines that a point on the image Gcorresponding to the point Pis not in the feature region Rin Step S. Therefore, the information control unitinvalidates the point information of the point Pin Step S. The cursor calculation unitcalculates a new position of the cursor Cin Step S. The position is in a region corresponding to the feature region R. The display control unitdisplays the cursor Cat the position in Step S. The measurement unitdoes not use the 3D coordinates of the point Pfor measurement.
32 FIG. 32 FIG. 14 FIG. 15 FIG. 32 FIG. 5 104 12 5 11 12 12 4 131 186 122 11 122 shows another example of an image displayed on the display unit. In the example shown in, the 2D image of the subject is not displayed in Step S. The same image Gas that shown inis displayed on the display unit. The same cursor Cas that shown inis displayed on the image G. When a position on the image Gis accepted through the operation unitin Step S, the region selection unitselects a feature region corresponding to the position in Step S.shows a state after the feature region Ris selected in Step S.
11 12 52 185 52 52 12 188 52 11 123 188 52 141 184 11 151 11 183 11 108 187 52 A user moves the cursor Cand designates a point on the image G. For example, a user designates a point P, and the point input unitaccepts the point P. The point Pis in the feature region R. The information control unitdetermines that the point Pis not in the feature region Rin Step S. Therefore, the information control unitinvalidates the point information of the point Pin Step S. The cursor calculation unitcalculates a new position of the cursor Cin Step S. The position is in the feature region R. The display control unitdisplays the cursor Cat the position in Step S. The measurement unitdoes not use the 3D coordinates of the point Pfor measurement.
185 11 12 13 14 188 11 123 187 There is a case in which the point input unitaccepts a point in a region (background) that is not any of the feature region R, the feature region R, the feature region R, and the feature region R. In such a case, the information control unitdetermines that the point is not in the feature region Rin Step S. Therefore, the point information of the point is invalid, and the measurement unitdoes not use the 3D coordinates of the point for measurement.
30 FIG. 185 131 186 122 185 109 187 111 185 1 In the three-dimensional measurement shown in, when the point input unitaccepts a first point on the subject in Step S, the region selection unitselects a feature region including the first point as a selected region in Step S. When the point input unitaccepts a second point in the selected region in Step S, the measurement unitmeasures the size of the subject on the basis of the 3D coordinates of two or more points including the first point and the second point in Step S. The second point is different from the first point. In a case in which the point input unitaccepts three or more points, the three or more points include the first point and the second point described above. Since two or more points in the selected region are used for measurement, the endoscope devicecan avoid using a point that is not intended by a user.
31 FIG. 32 FIG. 5 5 5 In the example shown in, the 2D image of the subject and the image of the feature regions are displayed on the display unit, and the 2D image is used for inputting a point. In the example shown in, the image of the feature regions is displayed on the display unit, and the image is used for inputting a point. A combination of an image displayed on the display unitand an image used for inputting a point is not limited to these examples. The combination is similar to that in the third embodiment. In the seventh embodiment, the image of the feature regions does not need to be displayed.
30 FIG. 187 131 187 In the three-dimensional measurement shown in, the measurement unituses the point accepted in Step Sfor measurement. The point may be used only for selecting a feature region. The measurement unitdoes not need to use the point for measurement.
183 122 183 5 122 The display control unitmay highlight the selected region in Step S. Alternatively, the display control unitmay display only the selected region on the display unitin Step S.
30 FIG. 5 101 102 103 5 102 103 The order of processing in the three-dimensional measurement is not limited to that shown in. For example, the 2D image of the subject may be displayed on the display unitbefore Step S, Step S, or Step Sis executed. In a case in which the 3D image of the point-cloud data is used instead of the 2D image of the subject, the 3D image may be displayed on the display unitbefore Step Sor Step Sis executed.
1 1 1 In the seventh embodiment, the endoscope deviceexecutes the second control. In the second control, a mark is displayed on an image, and the position of the mark is restricted to a position in a region corresponding to the selected region. The endoscope deviceuses the 3D coordinates of the point in the selected region for measurement. Therefore, the endoscope devicecan support correct designation of a point on the subject. A user can easily designate a point on the border of the selected region.
41 41 1 11 FIG. An eighth embodiment of the present invention will be described. Hereinafter, an example in which the PCshown inis a measurement device is described. The PCacquires the 2D image of the subject from the endoscope deviceand executes three-dimensional measurement.
16 1 41 16 41 41 1 The external device interfaceof the endoscope deviceperforms communication with the PC. Specifically, the external device interfacetransmits one or more 2D images of the subject to the PC. The PCreceives the 2D images from the endoscope device.
16 41 16 41 For example, the external device interfaceis connected to the PCwirelessly or by a cable. The communication between the external device interfaceand the PCmay be performed via a local area network (LAN) or the Internet.
33 FIG. 33 FIG. 41 41 43 44 45 shows a configuration of the PC. The PCshown inincludes a communication unit, a CPU, and a display unit.
43 16 1 43 16 44 45 45 The communication unitperforms communication with the external device interfaceof the endoscope device. Specifically, the communication unitreceives one or more 2D images of a subject from the external device interface. The CPUexecutes processing for three-dimensional measurement. The display unitis a monitor (display) such as an LCD. The display unitincludes a display screen and displays an image, an operation menu, and the like on the display screen.
34 FIG. 34 FIG. 44 44 440 441 442 443 444 445 446 447 448 44 shows a functional configuration of the CPU. The CPUhas functional units including a control unit, a generation unit, a region detection unit, a display control unit, a cursor calculation unit, a point input unit, a region selection unit, a measurement unit, and a communication control unit. At least one of the blocks shown inmay be constituted by a different circuit from the CPU.
34 FIG. 34 FIG. 34 FIG. Each unit shown inmay be constituted by at least one of a processor and a logic circuit. Each unit shown inmay include one or a plurality of processors. Each unit shown inmay include one or a plurality of logic circuits.
440 441 181 442 182 443 183 444 184 445 185 446 186 447 187 448 16 1 43 12 FIG. 12 FIG. 12 FIG. 12 FIG. 12 FIG. 12 FIG. 12 FIG. The control unitcontrols processing executed by each unit. The generation unithas the same function as that of the generation unitshown in. The region detection unithas the same function as that of the region detection unitshown in. The display control unithas the same function as that of the display control unitshown in. The cursor calculation unithas the same function as that of the cursor calculation unitshown in. The point input unithas the same function as that of the point input unitshown in. The region selection unithas the same function as that of the region selection unitshown in. The measurement unithas the same function as that of the measurement unitshown in. The communication control unitperforms communication with the external device interfaceof the endoscope deviceby controlling the communication unit.
44 44 188 44 188 44 13 FIG. 24 FIG. 20 FIG. 21 FIG. 27 FIG. 30 FIG. The CPUexecutes three-dimensional measurement shown inor. The CPUmay have the function of the information control unitshown in. In a case in which the CPUhas the function of the information control unit, the CPUmay execute three-dimensional measurement shown in,, or.
44 44 44 The CPUmay read a program including commands defining the operations of the CPUand may execute the read program. In other words, the function of the CPUmay be realized by software.
1 16 1 41 43 41 16 44 181 The endoscope devicemay generate point-cloud data on the basis of a 2D image of a subject, and the external device interfaceof the endoscope devicemay transmit the 2D image and the point-cloud data to the PC. The communication unitof the PCmay receive the 2D image and the point-cloud data from the external device interface. Therefore, the CPUdoes not need to include the generation unit.
41 In the eighth embodiment, the PCcan support correct designation of a point on the subject.
While preferred embodiments of the invention have been described and shown above, it should be understood that these are examples of the invention and are not to be considered as limiting. Additions, omissions, substitutions, and other modifications can be made without departing from the spirit or scope of the present invention. Accordingly, the invention is not to be considered as being limited by the foregoing description, and is only limited by the scope of the appended claims.
Cooperative Patent Classification codes for this invention. Click any code to explore related patents in that topic.
September 28, 2021
July 21, 2026
Browse 5M+ US patents with plain-English claim translations and AI-generated analysis.