3D model storage stores the 3D shape data of a target object and illumination base data in advance. A comparison image generator generates, as a comparison image, a reproduced image with the target object being arranged in the position/posture of the current estimation value under the same illumination condition as that for the input image on the basis of the 3D shape data and illumination base data. An image displacement distribution detector segments the comparison image into sub regions and detects the image displacement distribution between the comparison image and the input image for each sub region. A posture difference calculator calculates a position/posture difference value on the basis of the image displacement distribution and 3D shape data. An end determinator outputs the current position/posture estimation value as an optimum position/posture estimation value when determining that the position/posture difference value is smaller than a predetermined threshold value.
Legal claims defining the scope of protection, as filed with the USPTO.
1. An estimation system for estimating an object state, comprising: image input means for inputting an input image containing an object whose state is to be estimated, the state being at least one of a position and posture; 3D shape data storage means for storing 3D shape data of the object; comparison image generation means for generating, as a comparison image, an image containing the object in a predetermined state by using the 3D shape data stored in said 3D shape data storage means; image displacement distribution detection means for segmenting the comparison image generated by the comparison image generation means into partial images each corresponding to a sub region with a predetermined size; image positional relationship detection means for detecting, for each sub region, a positional relationship between the input image and the comparison image generated by said comparison image generation means; correction amount calculation means for calculating a correction amount of the object state in the comparison image by using the positional relationship detected by said image positional relationship detection means; and state correction means for correcting the object state set in comparison image generation by said comparison image generation means by using the correction amount obtained by said correction amount calculation means, thereby calculating a new object state.
2. The estimation system for estimating an object state according to claim 1 , further comprising state determination means for determining on the basis of the correction amount obtained by said correction amount calculation means whether the object state set by said comparison image generation means is appropriate, wherein when it is determined that the object state is appropriate, the object state set by said comparison image generation means is output as an estimation value, and when it is determined that the object state is not appropriate, estimation processing including the comparison image generation processing by said comparison image generation means, the positional relationship detection processing by said image positional relationship detection means, and the correction amount calculation processing by said correction amount calculation means is executed again by setting the new object state calculated by said state correction means to the predetermined state.
3. The estimation system for estimating an object state according to claim 2 , wherein said state determination means determines that the object state is appropriate when the correction amount obtained by said correction amount calculation means is smaller than a predetermined amount, and determines that the object state is not appropriate when the correction amount is not smaller than the predetermined amount.
4. The estimation system for estimating an object state according to claim 2 , further comprising: first similarity calculation means for calculating a first similarity between the comparison image and the input image after the estimation processing is executed again; and second similarity calculation means for calculating a second similarity between the comparison image and the input image before the estimation processing is executed again, wherein said state determination means compares the first similarity with the second similarity, determines that the object state is not appropriate when the first similarity is higher than the second similarity, and determines that the object state is appropriate when the first similarity is not higher than the second similarity.
5. The estimation system for estimating an object state according to claim 1 , wherein said image input means comprises means for inputting a moving image containing an object, and said image positional relationship detection means uses a latest frame image of the moving image as the input image.
6. The estimation system for estimating an object state according to claim 1 , wherein said comparison image generation means comprises: means for reproducing a luminance value of an object surface, which changes depending on an illumination condition; and means for generating the comparison image under an illumination condition close to that for the input image by using the reproduced luminance value.
7. The estimation system for estimating an object state according to claim 6 , further comprising illumination base image group storage means for storing an illumination base image group representing a variation in luminance of the object surface depending on the illumination condition, wherein said comparison image generation means reproduces the luminance value of the object surface by using the illumination base image group stored in said illumination base image group storage means.
8. The estimation system for estimating an object state according to claim 7 , further comprising: 3D shape measuring means for measuring the 3D shape data of the object and reflectance data of the object surface; and illumination base calculation means for calculating an illumination base image representing the variation in luminance of the object surface depending on the illumination condition by using the 3D shape data and the reflectance data of the object surface which are measured by said 3D shape measuring means.
9. The estimation system for estimating an object state according to claim 8 , wherein said illumination base calculation means calculates a vector group representing the luminance value of each point of the 3D shape data under a plurality of illumination conditions, obtains a base vector group in descending order of eigenvalues by principal component analysis of the vector group, and outputs the base vector group as the illumination base image group, and said comparison image generation means obtains, by using the 3D shape data of the object, a correspondence between each point of the 3D shape data of the object and a pixel of the image with the object being in an estimation value at current time, generates, by using the correspondence, an image illumination base group in which the illumination base image group is projected to the image with the object being in the estimation value, and generates, as the comparison image, an image nearest to the input image by linear connection of the image illumination base group.
10. The estimation system for estimating an object state according to claim 1 , wherein said correction amount calculation means calculates, as the correction amount, a 3D motion of the object which causes a moving amount of an object part corresponding to each sub region in the comparison image to be near to an image displacement distribution by using the 3D shape data of the object and the image displacement distribution representing the positional relationship between the comparison image and the input image for each sub region.
11. The estimation system for estimating an object state according to claim 1 , further comprising feature extraction means for extracting an image feature amount of each of the input image and comparison image on the basis of luminance values of the input image and the comparison image generated by said comparison image generation means, wherein said image positional relationship detection means detects the positional relationship between the input image and the comparison image for each sub region on the basis of the image feature amount extracted by said feature extraction means.
12. An estimation method of estimating an object state using a computerized estimating system that includes a processor, an image inputter, and a 3D shape measurer, the method comprising the steps of: inputting, using the image inputter, an input image containing an object whose state is to be estimated, the state being at least one of a position and posture; generating, by the processor, as a comparison image, an image containing the object in a predetermined state by using 3D shape data of the object measured by the 3D shape measurer; segmenting the comparison image generated by the generating step into partial images each corresponding to a sub region with a predetermined size; detecting, by the processor, for each sub region, a positional relationship between the comparison image and the input image; calculating, by the processor, a correction amount of the object state in the comparison image by using the detected positional relationship; and correcting, by the processor, the object state set in comparison image generation by using the calculated correction amount, thereby calculating a new object state.
13. The estimation method of estimating an object state according to claim 12 , further comprising the steps of: determining, by the processor, on the basis of the calculated correction amount whether the object state set in comparison image generation is appropriate; and outputting, by the processor, the object state set in comparison image generation as an estimation value when it is determined that the object state is appropriate, wherein when it is determined that the object state is not appropriate, estimation processing including the step of generating the comparison image, the step of detecting the positional relationship, and the step of calculating the correction amount is executed again by setting the calculated new object state to the predetermined state.
14. The estimation method of estimating an object state according to claim 13 , wherein in the determination step, it is determined that the object state is appropriate when the correction amount is smaller than a predetermined amount, and it is determined that the object state is not appropriate when the correction amount is not smaller than the predetermined amount.
15. The estimation method of estimating an object state according to claim 13 , further comprising the steps of: calculating, by the processor, a first similarity between the comparison image and the input image after the estimation processing is executed again; and calculating, by the processor, a second similarity between the comparison image and the input image before the estimation processing is executed again, wherein in the determination step, the first similarity is compared with the second similarity, it is determined that the object state is not appropriate when the first similarity is higher than the second similarity, and it is determined that the object state is appropriate when the first similarity is not higher than the second similarity.
16. The estimation method of estimating an object state according to claim 12 , wherein in the step of inputting the image, a moving image containing an object is input, and in the step of detecting the positional relationship, a latest frame image of the moving image is used as the input image.
17. The estimation method of estimating an object state according to claim 12 , wherein the step of generating the comparison image comprises the steps of: reproducing a luminance value of an object surface, which changes depending on an illumination condition; and generating the comparison image under an illumination condition close to that for the input image by using the reproduced luminance value.
18. The estimation method of estimating an object state according to claim 17 , wherein in the step of generating the comparison image, the luminance value of the object surface is reproduced by using an illumination base image group representing a variation in luminance of the object surface depending on the illumination condition.
19. The estimation method of estimating an object state according to claim 18 , further comprising the steps of: measuring, by the 3D shape measurer, the 3D shape data of the object and reflectance data of the object surface; and calculating, by the processor, an illumination base image representing the variation in luminance of the object surface depending on the illumination condition by using the 3D shape data and the reflectance data of the object surface.
20. The estimation method of estimating an object state according to claim 19 , wherein: in the step of calculating the illumination base image, a vector group representing the luminance value of each point of the 3D shape data under a plurality of illumination conditions is calculated, a base vector group is obtained in descending order of eigenvalues by principal component analysis of the vector group, and the base vector group is output as the illumination base image group, and in the step of generating the comparison image, a correspondence between each point of the 3D shape data of the object and a pixel of the image with the object being in an estimation value at current time is obtained by using the 3D shape data of the object, an image illumination base group in which the illumination base image group is projected to the image with the object being in the estimation value is generated by using the correspondence, and an image nearest to the input image is generated as the comparison image by linear connection of the image illumination base group.
21. The estimation method of estimating an object state according to claim 12 , wherein in the step of processor calculating the correction amount, a 3D motion of the object which causes a moving amount of an object part corresponding to each sub region in the comparison image to be near to an image displacement distribution is calculated as the correction amount by using the 3D shape data of the object and the image displacement distribution representing the positional relationship between the comparison image and the input image for each sub region.
22. The estimation method of estimating an object state according to claim 12 , further comprising the step of the processor extracting an image feature amount of each of the comparison image and input image on the basis of luminance values of the comparison image and input image, wherein in the step of detecting the positional relationship, the positional relationship between the input image and the comparison image for each sub region is detected on the basis of the image feature amount.
23. An estimation program for estimating an object state residing on a non-transitory computer recording medium, which causes a computer to execute the steps of: inputting an input image containing an object whose state is to be estimated, the state being at least one of a position and posture; generating, as a comparison image, an image containing the object in a predetermined state by using 3D shape data of the object; segmenting the comparison image generated by the generating step into partial images each corresponding to a sub region with a predetermined size; detecting, for each sub region, a positional relationship between the comparison image and the input image; calculating a correction amount of the object state in the comparison image by using the detected positional relationship; and correcting the object state set in comparison image generation by using the calculated correction amount, thereby calculating a new object state.
24. The estimation program for estimating an object state residing on a computer recording medium according to claim 23 , which causes the computer to further execute the steps of: determining on the basis of the calculated correction amount whether the object state set in comparison image generation is appropriate; outputting the object state set in comparison image generation as an estimation value when it is determined that the object state is appropriate; and executing again estimation processing including the step of generating the comparison image, the step of detecting the positional relationship, and the step of calculating the correction amount by setting the calculated new object state to the predetermined state when it is determined that the object state is not appropriate.
25. The estimation program for estimating an object state residing on a computer recording medium according to claim 24 , which causes the computer to execute, as the determination step, the step of determining that the object state is appropriate when the correction amount is smaller than a predetermined amount, and determining that the object state is not appropriate when the correction amount is not smaller than the predetermined amount.
26. The estimation program for estimating an object state residing on a computer recording medium according to claim 24 , which causes the computer to further execute: the step of calculating a first similarity between the comparison image and the input image after the estimation processing is executed again; the step of calculating a second similarity between the comparison image and the input image before the estimation processing is executed again; and as the determination step, the step of comparing the first similarity with the second similarity, determining that the object state is not appropriate when the first similarity is higher than the second similarity, and determining that the object state is appropriate when the first similarity is not higher than the second similarity.
27. The estimation program for estimating an object state residing on a computer recording medium according to claim 23 , which causes the computer to execute: as the step of inputting the image, the step of inputting a moving image containing an object; and as the step of detecting the positional relationship, the step of using a latest frame image of the moving image as the input image.
28. The estimation program for estimating an object state residing on a computer recording medium according to claim 23 , which causes the computer to execute, in the step of generating the comparison image, the steps of: reproducing a luminance value of an object surface, which changes depending on an illumination condition; and generating the comparison image under an illumination condition close to that for the input image by using the reproduced luminance value.
29. The estimation program for estimating an object state residing on a computer recording medium according to claim 28 , which causes the computer to execute, as the step of generating the comparison image, the step of reproducing the luminance value of the object surface by using an illumination base image group representing a variation in luminance of the object surface depending on the illumination condition.
30. The estimation program for estimating an object state residing on a computer recording medium according to claim 29 , which causes the computer to further execute the steps of: measuring the 3D shape data of the object and reflectance data of the object surface; and calculating an illumination base image representing the variation in luminance of the object surface depending on the illumination condition by using the 3D shape data and the reflectance data of the object surface.
31. The estimation program for estimating an object state residing on a computer recording medium according to claim 30 , which causes the computer to execute: as the step of calculating the illumination base image, the step of calculating a vector group representing the luminance value of each point of the 3D shape data under a plurality of illumination conditions, obtaining a base vector group in descending order of eigenvalues by principal component analysis of the vector group, and outputting the base vector group as the illumination base image group, and as the step of generating the comparison image, the step of obtaining a correspondence between each point of the 3D shape data of the object and a pixel of the image with the object being in an estimation value at current time by using the 3D shape data of the object, generating an image illumination base group in which the illumination base image group is projected to the image with the object being in the estimation value by using the correspondence, and generating, as the comparison image, an image nearest to the input image by linear connection of the image illumination base group.
32. The estimation program for estimating an object state residing on a computer recording medium according to claim 23 , which causes the computer to execute, as the step of calculating the correction amount, the step of calculating, as the correction amount, a 3D motion of the object which causes a moving amount of an object part corresponding to each sub region in the comparison image to be near to an image displacement distribution by using the 3D shape data of the object and the image displacement distribution representing the positional relationship between the comparison image and the input image for each sub region.
33. The estimation program for estimating an object state residing on a computer recording medium according to claim 23 , which causes the computer to further execute: the step of extracting an image feature amount of each of the comparison image and input image on the basis of luminance values of the comparison image and input image; and as the step of detecting the positional relationship, the step of detecting the positional relationship between the input image and the comparison image for each sub region on the basis of the image feature amount.
Cooperative Patent Classification codes for this invention. Click any code to explore related patents in that topic.
October 28, 2004
October 5, 2010
Browse 5M+ US patents with plain-English claim translations and AI-generated analysis.