Image Processing Apparatus and Image Processing Method for Aligning Polarized Images Based on a Depth Map and Acquiring a Polarization Characteristic Using the Aligned Polarized Images

1. An image processing apparatus comprising: a depth map generation unit configured to generate an integrated depth map indicating distance information on a subject from a plurality of polarized images obtained by picking up the subject at a plurality of different viewpoint positions; an alignment unit configured to align the plurality of polarized images obtained by picking up the subject at the plurality of different viewpoint positions through polarizing filters in at least four different polarization directions at the plurality of different viewpoint positions on the basis of the generated depth map indicating the distance information on the subject; a polarization characteristic acquisition unit configured to acquire a polarization characteristic of the subject from a desired viewpoint position by using the plurality of polarized images aligned by the alignment unit; and a normal map generation unit configured to generate a normal map of the subject at the desired viewpoint position on the basis of the polarization characteristic acquired by the polarization characteristic acquisition unit, wherein the depth map generation unit generates a plurality of depth maps including a respective depth map for each respective pair of a plurality of pairs of polarized images among the plurality of polarized images obtained by picking up the subject at the plurality of different viewpoint positions through the polarizing filters in the at least four different polarization directions and generates the integrated depth map by integrating the plurality of depth maps generated for the plurality of pairs of polarized images, wherein each respective depth map is generated by performing stereo matching processing on the respective pair of polarized images prior to integrating the plurality of depth maps, wherein the plurality of pairs of polarized images include pairs of each respective polarized image obtained using a polarizing filter in a respective polarization direction paired with only two other polarized images obtained using different polarization filters in different polarization directions among the at least four different polarization directions, and wherein the depth map generation unit, the alignment unit, the polarization characteristic acquisition unit, and the normal map generation unit are each implemented via at least one processor.

2. The image processing apparatus according to claim 1 , wherein the polarization characteristic acquisition unit acquires the polarization characteristic of the subject from the desired viewpoint position on the basis of polarization directions and luminance of the plurality of aligned polarized images.

3. The image processing apparatus according to claim 1 , wherein the normal map generation unit generates the normal map from which uncertainty of 180 degrees in polarization analysis is eliminated on the basis of the depth map used by the alignment unit.

4. The image processing apparatus according to claim 1 , further comprising a high-precision depth map generation unit configured to integrate the depth map used by the alignment unit and the normal map generated by the normal map generation unit to generate a depth map with higher precision than the depth map used by the alignment unit, wherein the high-precision depth map generation unit is implemented via at least one processor.

5. The image processing apparatus according to claim 1 , further comprising an image pickup unit configured to generate the plurality of polarized images by picking up the subject at the plurality of different viewpoint positions through the polarizing filters in different polarization directions at the different viewpoint positions.

6. The image processing apparatus according to claim 5 , wherein in the image pickup unit, image pickup sections are provided at the different plurality of viewpoint positions, and wherein the image pickup sections are provided with the polarizing filters in different polarization directions to generate the plurality of polarized images at the plurality of different viewpoint positions.

7. The image processing apparatus according to claim 5 , wherein in the image pickup unit, a plurality of lenses are arranged in a direction orthogonal to an optical axis direction on a light incident surface side of an imaging element, and wherein the lenses are provided with the polarizing filters in different polarization directions to generate the plurality of polarized images at the plurality of different viewpoint positions.

8. The image processing apparatus according to claim 5 , wherein the image pickup unit further comprises image pickup sections configured to pick up the subject at the plurality of different viewpoint positions without through the polarizing filters or through polarizing filters in an identical polarization direction to generate images.

9. The image processing apparatus according to claim 1 , further comprising a polarization characteristic use unit configured to perform image processing by using the polarization characteristic calculated by the polarization characteristic acquisition unit, wherein the polarization characteristic use unit is implemented via at least one processor.

10. The image processing apparatus according to claim 9 , wherein the polarization characteristic use unit uses the polarization characteristic calculated by the polarization characteristic acquisition unit to generate an image with a reflection component of the image at the desired viewpoint position adjusted.

11. The image processing apparatus according to claim 9 , wherein the polarization characteristic use unit calculates an image characteristic amount by using the polarization characteristic calculated by the polarization characteristic acquisition unit, and uses the image characteristic amount to perform processing in consideration of a surface shape of the subject.

12. An image processing method, executed by at least one processor, the method comprising: generating an integrated depth map indicating distance information on a subject from a plurality of polarized images obtained by picking up the subject at a plurality of different viewpoint positions; aligning the plurality of polarized images obtained by picking up the subject at the plurality of different viewpoint positions through polarizing filters in at least four different polarization directions at the plurality of different viewpoint positions on the basis of the generated depth map indicating the distance information on the subject; acquiring a polarization characteristic of the subject from a desired viewpoint position by using the plurality of aligned polarized images; and generating a normal map of the subject at the desired viewpoint position on the basis of the acquired polarization characteristic, wherein the integrated depth map is generated by generating a plurality of depth maps including a respective depth map for each respective pair of a plurality of pairs of polarized images among the plurality of polarized images obtained by picking up the subject at the plurality of different viewpoint positions through the polarizing filters in the at least four different polarization directions and integrating the plurality of depth maps generated for the plurality of pairs of polarized images, wherein each respective depth map is generated by performing stereo matching processing on the respective pair of polarized images prior to integrating the plurality of depth maps, and wherein the plurality of pairs of polarized images include pairs of each respective polarized image obtained using a polarizing filter in a respective polarization direction paired with only two other polarized images obtained using different polarization filters in different polarization directions among the at least four different polarization directions.