Image processing apparatus, imaging apparatus, image processing method

This patent application is based on and claims priority pursuant to 35 U.S.C. § 119(a) to Japanese Patent Application No. 2022-162948, filed on Oct. 11, 2022, in the Japan Patent Office, the entire disclosure of which is hereby incorporated by reference herein.

The present disclosure relates to an image processing technology, and more particularly, to an image processing apparatus, an imaging apparatus, and an image processing method.

Imaging apparatuses that capture an image in a range of 360 degrees, such as a panoramic image or a spherical image, are known. Such an imaging apparatus captures images using a plurality of wide-angle lenses or fisheye lenses. The images are stitched together, and a single image is generated, accordingly. In an image stitching technology according to a related art, distortion correction and projective transformation are performed on each of a plurality of captured images, and a stitching position where objects overlap is detected in an overlapping region of the processed images, by, for example, pattern matching.

In addition, a technology of stitching images by fixing an object distance for stitching without detecting a stitching position in order to increase the processing speed is known.

In such a related technology, an image processing apparatus includes an image input unit to which a first image and a second image are input, a storage unit that stores stitching positions of a plurality of parts between the images input to the image input unit, an evaluation unit that evaluates efficiency of previous stitching positions stored in the storage unit for each of the plurality of parts, and an image stitching processing unit that executes stitching processing based on the first image and the second image that are input to the image input unit based on the previous stitching positions, which are stored in the storage unit, for parts whose efficiency satisfies a criterion.

According to an embodiment of the present disclosure, an image processing apparatus includes circuitry to input at least a first captured image and a second captured image. The first captured image and the second captured image have an overlapping area. The overlapping area in the first captured image has a plurality of target areas in each of which a stitching position is to be obtained. The plurality of target areas is arranged in an orthogonal direction with respect to a direction in which the first captured image and the second captured image are arranged and have a corresponding plurality of sizes different from each other.

According to an embodiment of the present disclosure, an imaging apparatus includes a first image capturing device to capture a first image in a first direction to generate a first captured image; and a second image capturing device to capture a second image in a second direction to generate a second captured image. The first direction and the second direction are different from each other. The first captured image and the second captured image have an overlapping area. The overlapping area in the first captured image has a plurality of target areas in each of which a stitching position is to be obtained. The plurality of target areas is arranged in an orthogonal direction with respect to a direction in which the first captured image and the second captured image are arranged and have a corresponding plurality of sizes different from each other.

According to an embodiment of the disclosure, an image processing method includes receiving an input of a first captured image and an input of a second captured image and obtaining, in the first captured image, a stitching position in an overlapping area of the first captured image and the second captured image. The overlapping area in the first captured image has a plurality of target areas in each of which a stitching position is to be obtained. The plurality of target areas is arranged in an orthogonal direction with respect to a direction in which the first captured image and the second captured image are arranged and has a corresponding plurality of sizes different from each other.

The accompanying drawings are intended to depict embodiments of the present disclosure and should not be interpreted to limit the scope thereof. The accompanying drawings are not to be considered as drawn to scale unless explicitly noted. Also, identical or similar reference numerals designate identical or similar components throughout the several views.

In describing embodiments illustrated in the drawings, specific terminology is employed for the sake of clarity. However, the disclosure of this specification is not intended to be limited to the specific terminology so selected and it is to be understood that each specific element includes all technical equivalents that have a similar function, operate in a similar manner, and achieve a similar result.

Referring now to the drawings, embodiments of the present disclosure are described below. As used herein, the singular forms “a,” “an,” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise.

An embodiment according to the present disclosure is described below. However, the present disclosure is not limited to the embodiment described below. In the following description of the embodiment, a spherical imaging deviceis used as an example of an image processing apparatus, an image processing system, or an imaging apparatus. The spherical imaging deviceincludes an imaging body including two fisheye lenses in an optical system and has an image processing function of performing distortion correction and projective transformation on two partial images captured by the two fisheye lenses and combining the images to generate a spherical image.

Overall Configuration

The overall configuration of the spherical imaging deviceaccording to the present embodiment is described below with reference to() to.is an external view of the spherical imaging deviceaccording to the present embodiment. The spherical imaging deviceillustrated inincludes an imaging body, two image forming optical systemsA andB provided in the imaging body, a housingthat holds the imaging bodyand components such as a controller, and an operation buttonprovided on the housing.

is an external view of the spherical imaging devicewith the imaging bodystored in the housing.

In order to prevent dust and dirt on the two image forming optical systemsA andB, the imaging bodyis vertically movable and can be stored in the housingwhen not in use.is an external view of the spherical imaging deviceas viewed from above. The spherical imaging devicehas a finger groove, and the imaging bodycan be manually moved up and down by a finger placed through the finger groove.

are cross-sectional views of the spherical imaging deviceaccording to the present embodiment.illustrates the spherical imaging devicewith the imaging bodyexpanding from the housing.illustrates the spherical imaging devicewith the imaging bodystored in the housing. The imaging bodyillustrated inincludes the two image forming optical systemsA andB and two solid-state imaging elementsA andB such as charge-coupled device (CCD) sensors and complementary metal oxide semiconductor (CMOS) sensors. A combination of one of the image forming optical systemsand one of the solid-state imaging elementsis referred to as an imaging optical system. Each of the image forming optical systemscan be configured as, for example, a fisheye lens with six groups and seven elements. In the present embodiment illustrated in, the fish eye lens has a total angle of view greater than 180 degrees (=360 degrees/n; n=2), preferably has an angle of view of 185 degrees or more, and more preferably has an angle of view of 190 degrees or more.

The positional relationship between the optical elements (lenses, prisms, filters, and aperture stops) of the two image forming optical systemsA andB is determined with respect to the solid-state imaging elementsA andB. The positioning is performed such that the optical axes of the optical elements of each of the image forming optical systemsA andB is positioned so as to be orthogonal to the center of the light receiving area of corresponding one of the solid-state imaging elements, and such that the light receiving area serves as the imaging plane of corresponding one of the fisheye lenses. Each of the solid-state imaging elementsis a two-dimensional solid-state imaging element in which a light receiving area forms an area, and converts light condensed by corresponding one of the image forming optical systems, which is combined, into an image signal.

In the present embodiment illustrated in, the image forming optical systemsA andB have the same specifications, and are set to be combined in opposite directions so that the optical axes coincide with each other. The solid-state imaging elementsA andB convert the received light distribution into image signals and output the image signals to an image processing device on controllersA andB. The image processing device stitches and combines the partial images input from the solid-state imaging elementsA andB to generate an image with a solid angle of 4π steradians (in the following description, the generated image is referred to as a “spherical image”). The spherical image is obtained by capturing all directions that can be seen from an image capturing point. In the present embodiment illustrated in, the spherical image is generated, but an image obtained by capturing a horizontal plane in 360 degrees, a so-called panoramic image, may be generated. In the description of the present embodiment, an image in which a partial area of the spherical image is missing is also referred to a panoramic image for the sake of convenience. For example, the panoramic image is an image of 360 degrees in the horizontal direction and less than 180 degrees in the vertical direction, and includes an image in which a partial area is missing in the direction directly above or directly below the spherical imaging device, an image in which a partial area is missing in a vertically upward direction or a vertically downward direction of the spherical image, and an image in which a part of a predetermined area of the spherical image is missing. Regarding this, it is considered that a user usually does not carefully view a part that is directly above (for example, a ceiling of a room) or directly below (for example, ground) of an object captured in the spherical image when the user views a 4π steradian image, for example. In such a case, the imaging bodyand the housingmay be designed such that the above-described part is not to be captured. Alternatively, the above-described part may not be displayed, or the above-above described part may be displayed with a predetermined logo or information, such as a link to a predetermined web site, in a manner that such information is displayed on the above-described part in a superimposed manner, so that a 4π steradian image may not be displayed as it is.

The housingof the spherical imaging deviceincludes the controllersA andB that perform input of control information from the operation button, execution of image processing based on image signals of the solid-state imaging elementsA andB, and input and output of a processing result, for example. The controllersA andB are provided with connectors(connectorsA,B,C). The controllersor the controllersand the solid-state imaging elements(solid-state imaging elementsA andB) are connected through a cable(cablesA andB) and the connector. The cableis a flexible flat cable (FFC) or a flexible printed circuit (FPC), for example.

is a cross-sectional view of the spherical imaging devicewith the imaging bodystored in the housing. An elastic membersuch as rubber or sponge is provided to prevent the two image forming optical systems(A andB) from being damaged and to prevent dust from entering the housingwhen the imaging bodyis stored in or expanded from the housing.

is a block diagram illustrating a hardware configuration of the spherical imaging deviceaccording to the present embodiment. The spherical imaging deviceincludes a digital still camera processor (hereinafter, referred to simply as a processor), a lens barrel unit, and various components connected to the processor. The lens barrel unitincludes the above-described two pairs of the image forming optical systemsand the solid-state imaging elements, namely the pair of the image forming optical systemA and the solid-state imaging elementA, and the other pair of the image forming optical systemB and the solid-state imaging elementB. The solid-state imaging elementis controlled by a control command from a central processing unit (CPU), which is described later, in the processor.

The processorincludes an image signal processor (ISP), a direct memory access controller (DMAC), an arbiter (ARBMEMC)for arbitrating memory access, a memory controller (MEMC)for controlling memory access, and a distortion correction/image stitching block. The ISPsA andB perform white balance setting and gamma setting on the image data processed by the solid-state imaging elementsA andB, respectively. A synchronous dynamic random access memory (SDRAM)is connected to the MEMC. In the SDRAM, data is temporarily stored when the ISPA orB, or the distortion correction/image stitching blockperforms processing. The distortion correction/image stitching blockperforms distortion correction and vertical perspective correction on the two partial images obtained from the two imaging optical systems using information from a triaxial acceleration sensor, and combines, or stitches, the images.

The processorfurther includes a DMAC, an image processing block, the CPU, an image data transfer unit, a synchronous dynamic random access memory controller (SDRAMC), a memory card control block, a universal serial bus block, a peripheral block, an audio unit, a serial block, a liquid crystal display (LCD) driver, and a bridge.

The CPUcontrols the operation of each component of the spherical imaging device. The image processing blockperforms various types of image processing on the image data using a resize block, a still image compression block, or a moving image compression block, for example. The resize blockis a block for enlarging or reducing an image data size by interpolation processing. The still image compression blockis a codec block that compresses or expands a still image into a still image format such as Joint photographic experts group (JPEG) or tagged image file format (TIFF). The moving image compression blockis a codec block that compresses or decompresses a moving image in a moving image format of moving picture experts group (MPEG)-4 advanced video coding (AVC)/H.264, for example. The image data transfer unittransfers the image processed by the image processing block. The SDRAMCcontrols an SDRAMconnected to the processor, and the SDRAMtemporarily stores the image data when the image data is processed in various ways in the processor.

The memory card control blockcontrols reading and writing with respect to a memory card or a flash read only memory (ROM)inserted into a memory card slot. The memory card slotis a slot that allows a memory card to be detachably attached to the spherical imaging device. A universal serial bus (USB) blockcontrols USB communication with an external device such as a personal computer connected via a USB connector. A power switchis connected to the peripheral block. The audio unitis connected to a microphonethrough which an audio signal is input and a speakerthrough which a recorded audio signal is output, and controls audio input and output. The serial blockcontrols serial communication with an external device such as a personal computer, and is connected to a wireless network interface card (NIC). The LCD driveris a drive circuit for driving an LCD monitorand converts the signal into a signal for displaying various states on the LCD monitor.

The flash ROMstores a control program described in a code that can be decoded by the CPUand various parameters. When the power switchis operated to turn on the power supply, the control program is loaded into a main memory. The CPUcontrols the operation of each component of the device in accordance with the program read into the main memory, and temporarily stores information used for the control in the SDRAMand a local static random access memory (SRAM).

is a block diagram illustrating a hardware configuration of an information terminalthat can be used to control the spherical imaging deviceaccording to the present embodiment. The information terminalillustrated inincludes a CPU, a random access memory (RAM), a hard disk drive (HDD), an input device, external storage, a display, a wireless NIC, a USB connector, and a HIGH-DEFINITION MULTIMEDIA INTERFACE (HDMI) connector(HDMI is a registered trademark).

The CPUcontrols the operation of each component and the overall operation of the information terminal. The RAMprovides a working area for the CPU. The HDDstores programs, which are described in a code that can be decoded by the CPU, and such programs include an operating system and an application for processing performed on the information terminalaccording to the present embodiment.

The input deviceis an input device such as a mouse, a keyboard, a touch pad, or a touch screen, and provides a user interface. The external storageis a removable recording medium attached to, for example, a memory card slot, and records various types of data such as moving image data or still image data. The wireless NICestablishes connection of wireless local area network (LAN) communication with an external device such as the spherical imaging device. The USB connectorestablishes a USB connection with an external device such as the spherical imaging device. Although each of the wireless NICand the USB connectoris illustrated as an example, the present embodiment is not limited to a specific standard, and the external device may be connected by other wireless communication such as BLUETOOTH (registered trademark) or a wireless USB, or wired communication such as a wired LAN.

The displaydisplays an operation screen to be operated by a user, displays a monitor image of an image captured by the spherical imaging devicebefore or during image capturing, and displays a stored moving image or still image to be played back or viewed. With the displayand the input device, the user can instruct the spherical imaging deviceto capture an image and change various settings via the operation screen. In addition, the videos can be output to a display device such as an external display or a projector by the HDMI (registered trademark) connector. Although the HDMI (registered trademark) connectoris illustrated as an example, the present embodiment is not limited to a specific standard, and may use a configuration with DISPLAYPORT (registered trademark) or digital visual interface (DVI).

When the information terminalis powered on, the program is read from the ROM or the HDDand loaded into the RAM. The CPUcontrols the operation of each component of the device in accordance with the program read into the RAM, and temporarily stores information used for the control in a memory. Accordingly, the information terminalimplements functional units and processing, which are described later.

Image Stitching for Moving Image

A spherical-moving-image stitching function of the spherical imaging deviceis described below in detail with reference to.is a block diagram illustrating functional units for image stitching for a spherical moving image, implemented on the spherical imaging deviceaccording to the present embodiment. As illustrated in, an image processing blockincludes an image stitching distortion correction unit, an image stitching unit, a stitching position detection unit, and a table correction unit. The image stitching distortion correction unitis a distortion correction unit for image stitching.

The image processing blockreceives two partial images for each frame from the two solid-state imaging elementsA andB through various image signal processing. In the description of the present embodiment, an image for a frame and of which a source is one of the solid-state imaging elements (for example, the solid-state imaging elementA) is referred to as a “partial image F,” and another image for the frame, and of which a source is the other one of the solid-state imaging elements (for example, the solid-state imaging elementB) is referred to as a “partial image R.” In the present embodiment, one of the two solid-state imaging elementsA andB serves as a first imaging element for capturing an image in a first direction and generating a first image, and the other serves as a second imaging element for capturing an image in a second direction that is different from the first direction (for example, a direction different from the first direction by 180°) and generating a second image. The image processing blockis further provided with a distortion correction conversion tablegenerated in advance in a manufacture, in accordance with a predetermined projection model based on design data of an optical system such as each lens, for example. The distortion correction conversion tableis a conversion table for distortion correction.

The image stitching distortion correction unitperforms distortion correction on the partial image F and the partial image R, which are input, using an image stitching conversion tableto generate a corrected image F and a corrected image R. The image stitching conversion tableis a conversion table for image stitching. The input partial images F and R are image data represented by a plane coordinate system (x, y). The corrected images F and R obtained after the distortion correction using the image stitching conversion tableare image data in a spherical image format represented by a spherical coordinate system (a polar coordinate system having a radius vector of 1 and two arguments θ and φ). In the present embodiment, the image stitching distortion correction unitserves as a distortion correction unit. The distortion correction conversion tableand the image stitching conversion tableare described later.

Projection Method

are diagrams each illustrating a projection relation in the spherical imaging deviceaccording to the present embodiment. An image captured by one fisheye lens is obtained by capturing an image of an orientation corresponding to approximately a hemisphere from the image capturing point. As illustrated in, with the fisheye lens, an image having an image height h that corresponds to an angle of incidence φ with respect to an optical axis. The relation between the image height h and the angle of incidence φ is determined by a projection function according to a prescribed projection model. The projection function varies according to the properties and characteristics of the fisheye lens. Examples of the projection model include the equidistant projection (h=f×φ), the central projection (h=f·tan φ), the stereographic projection (h=2f·tan (φ/2)), the equi-solid-angle projection (h=2f·sin (φ/2)), and the orthogonal projection (h=f·sin φ). In any of the projections, the image height h of a formed image is determined according to the incident angle φ and the focal length f relative to the optical axis. Further, in the present embodiment, it is assumed that a so-called circular fisheye lens having an image circle diameter shorter than a diagonal line of an image is adopted. As illustrated in, a partial-view image obtained with such a lens is a planar image, which includes the entire image circle in which a part of the imaging range corresponding nearly to a hemisphere is projected.

Spherical Image Format

are diagrams each illustrating a data structure of image data in a spherical image format, according to the present embodiment. As illustrated in, the image data in the spherical image format is expressed as an array of pixel values where the vertical angle φ corresponding to the angle with respect to a certain axis and the horizontal angle θ corresponding to the angle of rotation around the axis are the coordinates. The horizontal angle θ ranges from 0 to 360 degrees (alternatively, expressed as from −180 to +180 degrees). In a similar manner, the vertical angle φ ranges from 0 to 180 degrees (alternatively, expressed as from −90 to +90 degrees). The respective coordinate values (θ, φ) are associated with the points on the spherical surface representing all directions from the image capturing point. Thus, the all directions are mapped on the spherical image. The planar coordinates of an image that is captured by a fisheye lens can be associated with the coordinates on the spherical surface in the spherical image format by using the projection function as described above with reference to.

Conversion Table

are diagrams for describing a conversion table referred by the image stitching distortion correction unitaccording to the present embodiment. The distortion correction conversion tableand the image stitching conversion tabledefine a projection to an image represented by the spherical coordinate system based on a partial image represented by the plane coordinate system. As illustrated in, the distortion correction conversion tableand the image stitching conversion tablehold information in which coordinate values (θ, φ) of a post-correction image and coordinate values (x, y) of a pre-correction partial image that is to be mapped on the coordinate values (θ, φ) are associated with each other, for all the coordinate values (θ, φ), where θ denotes 0 to 360 degrees and φ denotes 0 to 180 degrees. In the example illustrated in, the angle of each one of the pixels is one-tenth of a degree in both φ direction and θ direction. Each of the distortion correction conversion tableand the image stitching conversion tablehas information indicating a correspondence relationship of 3600×1800 for each fisheye lens.

The distortion correction conversion tableis calculated and tabulated after correcting distortion from an ideal lens model in a manufacture in advance, for example. The initial value of the image stitching conversion tableis the distortion correction conversion table, and is used for the first frame and several frames subsequent to the first frame in sequential order. For further subsequent frames after the several frames, the table correction unituses the image stitching conversion tablethat is updated by using the previous image stitching conversion tablethat is used last time and a stitching position detection result obtained by the stitching position detection unit. A detailed description of this is given later.

Corrected Image

Although details are described later, the stitching position detection unitreceives input of the corrected images F and R that are converted by the image stitching distortion correction unit, detects a stitching position for stitching the input corrected images F and R by pattern matching processing, and generates stitching position information. The table correction unitupdates the image stitching conversion tablebased on the stitching position information and the previous image stitching conversion tablethat is used last time.

is a diagram for describing mapping of two partial images captured by two fisheye lenses to a spherical coordinate system in stitching position detection and image stitching according to the present embodiment.

As illustrated in, the horizontal angle and the vertical angle are defined based on the axis perpendicular to the optical axis.

As a result of processing of the image stitching distortion correction unit, the two partial images F and R captured by the fisheye lenses are expanded on a spherical image format as illustrated in. The partial image F captured by a fisheye lens F is typically mapped substantially the left hemisphere of the entire celestial sphere, and the partial image R captured by a fisheye lens R is mapped to substantially the right hemisphere of the entire celestial sphere.

The image stitching unitcombines, or stitches, the corrected image F and the corrected image R to generate a frame of a stitched image. The stitched imagemay be referred to as a combined image. The image stitching unitserves as an image stitching processing unit that combines, or stitches, the corrected image F and the corrected image R, which are input, based on the stitching position obtained by the stitching position detection unit. The stitched imageis an image in a spherical image format and is an image represented by a polar coordinate system having two arguments. In the present embodiment, the image stitching unitis provided in the image processing blockimplemented on the spherical imaging device. However, in some embodiments, the image stitching unitmay be provided in an external device. For example, the processing up to the generation of the corrected image may be performed on the spherical imaging device, and processing of image stitching may be performed on another terminal (for example, the information terminal). In this case, an image processing system is configured by combining the spherical imaging deviceand the information terminalthat includes the image stitching unit.

Moving Image Streaming