Image Correction Apparatus That Performs Color Matching Between Multiple Image Pickup Apparatuses That Take Image of Display Apparatus, Image Pickup System, Control Method, and Storage Medium

This application is a continuation of U.S. application Ser. No. 18/181,861, filed Mar. 10, 2023, the entire disclosure of which is hereby incorporated by reference.

The present invention relates to an image correction apparatus, an image pickup system, a control method, and a storage medium, and in particular to an image correction apparatus that performs color matching between a plurality of image pickup apparatuses that takes an image of a display apparatus, an image pickup system, a control method, and a storage medium.

In video production, images are taken using a plurality of cameras, and in some cases, the color of the same subject in obtained images varies with the cameras due to individual differences between the cameras, differences in lenses to be used, and differences in imaging conditions such as a lighting environment. For this reason, there is a growing demand for brightness and color matching (hereafter referred to as “color matching”) between the cameras.

In recent years, there have been increasing use cases for virtual production that uses an LED display wall, whose large screen consists of high-definition LED displays arranged in a plane direction, as the background and creates an image that looks as if it was taken at a desired location.

In such use cases, along with color matching for a real object performed in advance using a color chart, color matching is performed using a color chart displayed on an LED display wall at the shooting site. The reason why color matching is performed not only for the real object but also for the LED display wall is that color shift in the LED displays whose spectral characteristic has a peak, tends to be visually recognized due to differences in characteristics between camera's color filters.

Moreover, when an image is taken using a plurality of cameras, some of the cameras shoot the LED display wall at an angle, and hence the cameras need to be adjusted according to environments at respective installation locations.

Japanese Laid-Open Patent Publication (Kokai) No. 2002-135790 discloses a technique in which a picked-up image obtained by taking an image of a subject and a re-picked-up image obtained by taking an image of a display apparatus that displays the picked-up image are compared with each other, a correction amount is calculated based on a comparison result, and the picked-up image is corrected by applying the correction amount to data of the picked-up image.

Japanese Laid-Open Patent Publication (Kokai) No. 2012-68364 discloses a technique of adjusting the color balance of an illumination light source in a projector by making the balance of a color signal value obtained by taking a reference image projected on an actually-used screen close to a reference color signal value obtained by taking a reference image projected on a reference screen.

The prior arts (Japanese Laid-Open Patent Publication (Kokai) No. 2002-135790, Japanese Laid-Open Patent Publication (Kokai) No. 2012-68364), however, have the problem that the accuracy of color matching between the plurality of cameras is low because no consideration is given to uneven color and brightness visually recognized when the LED displays are viewed at an angle (viewing angle characteristics).

The present invention provides an image correction apparatus that is capable of improving the accuracy of color matching between a plurality of cameras, an image pickup system, a control method, and a storage medium.

Accordingly, the present invention provides an image correction apparatus that performs color matching between a plurality of image pickup apparatuses using a plurality of picked-up images generated by displaying a display image on a display screen of a display unit and causing the plurality of images pickup apparatuses to shoot the display image, the image correction apparatus comprising at least one memory that stores a set of instructions, and at least one processor that is configured to, based on the instructions, cause the image correction apparatus to: store viewing angle correction values for use in correcting viewing angle characteristics resulting from positions of the plurality of image pickup apparatuses with respect to the display screen of the display unit, to a predetermined viewing angle characteristic; subject the display images in the plurality of picked-up images to correction using the viewing angle correction values to generate a plurality of corrected images; and calculate color correction values for use in color matching between the plurality of corrected images.

Accordingly, the present invention provides an image pickup system that includes a plurality of image pickup apparatuses, a display apparatus, and an image correction apparatus communicatively connected to the image pickup apparatuses and the display apparatus, the image pickup system comprising at least one memory that stores a set of instructions, and at least one processor that is configured to, based on the instructions, cause each of the plurality of image pickup apparatuses to: pick up a display image displayed on a display surface of a display unit in accordance with a shooting instruction and generate a first picked-up image; and subject the generated first picked-up image to correction based on color/brightness correction values, wherein each of the plurality of image pickup apparatuses receives the shooting instruction, sends the first picked-up image to the image correction apparatus, receives the color/brightness correction values, and sends a second picked-up image obtained by subjecting the first picked-up image to the correction to the image correction apparatus, and cause the image correction apparatus to: store viewing angle correction values for use in correcting viewing angle characteristics resulting from positions of the plurality of image pickup apparatuses with respect to the display screen of the display unit to a predetermined viewing angle characteristic; and after displaying the display image on the display screen of the display unit, send shooting instruction to the plurality of image pickup apparatuses, receive the plurality of first picked-up images sent from the plurality of image pickup apparatuses, send the color/brightness correction values to the plurality of image pickup apparatuses, and receive the plurality of second picked-up images sent from the plurality of image pickup apparatuses, wherein the image correction apparatus upon receiving the plurality of first picked-up images, generates a plurality of corrected images by subjecting the plurality of first picked-up images to correction using the viewing angle correction values, calculates color correction values for use in color matching between the plurality of corrected images, calculates color/brightness correction values for use in color/brightness matching between the plurality of first picked-up images using the color correction values and the viewing angle correction values, and upon receiving the plurality of second picked-up images sent from the plurality of image pickup apparatuses, generates an image including the plurality of second picked-up images and sends an instruction to display the generated image to the display apparatus.

According to the present invention, the accuracy of color matching between a plurality of cameras is improved.

Further features of the present invention will become apparent from the following description of exemplary embodiments with reference to the attached drawings.

The present invention will now be described in detail below with reference to the accompanying drawings showing embodiments thereof. A description will now be given of a first embodiment of the present invention.

1 FIG. 2 FIG. 1 FIG. 1 1 is a diagram showing a configuration of an image pickup systemaccording to the first embodiment, andis a block diagram showing a hardware configuration of the image pickup systemin.

1 FIG. 1 100 10 200 20 300 400 500 600 700 Referring to, the image pickup systemis comprised of an image pickup apparatus Aon which a lens Ais mounted, an image pickup apparatus Bon which a lens Bis mounted, a control apparatus, a display apparatus, a display wall, an image controller, and an image output apparatus.

2 FIG. 100 100 300 100 300 103 100 100 200 100 300 100 10 100 100 As shown in, the image pickup apparatus Aoutputs a picked-up image IMG_A picked up by the image pickup apparatus Ato the control apparatus. The image pickup apparatus Areceives parameters PARAM_A from the control apparatusand sets the parameters PARAM_A in an image processing unitin the image pickup apparatus Aso as to perform color matching between the image pickup apparatus Aand the image pickup apparatus B. Color/brightness correction parameters (color/brightness correction values) are parameters belonging to the parameters PARAM_A as will be described in detail later. The image pickup apparatus Aalso outputs image pickup apparatus information INFO_A to the control apparatus. The image pickup apparatus information INFO_A includes the model number and serial number of the image pickup apparatus A, the model number and serial number of the lens Amounted on the image pickup apparatus A, and information relating on the image pickup apparatus Aincluding illumination light information and image pickup conditions.

2 FIG. 10 100 1 As shown in, the lens Ais mounted on the image pickup apparatus Avia a mount A.

1 10 100 1 100 10 100 10 1 100 10 2 FIG. The mount Ais a mechanism for removably mounting the lens Aon the image pickup apparatus A. The mount Ais equipped with electric contacts for supplying power from the image pickup apparatus Ato the lens Aand enabling communication between the image pickup apparatus Aand the lens A. The mount Aincludes a part included in the image pickup apparatus Aand a part included in the lens A, and they are illustrated collectively infor the sake of convenience.

10 100 1 10 101 100 102 103 100 10 2 FIG. The lens Ais a lens mounted on the image pickup apparatus Avia the mount A. As shown in, light entering through the lens Ais received by a sensorin the image pickup apparatus A, subjected to A/D conversion by an A/D converter, and subjected to image processing by the image processing unit, to generate the picked-up image IMG_A. The image pickup apparatus Acontrols zooming, focusing, etc. of the lens A.

2 FIG. 200 200 300 200 300 200 203 200 200 100 200 300 200 20 200 200 As shown in, the image pickup apparatus Boutputs a picked-up image IMG_B picked up by the image pickup apparatus Bto the control apparatus. The image pickup apparatus Bis controlled based on parameters PARAM_B received from the control apparatus. The image pickup apparatus Bsets the parameters PARAM_B in an image processing unitin the image pickup apparatus Bso as to perform color matching between the image pickup apparatus Band the image pickup apparatus A. Color/brightness correction parameters are parameters belonging to the parameters PARAM_B as will be described in detail later. The image pickup apparatus Balso outputs image pickup apparatus information INFO_B to the control apparatus. The image pickup apparatus information INFO_B includes the model number and serial number of the image pickup apparatus B, the model number and serial number of the lens Bmounted on the image pickup apparatus B, and information relating on the image pickup apparatus Bincluding illumination light information and image pickup conditions.

2 FIG. 20 200 2 As shown in, the lens Bis mounted on the image pickup apparatus Bvia a mount B.

2 20 200 2 200 20 200 20 2 200 20 2 FIG. The mount Bis a mechanism for removably mounting the lens Bon the image pickup apparatus B. The mount Bis equipped with electric contacts for supplying power from the image pickup apparatus Bto the lens Band enabling communication between the image pickup apparatus Band the lens B. The mount Bincludes a part included in the image pickup apparatus Band a part included in the lens B, and they are illustrated collectively infor the sake of convenience.

20 200 2 20 201 200 202 203 200 20 2 FIG. The lens Bis a lens mounted on the image pickup apparatus Bvia the mount B. As shown in, light entering through the lens Bis received by a sensorin the image pickup apparatus B, subjected to A/D conversion by an A/D converter, and subjected to image processing by the image processing unit, to generate the picked-up image IMG_B. The image pickup apparatus Bcontrols zooming, focusing, etc. of the lens B.

1 FIG. 2 FIG. 300 400 300 100 200 300 300 100 200 100 200 300 500 600 50 50 100 200 500 As shown in, the control apparatus(image correction apparatus) outputs an image IMG_C to the display apparatus. In the control apparatus, an image including the picked-up image IMG_A and the picked-up image IMG_B which are obtained after color matching, received from the image pickup apparatus Aand the image pickup apparatus Brespectively, a menu for control by the user of the control apparatus, or the like is generated as the image IMG_C. As shown in, the control apparatusoutputs the parameters PARAM_A and the parameters PARAM_B, which include the color/brightness correction parameters, to the image pickup apparatus Aand the image pickup apparatus Brespectively, based on the image pickup apparatus information INFO_A and the image pickup apparatus information INFO_B respectively received from the image pickup apparatus Aand the image pickup apparatus B. The control apparatusalso outputs a command COMMAND_C, which is for outputting a background image for a live-action image or a CG image for use in shooting as an image IMG_E to a display wall(display unit), to the image controller. Note that the command COMMAND_C may include an instruction for outputting a pattern display image(or, pattern) (display image), which is for use in color matching between the image pickup apparatus Aand the image pickup apparatus B, to the display wallon the image IMG_E in a superimposed manner.

400 300 1 100 200 400 The display apparatusdisplays the image IMG_C received from the control apparatus. After the color matching is performed, a user who uses the image pickup systemcan check the picked-up image IMG_A and the picked-up image IMG_B respectively shot by the image pickup apparatus Aand the image pickup apparatus B, on the display apparatus.

500 600 500 500 500 500 100 200 500 500 500 500 100 200 1 FIG. The display walldisplays the image IMG_E received from the image controller. The display wallis comprised of a plurality of units, and the resolution of each unit is, for example, 384×216. In the description of the present embodiment, it is assumed that the display wallis curved as shown in; however, the present invention is not limited to this. The user is able to artificially produce an image as if the subject were shot at a location desired by the user by shooting the display walland a subject in front of the display wallusing the image pickup apparatus Aand the image pickup apparatus B. Note that in the present embodiment, the plurality of units constituting the display wallis a plurality of LED display units, and the display wallis constructed by the plurality of units being arranged lengthwise and breadthwise in a plane direction; however, the present invention is not limited to this. For example, the display wallmay be comprised of organic EL or liquid crystal panels or a rear projector, instead of LEDs. Each unit of the display wallhas a correction processing unit (not shown) with a color matrix or 3D-LUT (Lookup Table), and the color and the brightness are adjusted by a luminance colorimeter, or the image pickup apparatus Aand the image pickup apparatus Bso as to make the color and brightness match between the units.

2 FIG. 600 500 304 300 600 700 600 700 500 300 600 50 500 600 700 500 600 500 500 500 500 600 500 As shown in, the image controllerreceives the command COMMAND_C, which specifies an image to be displayed on the display wall, from a communication unitof the control apparatus. In accordance with the command COMMAND_C, the image controllergenerates a command COMMAND_D for sending an image IMG_D and sends the command COMMAND_D to the image output apparatus. The image controlleroutputs, as the image IMG_E, the image IMG_D received from the image output apparatusto the display wall. Further, in accordance with the command COMMAND_C received from the control apparatus, the image controllercan also display in a superimposed manner, a pattern display imageon the image IMG_E (display image) to be output to the display wall. Note that the image controllerdoes not send the image IMG_D, which is received from the image output apparatus, as it is as the image IMG_E to be output to the display wall. Specifically, the image controllerdivides the image IMG_D to generate split images at the resolution of the units constituting the display wall, assigns the split images to the units, and outputs the split images as the image IMG_E to the respective units. The aspect ratio of the image IMG_D is the same as that of the resolution of the entire display wallin some cases, and is different from that of the resolution of the entire display wallin other cases. In the case where the aspect ratio of the image IMG_D is different from that of the resolution of the entire display wall, image controllerenlarges or reduces the image IMG_D to generate an image with the same aspect ratio as the resolution of the entire display wall, and then divides the processed image (the image IMG_D enlarged/reduced) to generate the image IMG_E.

700 600 700 600 700 The image output apparatusstores various contents. Upon receiving an instruction to display a predetermined content from the image controlleras the command COMMAND_D, the image output apparatusoutputs the content as the image IMG_D to the image controller. The image output apparatusis, for example, a video server or a mass storage.

1 100 200 100 200 Although in the description of the present embodiment, the image pickup systemthat performs color matching between the two mage pickup apparatuses, i.e., the image pickup apparatus Aand the image pickup apparatus Bis taken as an example, the number of image pickup apparatuses to be subjected to color matching has only to be plural and may be three or more. Moreover, although in the present embodiment, it is assumed that the image pickup apparatus Aand the image pickup apparatus Bare image pickup apparatuses with interchangeable lenses, they may be image pickup apparatuses with built-in lenses.

2 FIG. 100 200 300 20 10 200 100 20 200 Referring to, a more detailed description will now be given of operations of the blocks in the image pickup apparatus A, the image pickup apparatus B, and the control apparatus. Note that the lens Bhas the same configuration as that of the lens A, and the image pickup apparatus Bhas the same configuration as that of the image pickup apparatus A; therefore, detailed description of the lens Band the image pickup apparatus Bis omitted.

10 11 12 13 The lens Ahas a taking lens, a lens control unit, and a memory.

11 11 12 11 11 The taking lenshas a drive unit (not shown), which is comprised of an actuator or a motor, and the drive unit drives the taking lensunder the control by the lens control unit. The actuator or motor of the taking lensmoves or operates a focus lens, a zoom lens, a diaphragm, an anti-vibration lens, etc. (none of which is shown) which the taking lenshas.

12 11 108 1 The lens control unithas a microcomputer (not shown), and controls the taking lensin accordance with a control signal received from an image pickup apparatus control unitthrough the mount A.

13 12 The memoryis used as a storage device for various types of data used by the lens control unit.

100 101 102 103 104 105 106 107 108 109 110 111 100 The image pickup apparatus Ahas the sensor(image pickup means), the A/D converter, an image processing unit, a display unit, a recording medium, a memory, an operating unit, an image pickup apparatus control unit, a communication unit, and an image output unit. They are connected to a busin the image pickup apparatus A.

101 101 11 10 The sensoris a photoelectric conversion device that has a plurality of pixels. The sensorphotoelectrically converts a subject image formed by the taking lensin the lens Aby each pixel to generate and output an analog electric signal corresponding to the subject image.

102 101 The A/D converterconverts the analog electric signal output from the sensorinto a digital signal.

103 102 103 11 The image processing unitperforms various types of image processing on the digital signal output from the A/D converterto generate the picked-up image IMG_A. The image processing unitcarries out what is called a development process such as an optical correction process for performing image processing to remove the influence of the optical characteristics of the taking lenson an image, pixel interpolation, brightness signal processing, and color signal processing.

103 104 105 303 300 110 103 100 200 110 300 300 400 The picked-up image IMG_A generated by the image processing unitis displayed on the display unit, recorded on the recording mediumsuch as a memory card, and output to an image output unitof the control apparatusvia the image output unit. The image processing unit(color/brightness correction means) also applies the color/brightness correction parameters for between the image pickup apparatus Aand the image pickup apparatus Bto the picked-up image IMG_A. The image output unit(first communication means) retransmits the picked-up image IMG_A, to which the color/brightness correction parameters have been applied, to the control apparatus. In response to the retransmission of the picked-up image IMG_A, the control apparatussends an image including the picked-up image IMG_A, to which the color/brightness correction parameters have been applied, as the image IMG_C to the display apparatus.

3 FIG. 3 FIG. 3 FIG. 3 FIG. 103 A description will now be given of how to apply the color/brightness correction parameters to the picked-up image IMG_A and the picked-up image IMG_B. The color/brightness correction performed using a 3D-LUT as an example will be described.shows an example of the 3D-LUT to explain the outline of the 3D-LUT. It is assumed here that an image that has not been subjected to color/brightness conversion is an 8-bit RGB image. The 3D-LUT is a table in which RGB output values are associated with specific RGB input values on a one-on-one basis. In, Rin, Gin, and Bin respectively represent gradation values of R, G, and B, before color/brightness conversion, and Rout, Gout, and Bout respectively represent gradation values of R, G, and B, after color/brightness conversion. In the 3D-LUT in, each of R, G, and B has 256 gradations (0-255), and the input gradation varies one step at a time, and therefore, a table comprised of R256×G256×B256=about 16.77 million combinations. For example, in a case where the RGB value of a pixel in a pre-color/brightness correction image is (Rin, Gin, Bin)=(0, 255, 255), (Rout, Gout, Bout)=(13, 245, 252) is obtained according to the 3D-LUT in. By applying the 3D-LUT to the RGB value of each of the pixels constituting a pre-color/brightness correction image using the above-described method, the image processing unitobtains an image whose color/brightness has been converted.

3 FIG. Although in the 3D-LUT in, each of R, G, and B has 256 gradations, and the input gradation varies one step at a time, a 3D-LUT in which the input gradation varies, for example, 16 steps at a time may be used so as to reduce the LUT size. Namely, a 3D-LUT comprised of about R16×G16×B16=4096 combinations may be used. In this case, by using triangular pyramid interpolation or the like, which is an existing technique, the input gradations other than the combinations of RGB values defined by the 3D-LUT may be converted. Moreover, although in the present embodiment, the 3D-LUT is taken as an example of the color/brightness correction method, the present invention is not limited to this, and a 1D-LUT or a color matrix, which is an existing technique may be used as the color/brightness correction method, and some methods may be used in combination.

2 FIG. 106 103 108 106 103 104 100 Referring again to, the memoryis used as a processing buffer for the image processing unitand a storage device for programs to be executed by the image pickup apparatus control unit. The memoryis also used as a storage device for optical correction data in which optical correction values to be used by the image processing unitare defined, and a storage device for GUI data such as a menu screen to be displayed on the display unitby the image pickup apparatus A.

107 100 100 The operating unitis an input device group for the user to input instructions to the image pickup apparatus A, which includes a power switch for turning on/off the power to the image pickup apparatus A, an image pickup switch for starting to record an image, a selecting/setting switch for setting various menus, and so forth.

108 106 100 103 10 The image pickup apparatus control unit, which has a microcomputer, executes programs stored in the memoryto control the overall operation of the image pickup apparatus Aincluding, for example, control of the image processing unitand control of communication with the lens A.

109 304 300 109 300 300 The communication unithas a function of communicating with other devices such as a LAN and a USB and communicates with the communication unitof the control apparatus. The communication unit(first communication means) sends the image pickup apparatus information INFO_A to the control apparatusand receives the parameters PARAM_A from the control apparatus.

110 303 300 The image output unithas a function of sending the picked-up image IMG_A to other devices such as a LAN and an HDMI (registered trademark) and outputs the picked-up image IMG_A to the image input unitof the control apparatus.

111 102 103 104 105 106 107 108 109 110 The busconnects to the A/D converter, the image processing unit, the display unit, the recording medium, the memory, the operating unit, the image pickup apparatus control unit, the communication unit, and the image output unitand exchanges signals between the processing blocks.

300 301 302 303 304 305 306 307 308 The control apparatushas a control unit, a color matching processing unit, the image input unit, the communication unit, a memory, an operating unit, and an image output unit, which are connected to a bus.

301 305 300 302 100 200 The control unit, which has a microcomputer, executes programs stored in the memoryto control the overall operation of the control apparatus (image correction apparatus)including, for example, control of the color matching processing unitand control of communication with each of the image pickup apparatus Aand the image pickup apparatus B.

302 3021 3022 3023 3024 The color matching processing unithas a pattern detection unit, a viewing angle correction value calculation unit, a viewing angle characteristic correction unit, and a color correction value calculation unit.

3021 50 500 100 200 3021 50 50 50 50 500 50 500 The pattern detection unit(pattern detection means) detects the pattern display imagedisplayed on the display wallfrom the picked-up image IMG_A and the picked-up image IMG_B obtained as a result of image pickup by the image pickup apparatus Aand the image pickup apparatus B. Specifically, the pattern detection unitis capable of detecting the pattern display imageusing a known technique of, for example, detecting the edge or apex of the pattern display image, or detecting a marker superimposed on the pattern display image, from the picked-up image IMG_A and the picked-up image IMG_B. In the present embodiment, a color chart image (second pattern image) including a plurality of patch images, which will be described later, for use in generating a color/brightness correction parameter LUT, which will be described later, is displayed as the pattern display imageon the display wall. A white patch image (first pattern image), which is a uniform image of a specific color and used in correcting viewing angle characteristics as will be described later, is displayed in the same area displaying the pattern display imageon the display wall. Note that the white patch image and the color chart image according to the present embodiment can be modified within the scope of the present invention irrespective of their pattern shapes and colors as long as they are patterns which can generate the color/brightness correction parameter LUT and perform correction of the viewing angle characteristics.

3021 3022 3021 100 200 3021 When a detected pattern area has a trapezoid shape, the pattern detection unittransforms the detected pattern area into a rectangular shape through, for example, projective transformation. As will be described later, considering that viewing angle corrections value using the viewing angle correction value calculation unitis calculated, the pattern detection unitperforms a transformation such that pattern areas obtained from the image pickup apparatus Aand the image pickup apparatus B, respectively, have the same resolution and the same aspect ratio. When the resolution of detected pattern areas is high, the pattern detection unitmay carry out a process in which it further reduces the detected pattern areas according to a calculation processing load while keeping the same resolution and aspect ratio of the detected pattern areas.

100 200 3022 3021 100 200 500 100 200 500 500 500 500 4 FIG. 4 FIG. 7 FIG.B To correct the viewing angle characteristics of the image pickup apparatus Aand the image pickup apparatus B, the viewing angle correction value calculation unit(viewing angle correction value calculation means) calculates such a correction value (viewing angle correction value) that each pixel of the white patch image detected by the pattern detection unithas a predetermined brightness. The viewing angle characteristic is characteristic resulting from the position (viewing angle) of the image pickup apparatus A(the image pickup apparatus B) with respect to the display screen of the display wall. More specifically, in the present embodiment, the viewing angle represent displacement from the direction of the normal from the image pickup apparatus A(the image pickup apparatus B) to the display screen of the display wall(hereafter referred to merely as “the direction of the normal to the display wall”). In the present embodiment, the viewing angle characteristic is characteristic of brightness and color when the white patch image on the display screen of the display wallis viewed at this viewing angle, and the brightness characteristic is presented as shown in. In the graph of, the horizontal axis represents the viewing angle in the horizontal direction, and the vertical axis represents the brightness. In the present embodiment, for the viewing angles in the horizontal direction represented by the horizontal axis, the right-hand side and the left-hand side with respect to the direction of the normal to the display wallare respectively designated as “+” and “−” ().

4 FIG. 4 FIG. 500 100 500 500 200 500 In general, the viewing angle characteristics include the characteristic that the brightness and color become lower/darker as the viewing angle increases. Although in, only the brightness characteristic of the viewing angle characteristics is illustrated, the same holds for the color characteristic. A description will now be given of a method for correcting the viewing angle characteristics using only the brightness characteristic for the sake of simplification, and the same holds for the method using the color characteristic. Moreover, the viewing angle characteristics are present in the vertical direction as well, and method for the vertical direction is c the same as the method for the horizontal direction, and hence description thereof is omitted. As shown in, as viewed from the direction of the normal to the display wall(the viewing angle is 0°), a white patch image A detected from the picked-up image IMG_A obtained by the image pickup apparatus Aplaced at a location in the direction of the normal to the display wallhas the characteristic that the brightness is even because the display brightness is 100%. On the other hand, as viewed from a direction deviating from the direction of the normal to the display wall(the viewing angle is a predetermined angle (≠0°)), the brightness of a white patch image B detected from the picked-up image IMG_B obtained by the image pickup apparatus Bplaced at a location deviating from the direction of the normal to the display wallis uneven because the display brightness is lower than 100%.

3022 100 200 3022 To correct the uneven brightness, the viewing angle correction value calculation unitcalculates a viewing angle correction value such that the brightness is even across the white patch image B. Namely, one of the image pickup apparatus Aand the image pickup apparatus Bwhich has a smaller viewing angle is regarded as a reference image pickup apparatus, and in order to eliminate the viewing angle characteristic of the other image pickup apparatus (non-reference image pickup apparatus) that the brightness is uneven, the viewing angle correction value is calculated such that the brightness distribution characteristic of the white patch image B matches that of the white patch image A. In a case where a pixel value at certain coordinates in the white patch image B is (R, G, B)=(128, 128, 128) and a pixel value at the same coordinates in the white patch image A is (R, G, B)=(255, 255, 255), the viewing angle correction value is calculated such that the pixel value at the certain coordinates in the white patch image B is doubled for all of R, G, and B. The viewing angle correction value calculation unitcalculates the viewing angle correction values for all the pixels in the detected white patch images.

3023 3021 3022 The viewing angle characteristic correction unit(viewing angle characteristic correction means□subjects pattern areas detected by the pattern detection unitin a color matching process, which will be described later, to correction using the viewing angle correction values calculated by the viewing angle correction value calculation unit.

3024 100 200 3021 The color correction value calculation unit(color correction value calculation means/color/brightness correction value calculation means) causes each of the image pickup apparatus Aand the image pickup apparatus Bto shoot a pattern image for calculating color correction values and carries out the color matching process using predetermined pixel values in areas detected by the pattern detection unit. Specifically, in the present embodiment, the color/brightness correction parameter LUT is generated in the color matching process.

5 FIG. 5 FIG. 501 512 100 200 3024 3024 100 200 100 200 3024 200 In general, the color/brightness correction parameter LUT is generated using a color chart image shown in. Patch imagestoof different colors are arranged in the color chart image in. Based on the color chart detected from the picked-up image IMG_A and the picked-up image IMG_B obtained by the image pickup apparatus Aand the image pickup apparatus B, the color correction value calculation unitgenerates the color/brightness correction parameter LUT for color matching between the picked-up image IMG_A and the picked-up image IMG_B. Here, the color correction value calculation unitsets one of the image pickup apparatus Aand the image pickup apparatus Bwhich has a smaller viewing angle as a reference image pickup apparatus, and sets the color/brightness correction parameter LUT for a non-reference image pickup apparatus. In the present embodiment, the reference image pickup apparatus is the image pickup apparatus A, and the non-reference image pickup apparatus is the image pickup apparatus B, and therefore, the color correction value calculation unitsets the color/brightness correction parameter LUT for the image pickup apparatus B.

501 512 100 501 512 200 501 512 100 200 Specifically, first, the patch imagestoincluded in the color chart image detected from the picked-up image IMG_A obtained by the image pickup apparatus A(the reference image pickup apparatus) are obtained as reference patch images. Next, the patch imagestoincluded in the color chart image detected from the picked-up image IMG_B obtained by the image pickup apparatus B(the non-reference image pickup apparatus) are obtained. Then, the patch imagestoobtained from the picked-up image IMG_B are subjected to correction using the viewing angle correction values described above. After that, a color correction parameter LUT (color correction values) is generated such that the colors of the corrected path images are close to the colors of the respective reference patch images. Further, the color/brightness correction parameter LUT for color matching between the image pickup apparatus Aand the image pickup apparatus Bis generated using the color correction parameter LUT and the viewing angle correction values.

100 200 1 1 1 Although in the present embodiment, the two image pickup apparatuses, i.e., the image pickup apparatus Aand the image pickup apparatus Bare used in the image pickup system, the number of image pickup apparatuses used in the image pickup systemmay be three or more. For example, in a case where the number of image pickup apparatuses used in the image pickup systemis N, the number of non-reference image pickup apparatuses is (N−1), and hence (N−1) color/brightness correction parameter LUTs are generated.

100 200 305 Note that the form of the color chart and the number of patches are not limited to the examples described above. For example, the number of patches may be increased, by which more precise color information on the image pickup apparatus Aand the image pickup apparatus Bis obtained. However, when the number of patches is increased, the load on the correction parameter LUT generating process will increase accordingly, and hence there is a trade-off. Color/brightness correction parameters LUT that can be generated by the method described above may be generated whenever image pickup conditions are changed. Here, examples of the cases where image pickup conditions are changed include a case where a lens mounted on a non-reference image pickup apparatus has been replaced, a case where the color temperature of a lighting environment has changed, a case where non-reference image pickup apparatuses have been changed, and a case where parameters such as the iris of a reference image pickup apparatus has been changed. The color/brightness correction parameters LUTs generated under the respective image pickup conditions are stored in the memory.

303 110 100 303 210 200 The image input unit(second communication means) has an image input interface such as a LAN or an HDMI (registered trademark) and obtains the picked-up image IMG_A sent or resent from the image output unitof the image pickup apparatus A. Likewise, the image input unitobtains the picked-up image IMG_B sent or resent from the image output unitof the image pickup apparatus B.

304 109 100 209 200 600 100 304 3024 200 304 100 200 109 100 304 500 50 600 300 100 200 600 The communication unithas a function of communicating with other devices such as a LAN and a USB and communicates with the communication unitof the image pickup apparatus A, the communication unitof the image pickup apparatus B, and a communication unit (not shown) of the image controller. For example, when a reference image pickup apparatus for color/brightness correction is the image pickup apparatus A, the communication unitsends a color/brightness correction parameter LUT determined by the color correction value calculation unitas PARAM_B to the image pickup apparatus B(non-reference image pickup apparatus). Descriptions of other signals which the communication unitsends and receives to and from the image pickup apparatus Aand the image pickup apparatus B, which are described above in the description of the communication unitof the image pickup apparatus A, is omitted. The communication unitsends the command COMMAND_C, which specifies a background image to be displayed on the display wallor issues an instruction to display the pattern display imagefor use in a viewing angle correction value calculation process and the color matching process, which will be described later, in a superimposed manner on the display image, to the image controller. In the present embodiment, the control apparatus, the image pickup apparatus A, the image pickup apparatus B, and the image controllerare connected to one another so as to communicate with one another. The connection here may be either wired connection using USB, LAN, or the like or wireless connection using Wi-Fi (registered trademark) as long as information can be exchanged through communication among the devices.

305 301 305 400 300 307 100 200 305 4 FIG. The memory(storage means) is used as various processing buffers, a storage device for programs to be executed by the control unit, and a storage device for color/brightness correction parameters as shown in. The memoryis also used as a storage device for GUI data such as a menu screen to be displayed on the display apparatusby the control apparatusvia the image output unit. Information received from the image pickup apparatus Aand the image pickup apparatus Bis stored in the memory.

306 300 300 The operating unitis a group of input devices for the user to input instructions to the control apparatus, such as a power switch for turning on/off the power to the control apparatus, and a switch for starting the application of color/brightness correction parameters to a non-reference image pickup apparatus and the update of the color/brightness correction parameters.

307 400 The image output unithas a function of sending images to other devices via an SDI, an HDMI (registered trademark), or the like and outputs the image IMG_C to the display apparatus.

6 10 FIGS.to 500 Referring to, a description will now be given of how to improve the accuracy of color matching between a reference image pickup apparatus and a non-reference image pickup apparatus by removing the viewing angle characteristics of the display wallfrom color chart images detected from the picked-up images obtained by the reference image pickup apparatus and the non-reference image pickup apparatus.

6 FIG. 400 306 300 301 300 305 is a flowchart of the viewing angle correction value calculation process. The present process is started when the user selects execution of color matching on a GUI (not shown) displayed on the display apparatusby operating the operating unitof the control apparatus. The present process is implemented by the control unitof the control apparatusreading out a program stored in the memoryand deploying it into a RAM (not shown).

601 300 500 600 304 600 50 700 600 500 50 500 500 50 50 100 200 In step S, the control apparatusoutputs an instruction to display the image IMG_E, on which a white patch image is superimposed, on the display wall, as the command COMMAND-C to the image controllervia the communication unit. The image controllerthat has received the command COMMAND-C generates the image IMG_E, in which the white patch image is superimposed on the area of the pattern display imagein the image IMG_D received from the image output apparatus. The image controllerthen sends the image IMG_E and an instruction to display the image IMG_E to the display wall. Note that the resolution of the pattern display imagesuperimposed on the image IMG_D is preferably equal to the resolution of one of the multiple LED display units constituting the display wall. The reason is that the LED display units constituting the display wallhave different characteristics, and hence if the pattern display imagehas a higher resolution than the resolution of one of the LED display units, there may be brightness and color differences at the boundaries of the LED display units, causing degradation in the accuracy of the color matching process, which will be described later. However, this does not apply to a case where, if the resolution of the pattern display imagesuperimposed on the image IMG_D is set as described above, the color matching process will become impossible due to the installation locations of the image pickup apparatus Aand the image pickup apparatus B.

602 300 50 500 100 200 304 In step S, the control apparatussends a shooting instruction command, which is an instruction to shoot the pattern display image(white patch image) displayed on the display wall, to each of the image pickup apparatus Aand the image pickup apparatus Bvia the communication unit.

603 300 100 200 300 604 300 300 400 307 In step S, the control apparatusstands by until it receives the picked-up image IMG_A and the picked-up image IMG_B from the image pickup apparatus Aand the image pickup apparatus B, respectively, and when the control apparatushas received the picked-up image IMG_A and the picked-up image IMG_B, the process proceeds to step S. In a case where the control apparatushas not received the picked-up image IMG_A and the picked-up image IMG_B even though a predetermined period of time has elapsed, the control apparatustimes out and may output an error message superimposed on the image IMG_C to the display apparatusvia the image output unit.

604 300 3021 100 200 603 604 605 604 602 300 100 200 300 400 In the step S, the control apparatuscauses the pattern detection unit(first image detection means) to detect an area where the white patch image is displayed from each of the picked-up image IMG_A and the picked-up image IMG_B respectively received from the image pickup apparatus Aand the image pickup apparatus Bin the step S. In a case where the white patch image can be detected from both the picked-up image IMG_A and the picked-up image IMG_B (YES in the step S), the process proceeds to step S. On the other hand, in a case where the white patch image cannot be detected from at least one of the picked-up image IMG_A and the picked-up image IMG_B (NO in the step S), the process returns to the step S, in which the control apparatusresends the shooting instruction command to the image pickup apparatus Aor the image pickup apparatus B, or both of them, which sent the image from which the white patch image cannot be detected. At this time, the control apparatusmay output an error message, which is to be displayed on the image IMG_C in a superimposed manner, to indicate that the white patch image cannot be detected, to the display apparatus.

605 300 3022 50 604 50 50 200 701 50 100 702 200 50 500 200 703 50 706 50 704 50 707 50 705 50 708 50 7 FIG.A 7 FIG.A 7 FIG.B 4 FIG. In step S, the control apparatuscauses the viewing angle correction value calculation unitto obtain brightness distribution information on the area of the white patch image in the pattern display imagedetected in the step Sand detect uneven brightness in the white patch image in the pattern display image. For example, the area of the white patch image in the pattern display imagedetected from the picked-up image IMG_B received from the image pickup apparatus Bhas a brightness distribution indicated in a patch detected imagein. On the other hand, the area of the white patch image in the pattern display imagedetected from the picked-up image IMG_A received from the image pickup apparatus Ahas a brightness distribution indicated in a patch detected imagein. The viewing angle characteristics in a case where the image pickup apparatus Bis shooting the pattern display imagefrom the right side of the direction of the normal to the display surface of the display wallas shown incorrespond to the viewing angle characteristics on the (+) side in. As for the image pickup apparatus B, a viewing anglewith respect to the left end of the pattern display imageis an angle with respect to a normalto the display position of the left end of the pattern display image, a viewing anglewith respect to the center of the pattern display imageis an angle with respect to a normalto the display position of the center of the pattern display image, and a viewing anglewith respect to the right end of the pattern display imageis an angle with respect to a normalto the display position of the right end of the pattern display image.

7 FIG.B 7 FIG.B 4 FIG. 703 704 705 701 200 702 100 100 500 100 702 100 In, the sizes of the viewing angles are as follows: the viewing angle<the viewing angle<the viewing angle. Thus, in the image (patch detected image) in the area of the white patch image detected from the picked-up image IMG_B received from the image pickup apparatus B, it becomes darker as it goes toward the right in the figure. The brightness distribution is obtained in, for example, the form of pixel values (R, G, B). The image (patch detected image) in the area of the white patch image detected from the picked-up image IMG_A received from the image pickup apparatus Ahas a uniform brightness distribution. In the case where the image pickup apparatus Ais shooting from the direction of the normal to the display wallas shown in, the viewing angle characteristics of the image pickup apparatus Acorrespond to the viewing angle characteristics (brightness characteristics) at the viewing angle of 0° in, and hence the patch detected imagedetected from the picked-up image IMG_A received from the image pickup apparatus Ahas even brightness.

500 200 500 500 703 705 701 7 FIG.A When an image pickup apparatus is shooting the display wallat an angle like the image pickup apparatus B, the brightness distribution will be different depending on the shape of the display wall. For example, in a case where the display wallis flat, the relationship among the viewing anglestochanges, and accordingly, the brightness distribution of the patch detected imagewill be different from the brightness distribution shown in.

606 300 3022 701 701 702 605 3022 701 702 305 In step S, the control apparatuscauses the viewing angle correction value calculation unitto calculate viewing angle correction values for the patch detected imagebased on brightness distribution information on the patch detected imageand brightness distribution information on the patch detected imageobtained in the step S. The viewing angle correction values are calculated for all the pixels of a patch detected image. Namely, the viewing angle correction value calculation unitcalculates the viewing angle correction values such that the brightness distribution in all the pixels of the patch detected imagecorresponds to the brightness distribution in the patch detected image, which is the uniform brightness distribution. The calculated viewing angle correction values are stored in the memory(storage means).

6 FIG. This completes the process in.

701 702 Although in the present embodiment, the viewing angle correction values are calculated such that the brightness distribution in the patch detected imageequivalent to the brightness distribution in the patch detected image, other calculation methods may be used as long as they fall within the scope of the present invention.

100 500 702 100 701 1 701 702 100 200 500 7 FIG.B For example, the image pickup apparatus Amay not be placed in the direction of the normal () to the display wall. In this case, the patch detected imageobtained from the image pickup apparatus Adoes not have a uniform brightness distribution but has uneven brightness like the patch detected image. Namely, it is only necessary to correct uneven brightness resulting from the viewing angle characteristics, and hence the locations of respective image pickup apparatuses in the image pickup systemare not particularly limited. For example, the respective viewing angle correction values for the patch detected imageand the patch detected imagerespectively detected from the picked-up image IMG_A and the picked-up image IMG_B respectively received from the image pickup apparatus Aand the image pickup apparatus Bmay be calculated using a reference image that is “even in a specific color”. To be “even in a specific color” means here that graduation values in the reference image are uniform. Namely, the reference image can be considered to be equivalent to a patch detected image shot from the direction of the normal to the display wall(the position where the viewing angle is 0°).

8 FIG. 6 FIG. 301 300 305 is a flowchart of the color matching process. The present process is carried out subsequently after the viewing angle correction value calculation process () is ended. The present process is implemented by the control unitof the control apparatusreading out a program stored in the memoryand deploying it into the RAM (not shown).

400 6 FIG. 8 FIG. Note that a viewing angle correction value calculation start button and a color matching start button may be separately provided on a GUI (not shown) displayed on the display apparatus. In this case, when the viewing angle correction value calculation start button is selected by user operation, the viewing angle correction value calculation process () is started, and when the color matching start button is selected by user operation, the color matching process () is started.

801 300 500 600 304 600 50 700 600 500 801 601 In step S, the control apparatusoutputs an instruction to display the image IMG_E, on which a color chart image is superimposed, on the display wall, as the command COMMAND-C to the image controllervia the communication unit. The image controllerthat has received the command COMMAND-C generates the image IMG_E, in which the pattern display image(color chart image) is superimposing on the image IMG_D received from the image output apparatus. The image controllerthen sends the image IMG_E and an instruction to display the image IMG_E to the display wall. Note that the resolution of the color chart image superimposed on the image IMG_D in the step Sis equal to the resolution of the white patch image superimposed on the image IMG_D in the step S.

802 300 50 500 100 200 304 In step S, the control apparatussends a shooting instruction command, which is an instruction to shoot the pattern display image(color chart image) displayed on the display wall, to each of the image pickup apparatus Aand the image pickup apparatus Bvia the communication unit(second communication means).

803 300 100 200 300 804 300 300 400 307 In step S, the control apparatusstands by until it receives the picked-up image IMG_A and the picked-up image IMG_B from the image pickup apparatus Aand the image pickup apparatus B, respectively, and when the control apparatushas received the picked-up image IMG_A and the picked-up image IMG_B, the process proceeds to step S. In a case where the control apparatushas not received the picked-up image IMG_A and the picked-up image IMG_B even though a predetermined period of time has elapsed, the control apparatustimes out and may output an error message superimposed on the image IMG_C to the display apparatusvia the image output unit.

804 300 3021 50 100 200 803 804 805 804 802 300 100 200 300 400 In the step S, the control apparatuscauses the pattern detection unit(second image detection means) to detect an area of the pattern display image(color chart image) from each of the picked-up image IMG_A and the picked-up image IMG_B respectively received from the image pickup apparatus Aand the image pickup apparatus B, which are obtained in the step S. In a case where the color chart image can be detected from both the picked-up image IMG_A and the picked-up image IMG_B (YES in the step S), the process proceeds to step S. On the other hand, in a case where the color chart image cannot be detected from at least one of the picked-up image IMG_A and the picked-up image IMG_B (NO in the step S), the process returns to the step S, in which the control apparatusresends the shooting instruction command to the image pickup apparatus Aor the image pickup apparatus B, or both of them, which sent the image from which the color chart image cannot be detected. At this time, the control apparatusmay output an error message, which is to be displayed on the image IMG_C in a superimposed manner, to indicate that the color chart image cannot be detected, to the display apparatus.

9 FIG. 804 is a view showing examples of the color chart image detected in the step S.

901 200 200 500 500 200 500 1 FIG. 4 FIG. A color chart imageis a color chart image detected from the picked-up image IMG_B received from the image pickup apparatus B, and has brightness distribution which becomes darker as it goes toward the right in the figure. The reason is that the image pickup apparatus Bis located on the right side in the figure of the direction of the normal to the display wall(the location where the viewing angle is greater than 0°) (), and hence the brightness of the display wallviewed from the image pickup apparatus Bdecreases due to the viewing angle characteristics () as it goes toward the right in the display wall. The brightness distribution is obtained in, for example, the form of pixel values (R, G, B).

902 100 100 500 500 100 1 FIG. 4 FIG. A color chart imageis a color chart image detected from the picked-up image IMG_A received from the image pickup apparatus A, and has a uniform brightness distribution. The reason is that as shown in, the image pickup apparatus Ais located in the direction of the normal to the display wall(the location where the viewing angle is 0°), and hence the display brightness is 100% due to the viewing angle characteristics (), and the brightness of the display wallviewed from the image pickup apparatus Alooks even.

805 3023 305 606 901 804 903 606 701 702 901 903 500 903 200 902 3023 902 902 606 901 7 FIG. 9 FIG. In step S, the viewing angle characteristic correction unitapplies the viewing angle correction values calculated and stored in the memoryin the step Sto the color chart imagedetected in the step Sto generate a color chart image(corrected image). In the present embodiment, the viewing angle correction values are calculated in the step Ssuch that the patch detected imageis corrected to be close to the patch detected image(). Thus, when the viewing angle correction value is applied to the color chart image, the color chart image() is generated. As a result of this correction, the viewing angle characteristics of the picked-up image resulting from the locations of image pickup apparatus with respect to the direction of the normal to the display wallare removed. However, in the color chart image, color shift of the color patches resulting from individual difference of a lens or the like in the image pickup apparatus Bstill remains. Note that since there are no viewing angle characteristics to be removed from the color chart image, the viewing angle characteristic correction unitgenerates the color chart imageitself as a corrected image. However, in a case where there are viewing angle characteristics to be removed from the color chart image, and the viewing angle correction values for them were calculated in the step S, a corrected image is generated in the same way as the color chart imageis generated.

806 3024 902 804 903 805 902 903 806 5 FIG. In step S, the color correction value calculation unitobtains gradation values in the respective color patches in each of the color chart imagedetected in the step Sand the color chart imagegenerated in the step S. Coordinate values of the respective color patches are determined in advance, and hence gradation values at specific coordinate values should be read. At this time, if only one pixel is taken out from each color patch, gradation values may vary due to image pickup noise, and hence a plurality of pixels are taken out from one color patch to obtain an average value and use it. In the present embodiment, there are 12 color patches as indicated in the color chart image in, gradation values in the 12 color patches are obtained from each of the color chart imagesandin the step S.

807 300 902 903 806 902 903 300 305 606 100 200 In the step S, the control apparatususes the gradation values in the respective color patches obtained from each of the color chart imagesandin the step Sto generate a color correction parameter LUT (color correction values) for color matching between the color chart imagesand. Then, the control apparatusgenerates, based on the color correction parameter LUT and the viewing angle correction values stored in the memoryin the step S, a color/brightness correction parameter LUT for color/brightness matching between the image pickup apparatus Aand the image pickup apparatus B(color matching). In this manner, the color/brightness correction parameter LUT is generated.

301 100 200 304 100 109 103 110 300 200 209 300 301 400 400 After that, the control unitgenerates the parameters PARAM_A and the parameters PARAM_B based on the generated color/brightness correction parameter LUT, the image pickup apparatus information INFO_A and the image pickup apparatus information INFO_B, and sends the generated parameters PARAM_A and parameters PARAM_B to the image pickup apparatus Aand the image pickup apparatus Bvia the communication unit. In the image pickup apparatus A, when the communication unitreceives the parameters PARAM_A, first, the image processing unit(color correction means) subjects the picked-up image IMG_A to correction using the parameters PARAM_A. Next, the image output unit(first communication means) resends the corrected picked-up image IMG_A to the control apparatus. In the image pickup apparatus B, when the communication unitreceives the parameters PARAM_B, the similar processing is performed, and the corrected picked-up image IMG_B is resent to the control apparatus. Upon receiving the corrected picked-up image IMG_A and the corrected picked-up image IMG_B, the control unit(display instruction means) generates an image including those images as the image IMG_C and sends it to the display apparatus, thus instructing the display apparatusto display the image IMG_C.

8 FIG. This completes the process in.

500 500 In the present embodiment, color matching between a plurality of image pickup apparatuses whose viewing angles with respect to the display wallare different is performed after color chart images shot by the plurality of image pickup apparatuses and displayed on the display wallare corrected using the viewing angle correction values and remove the viewing angle characteristics. This improves the accuracy of color matching between the plurality of image pickup apparatuses.

100 500 100 150 500 200 250 500 100 500 200 100 150 100 250 200 305 10 FIG. 10 FIG. 4 FIG. Note that although in the present embodiment, the image pickup apparatus Ais placed in the direction of the normal to the display wall, the present invention is not limited to this. For example, as shown in, the image pickup apparatus Amay be placed at a viewing anglewith respect to the direction of the normal to the display wall, and the image pickup apparatus Bmay be placed at a viewing anglewith respect to the direction of the normal to the display wall. In this case, the viewing angle correction values may be calculated, instead of using a reference image as described above, using the image pickup apparatus Awith the least deviation from (the least viewing angle with respect to) the direction of the normal to the display wallas a reference image pickup apparatus for color matching. In this case, the viewing angle correction values may be calculated such that the brightness distribution of the white patch image in the picked-up image IMG_B picked up by the image pickup apparatus Bis close to the brightness distribution of the white patch image in the picked-up image IMG_A picked up by the image pickup apparatus A. Alternatively, in the layout shown in, the viewing angleof the image pickup apparatus Aand the viewing angleof the image pickup apparatus Bmay be measured using a known measurement means (angle detection means), and the viewing angle correction values may be calculated based on the viewing angle characteristics () stored in the memoryin advance.

500 500 50 100 200 50 500 100 200 50 500 50 50 500 Since the display wallis comprised of the plurality of LED display units, the characteristics of the LEDs vary depending on their positions within the display wall. Thus, if the display position of the pattern display imageis changed when it is shot by the image pickup apparatus Aor the image pickup apparatus B, there may be differences in color/brightness between the cameras due to differences in viewing angle characteristics, differences in the characteristics of the cameras, and differences in the characteristics of the LEDs, which may result degradation in the accuracy of color matching. Therefore, it is preferred that the display position of the pattern display imageon the display wallis not changed each time it is shot by each of the image pickup apparatus Aor the image pickup apparatus B. Note that although in the present embodiment described above, the pattern display imageis displayed at the center of the display wall, the pattern display imagemay be displayed at another position as long as it is included in the angle of view of an image pickup apparatus. It is preferred that the pattern display imageis displayed on an LED display unit which is most frequently included in the angle of view when the user produces contents by shooting a subject in front of the display wall.

4 FIG. Moreover, although in the present embodiment described above, the viewing angle correction values are calculated using only brightness characteristics, they may be calculated using color characteristics as well as brightness characteristics. In this case, the viewing angle correction values can be calculated using a color distribution with respect to viewing angles as with the brightness distribution with respect to viewing angles in.

A description will now be given of a second embodiment of the present invention. The same components in the present embodiment as those in the first embodiment are designated by the same reference symbols, and descriptions that duplicates descriptions in the first embodiment are omitted.

11 FIG. 1 is a diagram showing a configuration of an image pickup system′ according to the second embodiment.

1 100 10 200 20 300 400 800 900 The image pickup system′ according to the second embodiment is comprised of the image pickup apparatus Aon which the lens Ais mounted, the image pickup apparatus Bon which the lens Bis mounted, the control apparatus, the display apparatus, a projection apparatus, and a screen(display unit).

1 500 800 900 In the first embodiment, the image pickup systemis constructed with the display wall, and in the present embodiment, an image pickup system is constructed with the projection apparatusand the screen.

800 900 307 900 800 900 800 The projection apparatusdisplays (projects), on the screen, an image which is transformed in advance by the image output unitsuch that the picked-up image does not extend off the screenwhen being displayed (projected). Alternatively, the projection apparatusdisplays (projects), on the screen, an image transformed by a transformation unit (not shown) in the projection apparatus.

50 900 50 500 The display position of the pattern display imageon the screenis the same as the display position of the pattern display imageon the display wallin the first embodiment, and hence description thereof is omitted.

900 500 500 800 900 900 900 900 900 900 800 900 4 FIG. The screenalso has the viewing angle characteristics like the display wall. The display brightness in the direction of the normal to the display wallis 100%. On the other hand, light from the projection apparatusis reflected on the screenand enters the user's eyes, and hence the display brightness in the direction of the normal to the display screen of the screen(hereafter referred to merely as “the direction of the normal to the screen”) is not 100%. Namely, the display brightness of the screenis 100% in an exit direction of light emitting at an exit angle of light from the screen, which is a position that is symmetric, about the direction of the normal to the screen, with an incident angle of light from the projection apparatus. Namely, the exit direction at the exit angle of the light from the screencan be replaced as the direction of 0° in the graph ofshowing the viewing angle characteristics.

The other configurations, functions, etc. are the same as those of the first embodiment, and hence description thereof is omitted.

Although the present invention has been described by way of the preferred embodiments, the present invention should not be limited to the above-described specific embodiments, but various modifications and alterations can be made without departing from the spirit and scope of the present invention. Some of the above-described embodiments may be combined where appropriate.

Moreover, the present invention encompasses a configuration in which programs of software that implements the functions of the above-described embodiments are supplied to a system or apparatus including a computer capable of executing the programs, from a recording medium directly or via wired/wireless communication, to execute the program.

Thus, program codes themselves which are supplied to and installed in a computer so that the computer can implement the functions and processes of the present invention also implement the present invention. Namely, the present invention includes computer programs themselves for implementing the functions and processes of the present invention.

In this case, the programs may be in any forms as long as they have functions of the programs such as an object code, a program to be executed by an interpreter, and script data to be supplied to an OS.

Examples of the recording medium for supplying the programs include a hard disk, a magnetic recording medium such as a magnetic tape, an optical/magneto-optical storage medium, and a nonvolatile semiconductor memory.

Examples of the method for supplying the programs include a method in which a computer program that forms the present invention is stored in a server on a computer network, and a client computer connects to the server, downloads the computer program, and executes the same.

Embodiment(s) of the present invention can also be realized by a computer of a system or apparatus that reads out and executes computer executable instructions (e.g., one or more programs) recorded on a storage medium (which may also be referred to more fully as a ‘non-transitory computer-readable storage medium’) to perform the functions of one or more of the above-described embodiment(s) and/or that includes one or more circuits (e.g., application specific integrated circuit (ASIC)) for performing the functions of one or more of the above-described embodiment(s), and by a method performed by the computer of the system or apparatus by, for example, reading out and executing the computer executable instructions from the storage medium to perform the functions of one or more of the above-described embodiment(s) and/or controlling the one or more circuits to perform the functions of one or more of the above-described embodiment(s). The computer may comprise one or more processors (e.g., central processing unit (CPU), micro processing unit (MPU)) and may include a network of separate computers or separate processors to read out and execute the computer executable instructions. The computer executable instructions may be provided to the computer, for example, from a network or the storage medium. The storage medium may include, for example, one or more of a hard disk, a random-access memory (RAM), a read only memory (ROM), a storage of distributed computing systems, an optical disk (such as a compact disc (CD), digital versatile disc (DVD), or Blu-ray Disc (BD)™), a flash memory device, a memory card, and the like.

While the present invention has been described with reference to exemplary embodiments, it is to be understood that the invention is not limited to the disclosed exemplary embodiments. The scope of the following claims is to be accorded the broadest interpretation so as to encompass all such modifications and equivalent structures and functions.

This application claims the benefit of Japanese Patent Application No. 2022-42125 filed on Mar. 17, 2022, which is hereby incorporated by reference wherein in its entirety.