Image Processing Method and Image Processing System

The present disclosure relates to an image processing method and so forth.

It has been proposed to apply compressed sensing to hyperspectral cameras, that is, to reconstruct a hyperspectral image from a compressed image. The following documents relate to such a technology: International Publication No. WO2022/202236 (Patent Document 1); International Publication No. WO2021/192891 (Patent Document 2); International Publication No. WO2020/080045 (Patent Document 3); “Establishing a World-First Technology for Capturing Hyperspectral Images and Videos by Combining Meta-lens and AI with Ordinary Digital Cameras~Turning an “ordinary camera” into a “camera that can see the nature of things” through the fusion of optical technology and AI~″, online, Oct. 24, 2022, Nippon Telegraph and Telephone Corporation, search on Nov. 16, 2023, Internet <URL:

https://group.ntt/en/newsrelease/2022/10/24/221024a.html> (Non-Patent Document 1); and Ahasan Ahamed et.al., “Reconstruction-based spectroscopy using CMOS image sensors with random photon-trapping nanostructure per sensor”, Proc. SPIE 11971, High-Speed Biomedical Imaging and Spectroscopy VII, 1197106, Mar. 2, 2022 (Non-Patent Document 2).

However, the amount of computation for reconstructing, from a compressed image, an image represented by information of four or more wavelength bands, such as a hyperspectral image, is enormous. Thus, the processing load is high, and also power consumption and energy consumption are high.

One non-limiting and exemplary embodiment provides an image processing method and so forth capable of reducing the processing load related to reconstruction of images. This makes it possible to reduce the amount of computation of one or more computers that execute the image processing method. That is, the performance of the one or more computers that execute the image processing method is improved.

In one general aspect, the techniques disclosed here feature an image processing method including: acquiring a compressed image; switching between a viewing mode and a save mode; in the viewing mode, displaying a first image on a display device and deleting the first image after displaying the first image, the first image being either the compressed image or a display-processed image that is generated based on the compressed image and represented by information of three or fewer wavelength bands; and in the save mode, generating, based on the compressed image, a second image represented by information of four or more wavelength bands, and saving the second image to a recording medium.

The image processing method according to an aspect of the present disclosure and so forth make it possible to reduce the processing load related to reconstruction of images.

It should be noted that general or specific embodiments may be implemented as a system, a device, a method, an integrated circuit, a computer program, a non-transitory recording medium such as a computer-readable compact disc read-only memory (CD-ROM), or any selective combination thereof.

Additional benefits and advantages of the disclosed embodiments will become apparent from the specification and drawings. The benefits and/or advantages may be individually obtained by the various embodiments and features of the specification and drawings, which need not all be provided in order to obtain one or more of such benefits and/or advantages.

For example, an RGB camera detects light and generates an RGB image represented by information of three wavelength bands corresponding to red, green, and blue. On the other hand, a hyperspectral camera detects light and generates a hyperspectral image represented by information of four or more wavelength bands. Hyperspectral cameras and hyperspectral images have been utilized in various fields, such as food inspection, biopsy, medical drug development, and mineral component analysis.

From the viewpoint of cost and flexibility, it has been proposed to apply compressed sensing to hyperspectral cameras, that is, to reconstruct a hyperspectral image from a compressed image. A compressed image herein is, for example, an image having pieces of information of four or more wavelength bands superimposed one on another. A hyperspectral image can be reconstructed from a compressed image in accordance with sparsity.

However, the amount of computation for reconstructing, from a compressed image, an image represented by information of four or more wavelength bands, such as a hyperspectral image, is enormous. Thus, the processing load is high, and also power consumption and energy consumption are high.

An image processing method of Example 1 includes: acquiring a compressed image; switching between a viewing mode and a save mode; in the viewing mode, displaying a first image on a display device and deleting the first image after displaying the first image, the first image being either the compressed image or a display-processed image that is generated based on the compressed image and represented by information of three or fewer wavelength bands; and in the save mode, generating, based on the compressed image, a second image represented by information of four or more wavelength bands, and saving the second image to a recording medium.

This makes it possible to, when saving is not performed, omit generation of the second image represented by information of four or more wavelength bands. Thus, it is possible to reduce the processing load, and reduce power consumption and energy consumption.

An image processing method of Example 2 may be an image processing method according to the image processing method of Example 1, in which the first image is the compressed image.

This makes it possible to display the compressed image as it is in the viewing mode. Thus, it is possible to further reduce the processing load.

An image processing method of Example 3 may be an image processing method according to the image processing method of Example 1, in which the first image is the display-processed image.

This makes it possible to generate the first image suitable for display with a small amount of computation in the viewing mode. Thus, it is possible to display the first image suitable for display, while reducing the processing load.

An image processing method of Example 4 may be an image processing method according to the image processing method of Example 3, in which the first image is an RGB image represented by information of three wavelength bands.

This makes it possible to generate the RGB image as the first image in the viewing mode. Thus, it is possible to display the first image more suitable for display.

An image processing method of Example 5 may be an image processing method according to the image processing method of Example 3, in which the first image is a monochrome image represented by information of one wavelength band.

This makes it possible to generate the monochrome image as the first image in the viewing mode. Thus, it is possible to further reduce the processing load.

An image processing method of Example 6 may be an image processing method according to the image processing method of any one of Examples 3 to 5, in which the first image has a lower resolution than the second image.

This makes it possible to generate the first image having a low resolution in the viewing mode. Thus, it is possible to further reduce the processing load.

An image processing method of Example 7 may be an image processing method according to the image processing method of any one of Examples 3 to 6, in which, in acquisition of the compressed image, the compressed image is acquired through light receiving regions having transmission spectra, in generation of the first image, the first image is generated based on the compressed image and first mask data that includes values reflecting the transmission spectra, in generation of the second image, the second image is generated based on the compressed image and second mask data that includes values reflecting the transmission spectra, and the values included in the first mask data are fewer than the values included in the second mask data.

This makes it possible to generate, in the viewing mode, the first image by using the first mask data that includes a smaller number of values than the second mask data used for generating the second image. Thus, it is possible to further reduce the processing load.

An image processing method of Example 8 may be an image processing method according to the image processing method of any one of Examples 1 to 7, in which the switching between the viewing mode and the save mode is performed based on an operation performed by a user.

This makes it possible to adaptively switch between the viewing mode and the save mode. Thus, it is possible to adaptively reduce the processing load.

An image processing method of Example 9 may be an image processing method according to the image processing method of any one of Examples 1 to 7, further including: in the viewing mode, determining whether the first image includes a specific subject, in which, in the switching between the viewing mode and the save mode, the viewing mode is continued in response to a determination being made that the first image does not include the specific subject, and switching from the viewing mode to the save mode is performed in response to a determination being made that the first image includes the specific subject.

This makes it possible to switch between the viewing mode and the save mode in accordance with whether the first image includes the specific subject. When the first image includes the specific subject, it is possible to save the second image including the specific subject. Thus, it is possible to efficiently save the second image including the specific subject.

An image processing method of Example 10 may be an image processing method according to the image processing method of any one of Examples 1 to 9, further including: in the save mode, displaying the first image on the display device.

This makes it possible to display the first image in both the viewing mode and the save mode. Thus, it is possible to display the first image in the save mode continuously from the viewing mode.

An image processing method of Example 11 may be an image processing method according to the image processing method of Example 10, further including: in the save mode, saving the first image to the recording medium.

This makes it possible to save the displayed first image in the save mode. Thus, it is possible to save the displayed first image together with the second image in the save mode.

An image processing method of Example 12 may be an image processing method according to the image processing method of Example 10 or 11, further including: in the save mode, displaying one or both of the second image and an analysis result of a subject based on the second image.

This makes it possible to display information corresponding to the second image in the save mode. Thus, it is possible to display the first image and information corresponding to the second image in the save mode.

An image processing method of Example 13 may be an image processing method according to the image processing method of any one of Examples 3 to 6, further including: in the save mode, generating the first image, based on the second image generated based on the compressed image, and displaying the first image on the display device.

This makes it possible to display the first image in both the viewing mode and the save mode. Thus, it is possible to display the first image in the save mode continuously from the viewing mode. Furthermore, it is possible to efficiently generate the first image based on the second image in the save mode. Thus, in the save mode, an increase in the processing load is suppressed.

An image processing system of Example 14 includes: an image sensor that acquires a compressed image; and a processing circuit that switches between a viewing mode and a save mode, in which, in the viewing mode, the processing circuit displays a first image on a display device and deletes the first image after displaying the first image, the first image being either the compressed image or a display-processed image that is generated based on the compressed image and represented by information of three or fewer wavelength bands, and in the save mode, the processing circuit generates, based on the compressed image, a second image represented by information of four or more wavelength bands, and saves the second image to a recording medium.

This makes it possible to, when saving is not performed, omit generation of the second image represented by information of four or more wavelength bands. Thus, it is possible to reduce the processing load, and reduce power consumption and energy consumption.

A program of Example 15 is a program for causing a computer to execute the image processing method of any one of Examples 1 to 13.

This makes it possible to implement the program for causing the computer to execute the foregoing image processing method.

Furthermore, these general or specific embodiments may be implemented as a system, a device, a method, an integrated circuit, a computer program, a non-transitory recording medium such as a computer-readable compact disc read-only memory (CD-ROM), or any selective combination thereof.

Hereinafter, embodiments will be described with reference to the drawings. The embodiments described below each illustrate a general or specific example. The numerical values, shapes, materials, components, arrangement positions and connection manner of the components, steps, order of the steps, etc., illustrated in the following embodiments are merely examples and are not intended to limit the scope of the claims.

1 FIG. 1 FIG. 100 111 121 100 130 140 100 110 120 110 111 120 121 is a block diagram illustrating a configuration example of an image processing system according to an embodiment. As illustrated in, an image processing systemincludes an image sensorand a processing circuit. The image processing systemmay further include a display deviceand a recording medium. The image processing systemmay include an imaging deviceand an image processing device. The imaging devicemay include the image sensor. The image processing devicemay include the processing circuit.

111 111 111 The image sensordetects optical signals on a pixel-by-pixel basis and generates an image represented by the optical signals corresponding to pixels, thereby acquiring the image. Here, the image sensoracquires a compressed image. Specifically, for example, the image sensoracquires a compressed image in which pieces of information of four or more wavelength bands are superimposed one on another on a pixel-by-pixel basis in accordance with a filter array, which will be described below. The wavelength bands may be simply referred to as bands.

111 111 The image sensormay be a monochrome photodetector including photodetection elements arranged in a matrix. More specifically, the image sensormay be a charge-coupled device (CCD) image sensor, a complementary metal-oxide-semiconductor (CMOS) image sensor, an infrared-array image sensor, a terahertz-array image sensor, or a millimeter-wave-array image sensor.

111 111 Alternatively, the image sensormay be a color photodetector. The wavelength range detectable by the image sensoris not limited and may be, for example, visible light, ultraviolet light, infrared light, terahertz waves, or any combination thereof.

121 121 The processing circuitis a circuit that performs information processing. For example, the processing circuitexecutes reconstruction computation on a compressed image to generate a hyperspectral image represented by information of four or more wavelength bands. For example, the hyperspectral image is constituted by four or more spectral images corresponding to the four or more wavelength bands. The reconstruction computation may be the same as that described in Patent Documents 2 and 3. Specifically, four or more spectral images may be generated as a hyperspectral image, based on the following expression (1).

Here, g represents data indicating a compressed image, and is represented by, for example, a one-dimensional array (i.e., a vector). When the compressed image is an image having n×m pixels, the data g is represented by a one-dimensional array having n×m elements, f represents data indicating w spectral images corresponding one-to-one to w wavelength bands and is represented by, for example, a one-dimensional array. When the w spectral images constitute a hyperspectral image, w is an integer greater than or equal to 4.

1 1 2 2 w frepresents data of a spectral image corresponding to a wavelength band W, frepresents data of a spectral image corresponding to a wavelength band W, . . . , and frepresents data of a spectral image corresponding to a wavelength band Ww.

1 2 w 1 2 w The data f, the data f, . . . , and the data fare each represented by, for example, a one-dimensional array. When each spectral image is an image having n×m pixels, the data f, the data f, . . . , and the data fare each represented by a one-dimensional array having n×m elements, and the data f is represented by a one-dimensional array having n×m×w elements. His a matrix of n×m rows and n×m×w columns, called a system matrix, and corresponds to mask data.

1 2 The matrix H may be determined based on the transmission spectrum of the wavelength band W, the transmission spectrum of the wavelength band W, . . . , and the transmission spectrum of the wavelength band Ww, of the filter array described below.

The data f satisfying expression (1) can be estimated by using a method of compressed sensing, and can specifically be estimated by using expression (2).

Expression (2) expresses calculating the data f that minimizes the sum of the first and second terms in the parentheses. The data f can be calculated as the data of a final computation result by converging the data of a computation result through recursive iterative computation.

The first term in the parentheses in expression (2) represents the sum of squared differences between data Hf, which is obtained by transforming the estimated data f with the matrix H, and the data g; this is a so-called residual term. While the sum of squares is used here, the sum of absolute values, the square root of the sum of squares, or the like may be used instead.

1 1 1 1 n×m n×m n×m n×m 1 n×m 1 n×m 1 n×m 1 n×m T T The sum of squared differences between the data Hf and the data g is expressed by (g−r)×(g−r)+ . . . +(g−r)×(g−r). Here, g, . . . , and gare each an element of the data g expressed by g=(g. . . g). Also, r, . . . , and rare each an element of the data Hf expressed by Hf=(r. . . r).

The second term in the parentheses in expression (2) is a regularization term, which may be referred to as a stabilization term. Φ(f) represents the constraint in the regularization of f and is a function that reflects sparse information of the data f. This function has an effect of smoothing or stabilizing the data f. Φ(f) can be represented by, for example, discrete cosine transform (DCT), wavelet transform, Fourier transform, total variation (TV), or any combination thereof.

τ is a weighting coefficient of the regularization term and corresponds to the degree of influence of regularization in reconstruction computation. As the value of τ increases, the influence of regularization becomes stronger, and the convergence of the solution in iterative computation increases. Conversely, as the value of τ decreases, the influence of regularization becomes weaker, and the convergence of the solution in iterative computation decreases.

130 130 121 130 The display deviceis a display for displaying information. On the display device, a compressed image, a monochrome image, an RGB image, a hyperspectral image, or the like is displayed by the processing circuit. The display devicemay be, for example, a liquid crystal display (LCD), an organic electroluminescence (organic light-emitting diode (OLED)) display, or the like.

130 130 130 121 130 130 121 130 The display devicemay display a graphical user interface (GUI). The display devicemay be a touch screen. The display devicemay receive information from a user. The processing circuitmay acquire information from a user via the display device. That is, the display devicemay be an input/output device. Alternatively, the processing circuitmay acquire information from a user via an input device different from the display device.

140 140 121 140 The recording mediumis a storage element for storing information. To the recording medium, a compressed image, a monochrome image, an RGB image, a hyperspectral image, or the like is saved by the processing circuit. The recording mediumcan be implemented by, for example, a compact disc read-only memory (CD-ROM), a hard disk drive, a solid-state drive, or the like.

1 FIG. 1 FIG. 100 100 illustrates a configuration example of the image processing system. The configuration of the image processing systemis not limited to the configuration example in. For example, multiple devices may be integrated into a single device, or a single device may be implemented as multiple distributed devices. The multiple distributed devices may be capable of communicating with each other by wired or wireless communication.

121 111 130 140 121 120 The processing circuitcorresponds to an acquirer that acquires an image from the image sensor, a switcher that switches between modes, a generator that generates an image, a discard controller that discards an image, a display controller that displays an image on the display device, a save controller that saves an image to the recording medium, and so forth. Instead of the processing circuit, the image processing devicemay include some or all of these components.

2 FIG. 1 FIG. 110 110 110 111 112 113 is a conceptual diagram illustrating a configuration example of the imaging deviceillustrated in. The imaging devicecan have a configuration similar to those of the imaging devices disclosed in Patent Documents 2 and 3. For example, the imaging deviceincludes the image sensor, a filter array, and an optical system.

112 113 111 112 112 111 The filter arrayis disposed on an optical path of light incident from a subject and is disposed between the optical systemand the image sensor. The filter arrayfunctions as the encoding element described in Patent Document 2. The filter arraymay be integrated with the image sensor.

112 112 111 113 111 112 113 112 113 2 FIG. The disposition of the filter arrayis not limited to the disposition in. For example, the filter arraymay be disposed away from the image sensor, between the optical systemand the image sensor. For example, the filter arraymay be disposed between the subject and the optical system. For example, the filter arraymay be disposed within the optical system.

113 112 113 111 112 The optical systemis disposed on an optical path of light incident from the subject and is disposed between the subject and the filter array. The optical systemincludes at least one lens and is capable of forming an image of the subject on the imaging surface of image sensorthrough the filter array.

113 113 112 111 113 112 2 FIG. The configuration and disposition of the optical systemare not limited to the configuration and disposition in. For example, the optical systemmay be disposed between the filter arrayand the image sensor. Furthermore, for example, the optical systemmay include lenses arranged on the optical path. In this case, the filter arraymay be disposed between adjacent lenses of these lenses.

3 FIG. 2 FIG. 3 FIG. 112 112 112 11 nm 11 nm 11 11 nm nm 11 nm is a conceptual diagram illustrating a configuration example of the filter arrayillustrated in. The filter arrayincludes filters F, . . . , and Farranged in a matrix. In the example illustrated in, the filter arrayincludes forty-eight filters F, . . . , and Farranged in six rows and eight columns. The filter Fis a filter disposed at the upper left of the forty-eight filters F, . . . , and F, and the filter Fis a filter disposed at the lower right of the forty-eight filters F, . . . , and F.

11 nm 11 nm 11 nm 11 nm 112 112 111 112 111 111 The number of the filters F, . . . , and Fincluded in the filter arrayis not limited to forty-eight. For example, the number of the filters F, . . . , and Fincluded in the filter arraymay be equivalent to the number of the pixels of the image sensor, and can be determined in accordance with the application, for example, in the range from several tens to several tens of millions. The number of the filters F, . . . , and Fincluded in the filter arraymay be the same as or different from the number of the pixels of the image sensor. The pixels included in the image sensormay correspond one-to-one to the filters F, . . . , and F.

112 11 nm 1 11 nm 11 nm 11 nm 11 nm For example, the filter arrayincludes n×m filters F, . . . , and Fcorresponding to n×m pixels. In the wavelength bands W, . . . , and Ww, the filters F, . . . , and Fhave transmission spectra S, . . . , and S, respectively. The transmission spectra S, . . . , and Smay all be different. Alternatively, some of the transmission spectra S, . . . , and Smay be the same. Here, the transmission spectrum can mean an optical transmittance spectrum.

11 nm 11W1 nmWw 1 w 11W1 nmWw 11W1 nmWw The n×m filters F, . . . , and Fhave n×m×w transmittances S, . . . , and Sfor the w wavelength bands W, . . . , and W. The transmittances S, . . . , and Smay all be different. Alternatively, some of the transmittances S, . . . , and Smay be the same. Here, the transmittance can mean optical transmittance.

α The transmittance of a filter β in the wavelength band Wmay be expressed by the following expression (3).

α α Here, Wαmin represents the minimum wavelength value of the wavelength band W, Wαmax is the maximum wavelength value of the wavelength band W, h(λ) represents a function indicating the transmission spectrum, and λ represents the wavelength.

α α α α0 α0 α The transmittance of the filter B in the wavelength band Wis not limited to the one expressed by expression (3). For example, the transmittance of the filter β in the wavelength band Wmay be the transmittance obtained by dividing expression (3) by (Wαmax−Wαmin). For example, the transmittance of the filter β in the wavelength band Wmay be the transmittance h(λ) in the wavelength λrepresenting the wavelength band W.

α0 α0 α Here, the wavelength λis a wavelength that satisfies Wαmin≤λ<Wαmax. For example, the wavelength Ago may be the center wavelength of the wavelength band W((Wαmax−Wαmin)/2).

4 FIG. 3 FIG. 5 FIG. 3 FIG. 6 FIG. 3 FIG. 7 FIG. 3 FIG. 6 7 FIGS.and 11 nm 1 2 112 112 illustrates an example of the transmission spectrum of the filter Fillustrated in.illustrates an example of the transmission spectrum of the filter Fillustrated in.is a diagram illustrating an example of the transmittance of the wavelength band Wof the filter arrayillustrated in.is a diagram illustrating an example of the transmittance of the wavelength band Wof the filter arrayillustrated in. In, the shading in each region represents the transmittance of the filter, with lighter regions indicating higher transmittance and darker regions indicating lower transmittance.

11 nm 11 1 2 nm 1 2 11 nm 112 The wavelength dependence of the transmittance varies among the filters F, . . . , and Fincluded in the filter array. For example, in the filter F, the transmittance of the wavelength band Wis considerably lower than the transmittance of the wavelength band W. On the other hand, in the filter F, the transmittance of the wavelength band Wis substantially the same as the transmittance of the wavelength band W. In other words, the wavelength dependence of the transmittance of the filter Fis different from the wavelength dependence of the transmittance of the filter F.

1 2 1 w Here, the illustration and description will be given of the transmittances of the two wavelength bands Wand Wof the w wavelength bands W, . . . , and W, and the illustration and description of the transmittances of the other wavelength bands will be omitted.

1 2 w 1 2 w For example, mask data is represented by w matrices corresponding to the w wavelength bands W, W, . . . , and W. Specifically, mask data is represented by a transmittance matrix of n rows and m columns corresponding to pixels in n rows and m columns, for each of the w wavelength bands W, W, . . . , and W. As a result of changing a representation form from the mask data represented by w matrices each having n rows and m columns, one matrix H having n×m rows and n×m×w columns is obtained.

1 2 w 1 2 w H included in expression (1) may be expressed by H=(HH. . . H). Each of H, H, . . . , and, His a submatrix of H. That is, when the components included in H are expressed by a(11), . . . , and a ((n×m)(n×m×w)), the following expression holds.

1 1 2 2 w w Hmay be interpreted as the mask data for the wavelength band W, Hmay be interpreted as the mask data for the wavelength band W, . . . , and Hmay be interpreted as the mask data for the wavelength band W.

1 2 w 1 2 w Each of H, H, . . . , and Hmay be a diagonal matrix. That is, H, H, . . . , and Hmay be expressed as follows.

8 FIG. 1 FIG. 100 111 110 121 120 111 101 111 121 111 is a flowchart illustrating an operation example of the image processing systemillustrated in. In this example, the image sensorof the imaging devicegenerates a compressed image to acquire the compressed image, and the processing circuitof the image processing deviceacquires the compressed image from the image sensor(S). That is, the operation of acquiring a compressed image may correspond to the operation in which the image sensorgenerates a compressed image to acquire the compressed image, or the operation in which the processing circuitacquires the compressed image from the image sensor.

121 102 121 Subsequently, the processing circuitswitches between a viewing mode and a save mode (S). Here, the processing circuitmay switch the mode from the viewing mode to the save mode, may switch the mode from the save mode to the viewing mode, may maintain the viewing mode, or may maintain the save mode.

103 121 130 104 121 105 In the viewing mode (viewing mode in S), the processing circuitdisplays a first image on the display device(S). Here, the first image may be the compressed image. Alternatively, the first image may be a display-processed image that is generated based on the compressed image and represented by information of three or fewer wavelength bands. After displaying the first image, the processing circuitdeletes the first image (S).

103 121 106 121 140 107 In the save mode (save mode in S), the processing circuitgenerates a second image based on the compressed image (S). The second image is an image represented by information of four or more wavelength bands. Subsequently, the processing circuitsaves the second image to the recording medium(S).

Accordingly, when saving is not performed, generation of the second image represented by information of four or more wavelength bands can be omitted. Thus, it is possible to reduce the processing load, and reduce power consumption and energy consumption.

101 107 For example, the above-described series of operations (Sto S) may be performed repeatedly. In the viewing mode, first images can be displayed as a video, and in the save mode, second images can be saved as a video.

For example, when the first image is a compressed image, the compressed image can be displayed as it is in the viewing mode. Thus, it is possible to further reduce the processing load.

121 For example, when the first image is a display-processed image, the processing circuitmay generate the first image based on the compressed image in the viewing mode. This makes it possible to generate the first image suitable for display with a small amount of computation in the viewing mode. Thus, it is possible to display the first image suitable for display, while reducing the processing load.

For example, the first image may be an RGB image represented by information of three wavelength bands. This makes it possible to generate the RGB image as the first image in the viewing mode. Thus, it is possible to display the first image more suitable for display.

For example, the first image may be a monochrome image represented by information of one wavelength band. This makes it possible to generate the monochrome image as the first image in the viewing mode. Thus, it is possible to further reduce the processing load.

For example, the first image may have a lower resolution than the second image. This makes it possible to generate the first image having a low resolution in the viewing mode. Thus, it is possible to further reduce the processing load.

111 111 112 112 For example, the image sensormay acquire the compressed image through light receiving regions having transmission spectra. Specifically, the image sensormay acquire the compressed image through the filter array. Each of the light receiving regions may correspond to a corresponding one of the filters included in the filter array.

112 112 Alternatively, the meta-lens described in Non-Patent Document 1 may be used instead of the filter array. In this case, the wavelength transmittance varies depending on a location. The light receiving region corresponds to such a location. Alternatively, the CMOS image sensor described in Non-Patent Document 2 may be used instead of the filter array. In this case, a sensing region is processed so that a predetermined transmittance is obtained in each pixel. The light receiving region corresponds to such a sensing region.

The light receiving region may be represented as an optical element, a modulation element, an encoding element, an optical processing region, a masking region, or the like.

121 121 The processing circuitmay generate the first image based on the compressed image and first mask data. Here, the first mask data includes values reflecting the transmission spectra. The processing circuitmay generate the second image based on the compressed image and second mask data. Here, the second mask data includes values reflecting the transmission spectra. The number of the values included in the first mask data is smaller than the number of the values included in the second mask data.

This makes it possible to generate, in the viewing mode, the first image by using the first mask data that includes a smaller number of values than the second mask data used for generating the second image. Thus, it is possible to further reduce the processing load.

For example, the first mask data may be generated in accordance with integration of the matrix elements of the second mask data or the like. This makes it possible to reduce the number of the values included in the first mask data. It is also possible to efficiently generate both the first mask data for generating the first image and the second mask data for generating the second image from the common transmission spectra. Thus, it is possible to reduce the number of calibrations.

121 For example, the processing circuitmay switch between the viewing mode and the save mode in accordance with an operation performed by a user. This makes it possible to adaptively switch between the viewing mode and the save mode. Thus, it is possible to adaptively reduce the processing load.

121 121 121 Furthermore, for example, the processing circuitmay determine, in the viewing mode, whether the first image includes a specific subject. In response to determining that the first image does not include the specific subject, the processing circuitmay continue the viewing mode. On the other hand, in response to determining that the first image includes the specific subject, the processing circuitmay switch the mode from the viewing mode to the save mode.

This makes it possible to switch between the viewing mode and the save mode in accordance with whether the first image includes the specific subject. When the first image includes the specific subject, it is possible to save the second image including the specific subject. Thus, it is possible to efficiently save the second image including the specific subject.

121 130 For example, the processing circuitmay display the first image on the display devicein the save mode. This makes it possible to display the first image in both the viewing mode and the save mode. Thus, it is possible to display the first image in the save mode continuously from the viewing mode.

121 140 For example, the processing circuitmay save the first image to the recording mediumin the save mode. This makes it possible to save the displayed first image in the save mode. Thus, it is possible to save the displayed first image together with the second image in the save mode.

121 For example, the processing circuitmay display one or both of the second image and an analysis result of a subject based on the second image in the save mode. This makes it possible to display information corresponding to the second image in the save mode. Thus, it is possible to display the first image and information corresponding to the second image in the save mode.

121 121 130 For example, the processing circuitmay generate the first image based on the second image generated based on the compressed image in the save mode. The processing circuitmay display the first image on the display device.

This makes it possible to display the first image in both the viewing mode and the save mode. Thus, it is possible to display the first image in the save mode continuously from the viewing mode. Furthermore, it is possible to efficiently generate the first image based on the second image in the save mode. Thus, in the save mode, an increase in the processing load is suppressed.

9 FIG. 1 FIG. 100 111 is a conceptual diagram illustrating an operation example of the image processing systemillustrated in. For example, the image sensorrepeats imaging (photography) in accordance with a frame rate, thereby repeatedly acquiring a compressed image.

121 The processing circuitselects a mode for each frame to switch between the viewing mode and the save mode. The viewing mode is also referred to as a display mode, a viewer mode, or an imaging (photography) mode, in which no image is saved. The save mode is also referred to as a recording mode, in which an image is saved.

121 111 130 121 In the viewing mode, the processing circuitrepeats operations of acquiring a compressed image via the image sensor, reconstructing a first image from the compressed image, displaying the first image on the display device, and discarding the first image. The first image may be an RGB image or a monochrome image. Alternatively, the first image may be the compressed image. In this case, the processing circuitdoes not necessarily have to perform reconstruction processing in the viewing mode.

121 111 140 121 130 In the save mode, the processing circuitrepeats operations of acquiring a compressed image via the image sensor, reconstructing a second image from the compressed image, and saving the second image to the recording medium. The second image may be a hyperspectral image. Furthermore, in the save mode, the processing circuitmay repeat operations of generating a first image from a second image, displaying the first image on the display device, and discarding the first image.

121 121 Alternatively, in the save mode, the processing circuitmay reconstruct a first image from a compressed image as in the viewing mode. When the first image is a compressed image, the process of generating (reconstructing) the first image does not necessarily have to be performed. Alternatively, in the save mode, the processing circuitmay display the second image instead of the first image.

121 130 140 121 140 In the save mode, the processing circuitmay save the first image displayed on the display deviceto the recording mediumin addition to the second image. That is, the processing circuitmay save the first image to the recording mediumwithout discarding the first image.

10 FIG. 9 FIG. is an explanatory diagram illustrating an example of combinations of the type of image to be displayed and the type of image to be saved in the operation example illustrated in.

Specifically, the image to be displayed is a first image. The image to be saved may be a second image or both a first image and a second image. The first image is, for example, a compressed image, a monochrome image, an RGB image, a low-resolution monochrome image, or a low-resolution RGB image. The second image is, for example, a hyperspectral image.

When the image to be displayed is a monochrome image, an RGB image, a low-resolution monochrome image, or a low-resolution RGB image, the image to be saved may include a compressed image. That is, in this case, the image to be saved may be two images, a hyperspectral image and a compressed image, or may be three images, a hyperspectral image, an image to be displayed, and a compressed image.

In any of these examples, when a hyperspectral image is not saved, that is, when the mode is not the save mode, no image is saved.

11 FIG. 1 FIG. 11 FIG. 130 130 121 is a conceptual diagram illustrating a first display example of the display deviceillustrated in. For example, the display devicedisplays the GUI illustrated in. The GUI includes a first image, a display button, and a save button. The first image included in the GUI is a first image generated (reconstructed) by the processing circuit. The display button and the save button are each switched between ON and OFF by a user operation.

Switching between ON and OFF of the display button and the save button enables switching between the viewing mode and the save mode. The display button may be a photography button or a viewing button. The save button may be a recording button.

12 FIG. is an explanatory diagram illustrating the relationship among the display button, the save button, and the mode according to the embodiment. When the display button is OFF, the save button is also OFF, and the mode has not been selected. In this case, any image is not generated and any image is not displayed.

When the display button is ON, the save button can be switched to ON or OFF. When the display button is ON and the save button is OFF, the mode is the viewing mode. When the display button is ON and the save button is ON, the mode is the save mode.

For example, when the display button is ON, a first image is displayed, and when the save button is ON, a second image is saved.

13 FIG. 1 FIG. 100 is a flowchart illustrating a first specific example of the operation of the image processing systemillustrated in.

121 201 In this example, the processing circuitdetermines whether the display button is ON (S).

201 121 202 111 121 111 121 203 121 204 When the display button is ON (Yes in S), the processing circuitacquires a compressed image (S). Specifically, the image sensorgenerates a compressed image to acquire the compressed image, and the processing circuitacquires the compressed image from the image sensor. Subsequently, the processing circuitdiscards the compressed image displayed in the preceding frame (S). Subsequently, the processing circuitdisplays the compressed image acquired in the current frame (S).

121 205 205 121 206 201 206 205 201 205 Furthermore, the processing circuitdetermines whether the save button is ON (S). When the save button is ON (Yes in S), the processing circuitreconstructs a hyperspectral image by using mask data of four or more wavelength bands, and saves the hyperspectral image (S). Subsequently, the series of steps (Sto S) is repeated. On the other hand, when the save button is OFF (No in S), the process (Sto S) is repeated without a hyperspectral image being reconstructed.

201 121 209 100 When the display button is OFF (No in S), the processing circuitdiscards the compressed image displayed in the preceding frame (S). Subsequently, the image processing systemends the operation.

13 FIG. In the example in, a hyperspectral image is reconstructed and saved when the save button is ON. When the display button is ON and the save button is OFF, a compressed image is displayed and a hyperspectral image is not reconstructed. Thus, the processing load is reduced.

14 FIG. 1 FIG. 100 is a flowchart illustrating a second specific example of the operation of the image processing systemillustrated in.

14 FIG. 13 FIG. 121 204 204 a In the example in, compared to the example in, the processing circuitreconstructs the RGB image of the current frame from the compressed image of the current frame by using mask data of three wavelength bands corresponding to the RGB image and displays the RGB image (S), instead of displaying the compressed image (S).

204 121 203 209 203 209 121 a a a In accordance with the reconstruction and display of the RGB image (S), the processing circuitdiscards the RGB image displayed in the preceding frame (Sand S), instead of discarding the compressed image displayed in the preceding frame (Sand S). At that time, the processing circuitmay discard the compressed image used in the preceding frame.

14 FIG. 13 FIG. 14 FIG. Except for the above, the example inis the same as the example in. In the example in, a hyperspectral image is reconstructed and saved when the save button is ON. When the display button is ON and the save button is OFF, an RGB image is reconstructed and displayed, and a hyperspectral image is not reconstructed. Thus, the processing load is reduced.

15 FIG. The reconstruction of an RGB image is performed in a similar manner to the reconstruction of a hyperspectral image by using mask data of three wavelength bands corresponding to the RGB image instead of mask data of four or more wavelength bands. For example, the reconstruction computation described in Patent Document 2 may be used. Specifically, a description will be given with reference to.

15 FIG. 15 FIG. 111 111 is a conceptual diagram illustrating the relationship between four or more wavelength bands and three wavelength bands. In, W represents a wavelength range detectable by the image sensor. The wavelength range W may be the transmission wavelength range of a bandpass filter included in the image sensor.

1 w R G B 1 w R G B The wavelength range W corresponds to the range covered by w (w is 4 or more) wavelength bands W, . . . , and Wof a hyperspectral image. The wavelength range W corresponds to the range covered by three wavelength bands W, W, and Wof an RGB image. That is, the range covered by the wavelength bands W, . . . , and Wcorresponds to the range covered by the three wavelength bands W, W, and Wof an RGB image.

1 w 1 2 w 1 w 1 w For example, the wavelength range W is in the range from 400 nm to 700 nm. For example, the w wavelength bands W, . . . , and Ware determined in 10 nm steps. Thus, w is 30, and the w wavelength bands W, W, . . . , and Ware 400 nm to 410 nm, 410 nm to 420 nm, . . . , and 690 nm to 700 nm, respectively. To generate a hyperspectral image of these w wavelength bands W, . . . , and W, mask data of the w wavelength bands W, . . . , and Wis prepared.

1 2 w The mask data of the w wavelength bands W, W, . . . , and Wincludes a transmittance matrix of the wavelength band of 400 nm to 410 nm, a transmittance matrix of the wavelength band of 410 nm to 420 nm, . . . , and a transmittance matrix of the wavelength band of 690 nm to 700 nm.

B B G G R R As a result of integrating ten matrices included in the range of the wavelength band Wof 400 nm to 500 nm, a transmittance matrix of the wavelength band Wis obtained. As a result of integrating ten matrices included in the range of the wavelength band Wof 500 nm to 600 nm, a transmittance matrix of the wavelength band Wis obtained. As a result of integrating ten matrices included in the range of the wavelength band Wof 600 nm to 700 nm, a transmittance matrix of the wavelength band Wis obtained.

R G B 1 w R G B 1 w In the integration of ten transmittance matrices, ten transmittances may be added together or ten transmittances may be averaged for each element. Accordingly, the mask data of the three wavelength bands W, W, and Wcorresponding to an RGB image is obtained from the mask data of the w wavelength bands W, . . . , and W. That is, the mask data of the three wavelength bands W, W, and Wof an RGB image can be prepared from the mask data of the w wavelength bands W, . . . , and Win order to generate the RGB image.

1 2 w 1 2 P P+1 P+2 Q Q+1 Q+2 w R G B 1 2 P P+1 P+2 Q+1 Q+2 w When H included in expression (1) is expressed by H=(HH. . . . H)=(HH. . . HHH. . . HHH. . . H), a matrix HR indicating the mask data for the wavelength band W, a matrix HG indicating the mask data for the wavelength band W, and a matrix HB indicating the mask data for the wavelength band Wmay be expressed by expression (4) or expression (5). Each of H, H, . . . , H, H, H, . . . , HQ, H, H, . . . , and His a submatrix of H and has components arranged in n×m rows and n×m columns.

In the above example, P=10, Q=20, and W=30.

121 R G B The processing circuitis capable of generating an RGB image, based on the mask data of the three wavelength bands W, W, and Wand the foregoing expressions (1) and (2). H and f included in expression (1) and expression (2) may be expressed as follows.

B G R H, H, and Hare each a submatrix of H.

R R R frepresents the data of a spectral image corresponding to the wavelength band W, that is, the pixel values of the image corresponding to the wavelength band W.

G G G frepresents the data of a spectral image corresponding to the wavelength band W, that is, the pixel values of the image corresponding to the wavelength band W.

B B B frepresents the data of a spectral image corresponding to the wavelength band W, that is, the pixel values of the image corresponding to the wavelength band W.

R G B The pixel values included in the RGB image may be determined based on the n×m pixel values included in each of f, f, and f.

1 w 11 nm R B G 11 nm The mask data of the w wavelength bands W, . . . , and Whas n×m×w (w is 4 or more) transmittance values reflecting n×m transmission spectra S, . . . , and Sfor n×m pixels. Similarly, the mask data of the three wavelength bands W, W, and Whas n×m×3 transmittance values reflecting n×m transmission spectra S, . . . , and Sfor n×m pixels.

That is, it is possible to efficiently generate both the mask data for generating an RGB image and the mask data for generating a hyperspectral image from common transmission spectra. Thus, it is possible to reduce the number of calibrations.

1 w R B G R B G R G B In the above description, the mask data of the w wavelength bands W, . . . , and Wof the hyperspectral image is used to derive the mask data of the three wavelength bands W, W, and Wof the RGB image. However, the three transmittances of the three wavelength bands W, W, and Wmay be derived from the transmission spectra for each pixel, and thereby the mask data of the three wavelength bands W, W, and Wmay be derived, based on the foregoing expression (3).

16 FIG. 1 FIG. 100 is a flowchart illustrating a third specific example of the operation of the image processing systemillustrated in.

16 FIG. 14 FIG. 205 121 208 In the example in, compared to the example in, when the save button is ON (Yes in S), the processing circuitsaves an RGB image in addition to a hyperspectral image (S).

16 FIG. 14 FIG. 16 FIG. Except for the above, the example inis the same as the example in. In the example in, a hyperspectral image is reconstructed and saved when the save button is ON. When the display button is ON and the save button is OFF, an RGB image is reconstructed and displayed, and a hyperspectral image is not reconstructed. Thus, the processing load is reduced.

16 FIG. In the example in, when the save button is ON, the displayed RGB image is saved. Thus, in the save mode, it is possible to save the RGB image that is effective for display, together with the hyperspectral image.

17 FIG. 1 FIG. 17 FIG. 14 FIG. 100 is a flowchart illustrating a fourth specific example of the operation of the image processing systemillustrated in. In the example in, the method of generating an RGB image is different from that in the example in.

17 FIG. 17 FIG. 14 FIG. 14 FIG. 205 204 204 204 a a a Specifically, in the example in, the method of generating an RGB image varies depending on whether the save button is ON. Specifically, when the save button is OFF (No in S), an RGB image is reconstructed by using the mask data of three wavelength bands and is displayed (S). This process (Sin) is the same as the process that is performed regardless of whether the save button is ON in the example in(Sin).

205 121 207 a R R On the other hand, when the save button is ON (Yes in S), the processing circuitreconstructs a hyperspectral image, and then generates an RGB image from the hyperspectral image and displays the RGB image (S). Specifically, among the spectral images included in the hyperspectral image, spectral images of wavelength bands included in the red wavelength band Ware integrated, and thereby a spectral image of the red wavelength band Wis obtained. In the integration of the spectral images, the values of the spectral images may be added together, weighted added, or averaged, on a pixel-by-pixel basis.

G G B B Similarly, among the spectral images included in the hyperspectral image, spectral images of wavelength bands included in the green wavelength band Ware integrated, and thereby a spectral image of the green wavelength band Wis obtained. Furthermore, among the spectral images included in the hyperspectral image, spectral images of wavelength bands included in the blue wavelength band Ware integrated, and thereby a spectral image of the blue wavelength band Wis obtained.

R G B Subsequently, the spectral image of the red wavelength band W, the spectral image of the green wavelength band W, and the spectral image of the blue wavelength band Ware combined to obtain an RGB image.

17 FIG. 14 FIG. 17 FIG. Except for the above, the example inis the same as the example in. In the example in, a hyperspectral image is reconstructed and saved when the save button is ON. When the display button is ON and the save button is OFF, an RGB image is reconstructed and displayed, and a hyperspectral image is not reconstructed. Thus, the processing load is reduced.

17 FIG. In the example in, when the save button is ON, an RGB image is generated from the reconstructed hyperspectral image. The processing load of such a generation process is smaller than the processing load of a reconstruction process of reconstructing an RGB image from a compressed image. Thus, the above operation reduces the processing load.

18 FIG. 1 FIG. 100 is a flowchart illustrating a fifth specific example of the operation of the image processing systemillustrated in.

18 FIG. 17 FIG. 205 121 208 a In the example in, compared to the example in, when the save button is ON (Yes in S), the processing circuitsaves, in addition to a hyperspectral image, an RGB image generated from the hyperspectral image (S).

18 FIG. 17 FIG. 18 FIG. Except for the above, the example inis the same as the example in. In the example in, a hyperspectral image is reconstructed and saved when the save button is ON. When the display button is ON and the save button is OFF, an RGB image is reconstructed and displayed, and a hyperspectral image is not reconstructed. Thus, the processing load is reduced.

18 FIG. In the example in, when the save button is ON, the displayed RGB image is saved. Thus, in the save mode, it is possible to save the RGB image that is effective for display, together with the hyperspectral image.

19 FIG. 1 FIG. 100 is a flowchart illustrating a sixth specific example of the operation of the image processing systemillustrated in.

19 FIG. 14 FIG. 121 204 204 121 b a In the example in, compared to the example in, the processing circuitreconstructs and displays a monochrome image (S), instead of reconstructing and displaying an RGB image (S). Specifically, at that time, the processing circuitreconstructs a monochrome image of the current frame from a compressed image of the current frame by using the mask data of one wavelength band corresponding to the monochrome image, and displays the monochrome image.

204 121 203 209 203 209 121 b b b a a In accordance with the reconstruction and display of the monochrome image (S), the processing circuitdiscards the monochrome image displayed in the preceding frame (Sand S), instead of discarding the RGB image displayed in the preceding frame (Sand S). At that time, the processing circuitmay discard the compressed image used in the preceding frame.

19 FIG. 14 FIG. 19 FIG. Except for the above, the example inis the same as the example in. In the example in, a hyperspectral image is reconstructed and saved when the save button is ON. When the display button is ON and the save button is OFF, a monochrome image is reconstructed and displayed, and a hyperspectral image is not reconstructed. Thus, the processing load is reduced.

The reconstruction of a monochrome image is performed in a similar manner to the reconstruction of a hyperspectral image by using mask data of one wavelength band corresponding to the monochrome image instead of mask data of four or more wavelength bands. Specifically, mask data of four or more wavelength bands may be integrated as in the reconstruction of an RGB image.

In the reconstruction of an RGB image, mask data of four or more wavelength bands is integrated into mask data of three wavelength bands with fewer elements. In the reconstruction of a monochrome image, mask data of four or more wavelength bands is integrated into mask data of one wavelength band with fewer elements.

111 Alternatively, one transmittance of one wavelength band may be derived from the transmission spectrum on a pixel-by-pixel basis in accordance with the foregoing expression (3), and thereby the mask data of one wavelength band may be derived. The one wavelength band may correspond to the wavelength range W detectable by the image sensor.

As a result of using a monochrome image, the amount of computation is reduced and the processing load is reduced compared to the case of using an RGB image.

205 121 140 When the save button is ON (Yes in S), the processing circuitmay save, in addition to the hyperspectral image, the displayed monochrome image to the recording medium. This makes it possible to save the monochrome image that is effective for display, together with the hyperspectral image, in the save mode.

20 FIG. 1 FIG. 20 FIG. 19 FIG. 100 is a flowchart illustrating a seventh specific example of the operation of the image processing systemillustrated in. In the example in, the method of generating a monochrome image is different from that in the example in.

20 FIG. 20 FIG. 19 FIG. 19 FIG. 205 204 204 204 b b b Specifically, in the example in, the method of generating a monochrome image varies depending on whether the save button is ON. Specifically, when the save button is OFF (No in S), a monochrome image is reconstructed by using the mask data of one wavelength band and is displayed (S). This process (Sin) is the same as the process that is performed regardless of whether the save button is ON in the example in(Sin).

205 121 207 b On the other hand, when the save button is ON (Yes in S), the processing circuitreconstructs a hyperspectral image, and then generates a monochrome image from the hyperspectral image and displays the monochrome image (S). Specifically, the spectral images included in the hyperspectral image are integrated, and thereby a monochrome image is obtained. In the integration of the spectral images, the values of the spectral images may be added together or averaged, on a pixel-by-pixel basis.

20 FIG. 19 FIG. 20 FIG. Except for the above, the example inis the same as the example in. In the example in, a hyperspectral image is reconstructed and saved when the save button is ON. When the display button is ON and the save button is OFF, a monochrome image is reconstructed and displayed, and a hyperspectral image is not reconstructed. Thus, the processing load is reduced.

20 FIG. In the example in, when the save button is ON, a monochrome image is generated from the reconstructed hyperspectral image. The processing load of such a generation process is smaller than the processing load of a reconstruction process of reconstructing a monochrome image from a compressed image. Thus, the above operation reduces the processing load.

205 121 140 When the save button is ON (Yes in S), the processing circuitmay save, in addition to the hyperspectral image, the displayed monochrome image to the recording medium. This makes it possible to save the monochrome image that is effective for display, together with the hyperspectral image, in the save mode.

21 FIG. 1 FIG. 100 is a flowchart illustrating an eighth specific example of the operation of the image processing systemillustrated in.

21 FIG. 14 FIG. 121 204 204 121 c a In the example in, compared to the example in, the processing circuitreconstructs and displays a low-resolution RGB image (S), instead of reconstructing and displaying an RGB image (S). Specifically, at that time, the processing circuitreconstructs a low-resolution RGB image of the current frame from a compressed image of the current frame by using low-resolution mask data of three wavelength bands corresponding to the RGB image, and displays the low-resolution RGB image.

204 121 203 209 203 209 121 c c c a a In accordance with the reconstruction and display of the low-resolution RGB image (S), the processing circuitdiscards the low-resolution RGB image displayed in the preceding frame (Sand S), instead of discarding the RGB image displayed in the preceding frame (Sand S). At that time, the processing circuitmay discard the compressed image used in the preceding frame.

21 FIG. 14 FIG. 21 FIG. Except for the above, the example inis the same as the example in. In the example in, a hyperspectral image is reconstructed and saved when the save button is ON. When the display button is ON and the save button is OFF, a low-resolution RGB image is reconstructed and displayed, and a hyperspectral image is not reconstructed. Thus, the processing load is reduced. In addition, the amount of data of the low-resolution RGB image is smaller than that of the original RGB image. Thus, the processing load is reduced.

205 121 140 When the save button is ON (Yes in S), the processing circuitmay save, in addition to the hyperspectral image, the displayed low-resolution RGB image to the recording medium. This makes it possible to save the low-resolution RGB image that is effective for display, together with the hyperspectral image, in the save mode.

There are three methods of reconstructing a low-resolution RGB image. The first method is a method of averaging both the mask data and the compressed image in the spatial direction and performing reconstruction computation of compressed sensing, thereby reconstructing a low-resolution RGB image.

14 FIG. 15 FIG. Specifically, first, mask data of three wavelength bands corresponding to an RGB image is derived by using the method described with reference toand. The mask data is averaged in the spatial direction, and thereby low-resolution mask data is obtained. Similarly, the compressed image is averaged in the spatial direction, and thereby a low-resolution compressed image is obtained. Subsequently, reconstruction computation of compressed sensing is performed by using the low-resolution mask data and the low-resolution compressed image, and thereby a low-resolution RGB image is derived.

Averaging in the spatial direction is performed by, for example, averaging four values into one value in units of 2×2-pixel blocks. Averaging may be performed in units of blocks other than 2×2-pixel blocks.

The second method is a method of subsampling both the mask data and the compressed image in the spatial direction and performing reconstruction computation of compressed sensing, thereby reconstructing a low-resolution RGB image. That is, thinning is performed instead of the averaging in the first method.

Subsampling in the spatial direction is performed by, for example, extracting one value at the upper left among four values in units of 2×2-pixel blocks. Subsampling may be performed in units of blocks other than 2×2-pixel blocks. In addition, the value at a position other than the upper left may be extracted.

The third method is a method of performing linear computation defined based on the assumption that pixels spatially adjacent to each other have the same pixel value spectrum, thereby reconstructing a low-resolution RGB image. The third method will be described in detail below.

22 FIG. First, under the assumption that 2×2 pixels have the same pixel value spectrum, it is considered to reconstruct, as a reconstructed image, three low-resolution spectral images of three wavelength bands corresponding to a low-resolution RGB image. The relationship between the data g indicating the compressed image and the spectral images, that is, the data f indicating the reconstructed image, is expressed by g=Hf, as in expression (1). His the system matrix described above. To simplify the description, f is transformed into the form expressed in.

22 FIG. 22 FIG. is a diagram illustrating an example of the data f indicating a reconstructed image of three wavelength bands. In, f(a, b, c) indicates the pixel value of the c-th wavelength band at the pixel position (a, b).

22 FIG. That is, in expression (1), the order used is: the pixel value of the first wavelength band of the first pixel, the pixel value of the first wavelength band of the second pixel, . . . , the pixel value of the second wavelength band of the first pixel, the pixel value of the second wavelength band of the second pixel, . . . . In contrast, in the example in, the order used is: the pixel value of the first wavelength band of the first pixel, the pixel value of the second wavelength band of the first pixel, . . . , the pixel value of the first wavelength band of the second pixel, the pixel value of the second wavelength band of the second pixel, . . . .

23 FIG. 23 FIG. is a diagram illustrating an example of the matrix H corresponding to mask data of three wavelength bands. The ordering of the columns of the matrix His transformed in accordance with the transformation of the data f. In, h(a, b, c) indicates the transmittance of the c-th wavelength band at the pixel position (a, b).

24 FIG. 24 FIG. 1 1 1 1 2 1 2 1 1 2 2 1 1 1 1 is a diagram illustrating an example of the data f indicating a low-resolution reconstructed image of three wavelength bands. In the example in, for example, the pixel values f(,,), f(,,), f(,,), and f(,,) have the same value in accordance with the assumption, and thus these four pixel values can be aggregated into one pixel value f(,,). The aggregation into one pixel value is performed in units of four pixel values in a similar manner, and accordingly the number of rows of the data f is reduced to ¼.

25 FIG. 1 1 1 is a diagram illustrating an example of the matrix H corresponding to low-resolution mask data of three wavelength bands. The number of columns in the matrix His reduced as the number of rows in the data f is reduced. The number of rows in the matrix H is maintained at the number of pixels of the compressed image. For example, the pixel value f(,,) obtained through aggregation has an influence on the four pixel values included in the data g. The leftmost column in the matrix H represents the influence.

+ + In accordance with the transformation described above, the number of rows in the data f becomes smaller than the number of rows in the data g. Thus, the number of unknown numbers reduces, and the data f can be derived by linear computation. For example, the data f can be derived by linear computation of f=Hg. Here, Hrepresents the pseudo-inverse matrix of the matrix H.

In the above, the data f of three wavelength bands is derived based on the assumption that 2×2 pixels have the same pixel value spectrum. However, pixels having the same pixel value spectrum are not limited to 2×2 pixels, and the data f is not limited to data of three wavelength bands.

−1 + For example, in the case of deriving the data f of N wavelength bands based on the assumption that a×b pixels have the same pixel value spectrum, when a×b=N, the data f can be derived by linear computation of f=Hg. When a×b>N, the data f can be derived by linear computation of f=Hg.

26 FIG. 1 FIG. 26 FIG. 21 FIG. 100 is a flowchart illustrating a ninth specific example of the operation of the image processing systemillustrated in. In the example in, the method of generating a low-resolution RGB image is different from that in the example in.

26 FIG. 26 FIG. 21 FIG. 21 FIG. 205 204 204 204 c c c Specifically, in the example in, the method of generating a low-resolution RGB image varies depending on whether the save button is ON. Specifically, when the save button is OFF (No in S), a low-resolution RGB image is reconstructed by using the low-resolution mask data of three wavelength bands and is displayed (S). This process (Sin) is the same as the process that is performed regardless of whether the save button is ON in the example in(Sin).

205 121 207 c R R On the other hand, when the save button is ON (Yes in S), the processing circuitreconstructs a hyperspectral image, and then generates a low-resolution RGB image from the hyperspectral image and displays the low-resolution RGB image (S). Specifically, among the spectral images included in the hyperspectral image, spectral images of wavelength bands included in the red wavelength band Ware integrated, and thereby a low-resolution spectral image of the red wavelength band Wis obtained.

In the integration of the spectral images, the values of the spectral images may be added together or averaged, for example, in units of 2×2-pixel blocks. The integration may be performed in units of blocks other than 2×2-pixel blocks.

G G B B Similarly, among the spectral images included in the hyperspectral image, spectral images of wavelength bands included in the green wavelength band Ware integrated, and thereby a low-resolution spectral image of the green wavelength band Wis obtained. Also, among the spectral images included in the hyperspectral image, spectral images of wavelength bands included in the blue wavelength band Ware integrated, and thereby a low-resolution spectral image of the blue wavelength band Wis obtained.

R G B Subsequently, the low-resolution spectral image of the red wavelength band W, the low-resolution spectral image of the green wavelength band W, and the low-resolution spectral image of the blue wavelength band Ware combined to obtain a low-resolution RGB image.

26 FIG. 21 FIG. 26 FIG. Except for the above, the example inis the same as the example in. In the example in, a hyperspectral image is reconstructed and saved when the save button is ON. When the display button is ON and the save button is OFF, a low-resolution RGB image is reconstructed and displayed, and a hyperspectral image is not reconstructed. Thus, the processing load is reduced.

26 FIG. In the example in, when the save button is ON, a low-resolution RGB image is generated from the reconstructed hyperspectral image.

205 121 140 When the save button is ON (Yes in S), the processing circuitmay save, in addition to the hyperspectral image, the displayed low-resolution RGB image to the recording medium. This makes it possible to save the low-resolution RGB image that is effective for display, together with the hyperspectral image, in the save mode.

27 FIG. 1 FIG. 130 121 130 is a conceptual diagram illustrating a second display example of the display deviceillustrated in. For example, the processing circuitdisplays an RGB image as a first image on the display devicein the viewing mode. In the viewing mode, the RGB image that is displayed is an RGB image reconstructed from a compressed image. In the viewing mode, a hyperspectral image is not reconstructed and is not displayed.

110 For example, the RGB image is updated in accordance with a frame rate. That is, the RGB image corresponds to a video. The RGB image is checked by a user, and the position, orientation, angle of view, and so forth of the imaging deviceare adjusted so that the subject to be imaged is included in a hyperspectral image. After the adjustment, the mode is switched from the viewing mode to the save mode.

28 FIG. 1 FIG. 130 121 130 121 130 is a conceptual diagram illustrating a third display example of the display deviceillustrated in. For example, the processing circuitdisplays an RGB image and a hyperspectral image respectively serving as a first image and a second image on the display devicein the save mode. The processing circuitdisplays the hyperspectral image in the form of spectral images on the display device. In this example, the hyperspectral image includes twelve spectral images respectively corresponding to twelve wavelength bands.

In the save mode, the RGB image that is displayed may be an RGB image reconstructed from a compressed image or may be an RGB image generated from a hyperspectral image. For example, an RGB image generated by combining one or more spectral images corresponding to R, one or more spectral images corresponding to G, and one or more spectral images corresponding to B may be displayed.

After the mode has been switched from the save mode to the viewing mode, the display of the hyperspectral image last reconstructed in the save mode may be continued in the viewing mode.

29 FIG. 1 FIG. 130 121 130 is a conceptual diagram illustrating a fourth display example of the display deviceillustrated in. For example, the processing circuitdisplays an RGB image and an analysis result of a subject based on a hyperspectral image on the display devicein the save mode.

The analysis result of a subject based on a hyperspectral image may be an image showing a recognition result of the subject recognized based on the hyperspectral image. The analysis result may be an image obtained by processing the RGB image and the hyperspectral image. The analysis result may be statistical information about the hyperspectral image and may be displayed in the text format.

121 130 121 130 The processing circuitmay further display the hyperspectral image on the display devicein the save mode. That is, the processing circuitmay display the RGB image, the hyperspectral image, and the analysis result of the subject based on the hyperspectral image on the display devicein the save mode.

121 130 121 130 Alternatively, the processing circuitmay display the hyperspectral image on the display deviceinstead of the RGB image in the save mode. That is, the processing circuitmay display the hyperspectral image and the analysis result of the subject based on the hyperspectral image on the display devicein the save mode.

121 140 The processing circuitmay save, in addition to the hyperspectral image, the analysis result of the subject based on the hyperspectral image to the recording mediumin the save mode.

After the mode has been switched from the save mode to the viewing mode, the display of the analysis result obtained last in the save mode may be continued in the viewing mode.

30 FIG. 1 FIG. 100 is a flowchart illustrating a tenth specific example of the operation of the image processing systemillustrated in.

121 301 111 121 111 In this example, the processing circuitacquires a compressed image (S). Specifically, the image sensorgenerates a compressed image to acquire the compressed image, and the processing circuitacquires the compressed image from the image sensor.

121 302 Subsequently, the processing circuitperforms a recognition process on the compressed image (S). The recognition process performed here may be the recognition process described in Patent Document 3.

121 121 Specifically, for example, the processing circuitmay apply preprocessing for increasing recognition accuracy to the compressed image. The preprocessing may include region extraction, smoothing, feature extraction, edge detection, or any combination thereof. Thereafter, the processing circuitmay perform a recognition process for a subject included in the pre-processed compressed image by using a learning model. The learning model may be a deep learning model such as a convolutional neural network (CNN) or a recurrent neural network (RNN).

121 303 121 The processing circuitdetermines, based on a result of the recognition process, whether the compressed image includes a specific subject (S). That is, the processing circuitdetermines whether a specific subject is present in the compressed image.

303 121 140 304 121 130 305 When the compressed image includes a specific subject (Yes in S), the processing circuitreconstructs a hyperspectral image by using mask data of four or more wavelength bands, and saves the hyperspectral image to the recording medium(S). Subsequently, the processing circuitgenerates an RGB image from the hyperspectral image and displays the RGB image on the display device(S).

303 121 130 306 When the compressed image does not include a specific subject (No in S), the processing circuitreconstructs an RGB image by using mask data of the three wavelength bands, and displays the RGB image on the display device(S).

121 307 100 100 301 307 Subsequently, the processing circuitdiscards the displayed RGB image (S). After that, the image processing systemends the process. The image processing systemmay repeatedly perform the series of operations (Sto S).

In the above-described operation, switching between the viewing mode and the save mode is performed in accordance with whether a subject is included in the compressed image. That is, when a subject is included in the compressed image, the save mode is used. When a subject is not included in the compressed image, the viewing mode is used. Thus, the processing load is reduced. In addition, the recognition process is performed on the compressed image without the hyperspectral image being reconstructed. Accordingly, the total throughput and processing time are reduced.

100 100 100 100 The image processing systemthat performs the above-described recognition process may be applied to a foreign matter inspection in a factory. Specifically, the image processing systemmay perform a foreign matter recognition process on a compressed image by using a learning model. When a foreign matter is present in the compressed image, the image processing systemmay reconstruct a hyperspectral image from the compressed image and save the hyperspectral image. The image processing systemmay present the hyperspectral image to a user or remove the foreign matter by using the hyperspectral image.

100 100 100 100 The image processing systemthat performs the above-described recognition process may be applied to recognition of a facial skin condition. Specifically, the image processing systemmay perform a face recognition process on a compressed image by using a learning model. When a face is present in the compressed image, the image processing systemmay reconstruct a hyperspectral image from the compressed image and save the hyperspectral image. The image processing systemmay perform recognition of a facial skin condition by using the hyperspectral image.

100 100 100 The image processing systemthat performs the above-described recognition process may be applied to a recognition process in a belt conveyor. Specifically, the image processing systemmay perform a recognition process for a two-dimensional barcode installed near an object, based on a compressed image. When a two-dimensional barcode is present in the compressed image, the image processing systemmay reconstruct a hyperspectral image for processing the object from the compressed image.

31 FIG. 1 FIG. 100 is a flowchart illustrating an eleventh specific example of the operation of the image processing systemillustrated in.

121 301 30 FIG. In this example, the processing circuitacquires a compressed image (S). This process is the same as that in the example in.

121 130 306 30 FIG. Subsequently, the processing circuitreconstructs an RGB image by using mask data of three wavelength bands and displays the RGB image on the display device(S). This process is the same as that performed in the example inwhen a specific subject is not included in the compressed image.

121 302 121 303 a a 30 FIG. 31 FIG. 30 FIG. Subsequently, the processing circuitperforms a recognition process on the RGB image (S). The processing circuitdetermines, based on a result of the recognition process, whether a specific subject is included in the RGB image (S). In the example in, a compressed image is used, whereas in the example in, an RGB image is used. Except for the difference between the compressed image and the RGB image, the process is the same as that in the example in.

303 121 140 304 303 a a 30 FIG. When the RGB image includes a specific subject (Yes in S), the processing circuitreconstructs a hyperspectral image by using mask data of four or more wavelength bands and saves the hyperspectral image to the recording medium(S). This process is the same as that in the example in. When the RGB image does not include a specific subject (No in S), a hyperspectral image is not reconstructed and is not saved.

121 307 100 100 301 307 31 FIG. Subsequently, the processing circuitdiscards the displayed RGB image (S). After that, the image processing systemends the process. The image processing systemmay repeatedly perform the series of operations (Sto S). In the example in, an RGB image is used for a recognition process. Thus, color information can be used in the recognition process. Accordingly, a more complicated recognition process can be performed with high accuracy.

While the image processing system and so forth have been described in accordance with the embodiment, aspects of the image processing system and so forth are not limited to the embodiment. Modifications conceived by those skilled in the art may be applied to the embodiment, or components in the embodiment may be combined in any manner.

For example, a process performed by a specific component in the embodiment may be performed by another component instead of the specific component. The order of processes may be changed, or processes may be performed in parallel. The ordinal numbers used in the description, such as first and second, may be substituted, removed, or newly given as appropriate. These ordinal numbers do not necessarily correspond to a meaningful order and may be used to identify elements.

For example, the expression “at least one of a first element, a second element, or a third element” corresponds to the first element, the second element, the third element, or any combination thereof.

A method including steps performed by the individual components of the image processing system or the like may be executed by any system or device. That is, the method may be executed by the above-described image processing system or the like, or may be executed by another system or device.

For example, a part or the entirety of the method may be executed by a computer including a processor, a memory, an input/output circuit, and so forth. In this case, the computer may execute a program for causing the computer to execute the method, and thereby the method may be executed.

For example, the foregoing program causes the computer to execute an image processing method including: acquiring a compressed image; switching between a viewing mode and a save mode; in the viewing mode, displaying a first image on a display device and deleting the first image after displaying the first image, the first image being either the compressed image or a display-processed image that is generated based on the compressed image and represented by information of three or fewer wavelength bands; and in the save mode, generating, based on the compressed image, a second image represented by information of four or more wavelength bands, and saving the second image to a recording medium.

Furthermore, the foregoing program may be recorded on a non-transitory computer-readable recording medium such as a CD-ROM.

The individual components of the image processing system or the like may be constituted by dedicated hardware, may be constituted by general-purpose hardware that executes the foregoing program or the like, or may be constituted by a combination thereof. The general-purpose hardware may be constituted by a memory storing a program, and a general-purpose processor or the like that reads the program from the memory and executes the program. The memory herein may be a semiconductor memory, a hard disk, or the like, and the general-purpose processor may be a central processing unit (CPU) or the like.

The dedicated hardware may be constituted by a memory and a dedicated processor or the like. For example, the dedicated processor may execute the above-described method with reference to the memory.

The individual components of the image processing system or the like may be electrical circuits. These electrical circuits may constitute one electrical circuit as a whole, or may be separate electrical circuits. These electrical circuits may correspond to dedicated hardware or to general-purpose hardware that executes the foregoing program or the like.

Modifications of the embodiment of the present disclosure may be as follows.

an image sensor including first pixels; and a controller, in which the image sensor receives first light from a filter array including four or more filters having transmission spectra different from each other in a wavelength range, and thereafter outputs first pixel values of the first pixels, the image sensor receives second light from the filter array and thereafter outputs second pixel values of the first pixels, the image sensor receives the first light and thereafter receives the second light, the controller executes a first process before receiving an instruction, the controller executes a second process after receiving the instruction, in the first process, the controller generates third pixel values corresponding to a first wavelength band, fourth pixel values corresponding to a second wavelength band, and fifth pixel values corresponding to a third wavelength band, based on the first pixel values and first information, a total number of the third pixel values is identical to a total number of the first pixel values, a total number of the fourth pixel values is identical to the total number of the first pixel values, a total number of the fifth pixel values is identical to the total number of the first pixel values, in the first process, the controller generates a first image, based on the third pixel values, the fourth pixel values, and the fifth pixel values, in the first process, the controller does not generate four or more second images corresponding one-to-one to four or more wavelength bands, based on the first pixel values and second information, in the second process, the controller generates sixth pixel values corresponding to the first wavelength band, seventh pixel values corresponding to the second wavelength band, and eighth pixel values corresponding to the third wavelength band, based on the second pixel values and the first information, a total number of the sixth pixel values is identical to the total number of the first pixel values, a total number of the seventh pixel values is identical to the total number of the first pixel values, a total number of the eighth pixel values is identical to the total number of the first pixel values, in the second process, the controller generates a third image, based on the sixth pixel values, the seventh pixel values, and the eighth pixel values, in the second process, the controller generates four or more fourth images corresponding one-to-one to the four or more wavelength bands, based on the second pixel values and the second information, the wavelength range includes the first wavelength band, the second wavelength band, and the third wavelength band, the wavelength range includes the four or more wavelength bands, the first wavelength band has a first width, the second wavelength band has a second width, and the third wavelength band has a third width, and the four or more wavelength bands each have a bandwidth that is smaller than the first width, smaller than the second width, and smaller than the third width. A device including:

The device according to the foregoing modification A, further including a memory, in which the first information and the second information are stored in the memory before the image sensor receives the first light.

The device according to the foregoing modification A or B, in which the instruction indicates displaying four or more images corresponding one-to-one to the four or more wavelength bands on a display.

1 The first pixel values may be the matrix g of n×m rows and one column expressed by expression (1). In this description, the first pixel values are referred to as a matrix gof n×m rows and one column.

2 The second pixel values may be the matrix g of n×m rows and one column expressed by expression (1). In this description, the second pixel values are referred to as a matrix gof n×m rows and one column.

32 FIG. is a diagram illustrating an example of the relationship between the first pixels included in the image sensor and the first pixel values, and the relationship between the first pixels included in the image sensor and the second pixel values.

The first pixels may be a pixel p(1, 1), . . . , and a pixel p(n, m).

1 1 n, m The first pixel values may be a pixel value g(1, 1) output by the pixel p(1, 1), . . . , and a pixel value g() output by the pixel p(n, m).

2 2 n, m The second pixel values may be a pixel value g(1, 1) output by the pixel p(1, 1), . . . , and a pixel value g() output by the pixel p(n, m).

15 FIG. The wavelength range may be the wavelength range W illustrated in.

1 B G R B G R 1 The first information may be a matrix H=(HHH) of n×m rows and n×m×3 columns. H, H, and Hare each a submatrix of H.

1R R 1G G 1B B The third pixel values corresponding to the first wavelength band may be represented as n×m components included in a matrix fof n×m rows and one column corresponding to the wavelength band W, the fourth pixel values corresponding to the second wavelength band may be represented as n×m components included in a matrix fof n×m rows and one column corresponding to the wavelength band W, and the fifth pixel values corresponding to the third wavelength band may be represented as n×m components included in a matrix fof n×m rows and one column corresponding to the wavelength band W.

Expression (1) can be expressed as follows.

1R 1G 1B The first image may be a first RGB image generated based on f, f, and f.

2 1 2 w 1 2 w 2 The second information may be a matrix H=(HH. . . . H) of n×m rows and n×m×w columns. H, H, . . . , and Hare each a submatrix of H.

1 w 15 FIG. The four or more wavelength bands may be the wavelength band W, . . . , and the wavelength band Willustrated in.

1 w 2 11 1 1w w 1 1 1 T In modification A, “in the first process, the controller does not generate four or more second images corresponding one-to-one to four or more wavelength bands, based on the first pixel values and second information” means that in the first process, the controller does not generate four or more second images corresponding one-to-one to four or more wavelength bands (i.e., the wavelength band W, . . . , and the wavelength band W), based on g=(g(1, 1) . . . g(n, m))and H. That is, the controller does not calculate fcorresponding to the wavelength band W, . . . , and fcorresponding to the wavelength band W, based on expression (2) corresponding to the following expression.

This makes it possible to reduce the amount of computation of the controller.

2R R 2G G 2B B The sixth pixel values corresponding to the first wavelength band may be represented as n×m components included in a matrix fof n×m rows and one column corresponding to the wavelength band W, the seventh pixel values corresponding to the second wavelength band may be represented as n×m components included in a matrix fof n×m rows and one column corresponding to the wavelength band W, and the eighth pixel values corresponding to the third wavelength band may be represented as n×m components included in a matrix fof n×m rows and one column corresponding to the wavelength band W.

Expression (1) can be expressed as follows.

2R 2G 2B The third image may be a second RGB image generated based on f, f, and f.

1 w 2 21 1 22 2w w 2 2 2 n, m In modification A, “in the second process, the controller generates four or more fourth images corresponding one-to-one to the four or more wavelength bands, based on the second pixel values and the second information” means that in the second process, the controller generates four or more fourth images corresponding one-to-one to four or more wavelength bands (i.e., the wavelength band W, . . . , and the wavelength band W), based on g=(g(1, 1) . . . g()) T and H. That is, the controller calculates fcorresponding to the wavelength band W, f, . . . , and fcorresponding to the wavelength band W, based on expression (2) corresponding to the following expression.

21 1 21 2w w 2w The controller generates an image Icorresponding to the wavelength band Wbased on f, . . . , and an image Icorresponding to the wavelength band Wbased on f.

33 FIG. 21 2w is a diagram illustrating an example of the image I, . . . , and the image I.

21 21 21 21 21 2w 2w 2w 2w 2w The image Iincludes a pixel p(1, 1) having a pixel value f(1, 1), . . . , and a pixel p(n, m) having a pixel value f(n, m), and the Image Iincludes a pixel p(1, 1) having a pixel value f(1, 1), . . . , and a pixel p(n, m) having a pixel value f(n, m).

The present disclosure can be applied to an image processing method for reconstructing an image and can be utilized in an image processing system, an imaging system, a camera system, an analysis system, a recognition system, and the like.