Image processing method, electronic device, and storage medium

This application is a continuation of International Application No. PCT/CN2021/134021, filed Nov. 29, 2021, which claims priority to Chinese Patent Application No. 202110064269.6, filed Jan. 18, 2021, the entire disclosures of which are incorporated herein by reference.

The present disclosure relates to the field of image processing technologies, and more particularly, to an image processing method, an electronic device, and a storage medium.

With the continuous development of smart terminal technology, the usage of electronic devices (such as smart phones and cameras) is becoming more and more popular, where the majority of users record their studies, work and life through images captured by electronic devices; also, the users put forward higher requirements for the quality of the captured images. In order to improve the quality of the captured images, a camera with an HDR sensor function are usually used to generate multiple images through multiple exposures, and an HDR image is formed by fusing the multiple images obtained from multiple exposures.

However, due to time intervals between the multiple exposures, a problem of ghosting in image would be easily caused when the multiple images formed after exposures are fused.

In view of the above, the present disclosure proposes an image processing method, an electronic device, and a storage medium.

In a first aspect, an embodiment of the present disclosure provides an image processing method, for an electronic device. The electronic device includes an image sensor and a Pre-ISP. The method includes: acquiring, by the image sensor, an original image; applying, by the Pre-ISP, a gain to the original image to obtain a first image, and denoising, by the Pre-ISP, the first image to obtain a second image; and fusing the original image with the second image, to obtain a fused image.

In a second aspect, an embodiment of the disclosure provides an electronic device. The electronic device includes an image sensor and a Pre-Image Signal Processor (Pre-ISP). The image sensor is configured to acquire an original image, where the original image is acquired by the image sensor without using a high dynamic range function. And the Pre-ISP is configured to apply a gain to the original image, to obtain a first image; denoise the first image to obtain a second image; and fuse the original image with the second image, to obtain a fused image.

In a third aspect, an embodiment of the present disclosure provides a non-transitory computer-readable storage medium. The computer-readable storage medium stores therein program codes which, when being executed, cause an image processing method to be implemented. The method includes: acquiring an original image, where the original image is acquired without a high dynamic range function; applying a gain to the original image, to obtain a first image; denoising the first image to obtain a second image; and fusing the original image with the second image, to obtain a fused image.

Other features and aspects of the disclosed features will become apparent from the following detailed description, take in conjunction with the accompanying drawings, which illustrate, by way of example, the features in accordance with embodiments of the disclosure. The summary is not intended to limit the scope of any embodiments described herein.

The technical solutions in the embodiments of the present disclosure will be described clearly and comprehensively, in conjunction with the drawings in the embodiments of the present disclosure. It is clear that the described embodiments are only a part, but not all, of embodiments of the present disclosure. Based on the embodiments in the present disclosure, all other embodiments, obtained by those skilled in the art without creative labor, fall within the scope of protection of the present disclosure.

In recent years, with the continuous development of smart terminal technology, the technology of photographing images through smart terminals has become better and better. For example, for a high dynamic range scene, a high-quality image may be obtained by performing multiple exposures for the same scene, and then synthesizing multiple images, formed at different exposure parameters, into one image.

Embodiments of the present disclosure will be described in detail below, in conjunction with the drawings.

Referring to, an embodiment of the present disclosure provides an image processing method applicable to an electronic device. The electronic device includes an image sensor, a Pre-ISP (Pre-Image Signal Processor), an image signal processor, and a control module. The Pre-ISP includes a gain module, a denoising module, and a fusion module. The method includes operations as follows.

At block S, the image sensor acquires an original image.

It is notable that the electronic device in the embodiment may include an image sensor (sensor) supporting an HDR function. In this case, the original image is an image captured by the electronic device when the HDR function is off.

Alternatively, the electronic device may also include an image sensor not supporting the HDR function. In this case, the original image is an image captured by the electronic device. The original image may be an image captured by the image sensor of the electronic device, or an image transmitted through other devices, or an image downloaded from a cloud server through the network. In the disclosure, the source of the original image is not limited.

In an implementation, the original image is an image of single frame.

At block S, the Pre-ISP applies a gain to the original image, to obtain a first image.

In this embodiment, before inputting the original image into the image signal processor for image processing, the brightness of the original image may be increased to improve the overall brightness-and-darkness of the original image. Specifically, the gain module in the Pre-ISP applies a gain to the original image to obtain the first image, and at that time, the brightness of the first image is greater than the brightness of the original image.

In an implementation, the gain module may determine a brightness parameter of the original image, and apply, based on the brightness parameter, a gain to the original image to obtain the first image. Among them, the brightness parameter may include an automatic exposure parameter of the original image. For different brightness parameters of the original image, the corresponding gains applied to the original image may vary in terms of magnitude.

For example, in one implementation, one frame of the original image may include a plurality of brightness parameters, and the plurality of brightness parameters may form a brightness value interval (that is, a color gradation index interval). The broader the brightness value interval, the smaller the magnitude of the gain applied to the original image; and the narrower the brightness value interval, the larger the magnitude of the gain applied to the original image.

In another implementation, the magnitude of the gain for the original image may also be determined according to the value of the brightness parameter. Specifically, the smaller the value of the brightness parameter, the larger the magnitude of the gain applied to the original image, that is, the greater the degree at which the brightness of the original image is increased; and the larger the value of the brightness parameter, the smaller the magnitude of the gain applied to the original image, that is, the less the degree at which the brightness of the original image is increased.

Alternatively, the magnitude of the gain applied to the original image may be a default value, or an adjustment value adaptive to the value of the brightness parameter, where the larger the value of the brightness parameter, the smaller the corresponding adjustment value; and the smaller the value of the brightness parameter, the larger the corresponding adjustment value. For example, in an implementation, for a first frame of the original image, a gain of a default value may be applied thereto, and in this way, for other frames subsequent to the first frame, the gain may be applied at the adjustment value which is adaptive to the value of the brightness parameter. In another implementation, for all single frames of the original image, the gain may be applied according to the adjustment value which is adaptive to the value of the brightness parameter. It is notable that the application of the gain in the embodiments is an operation in the Pre-Image signal processor, which is not the same operation as the application of a gain inside the image sensor.

In the embodiment, the control module of the electronic device may calculate contrast of the original image based on the brightness parameter(s), and then determine, based on the contrast, the magnitude of the gain for the original image.

It can be understood that, for a high-definition image, the brightness of the various colors of the image is relatively uniform, that is, the contrast of the image is apparent, otherwise the image may be caused to be whitened or blackened which affecting the clarity of the image. Thus, in order to better increase the brightness of the original image during the application of the gain, the contrast of the original image may be calculated based on the obtained brightness parameter(s), for example, the contrast of the original image may be calculated based on a plurality of automatic exposure parameters included in the original image.

Alternatively, the greater the contrast, the more likely that some parts of the image are too bright or too dark. Thus, to alleviate such phenomenon, in the embodiment, the greater the contrast, the larger the magnitude of the corresponding gain applied to the original image, so that the brightness distribution of the first image obtained after the application of the gain gets more uniform. If the contrast is small, the magnitude of the gain applied to the original image may be controlled to be reduced (that is, it is reduced relative to the magnitude of the gain that is correspondingly applied to the original image when the contrast is large, and the specific reducing amount may be set according to the actual demand), so that the brightness of the first image, obtained after the application of the gain, is increased relative to the brightness of the original image, thus improving the dynamic range of the image; furthermore, it can avoid a problem of whitening or blackening of the image that would be caused by imbalance of the contrast occurring when the brightness is increased excessively.

In some other embodiments, the original brightness of the original image may be maintained if the contrast is small. That is, in this way, the magnitude of the gain for the original image may be zero, that is to say, the brightness of the first image may be equal to the brightness of the original image.

At block S, the first image is denoised to obtain a second image.

The increase of the brightness of the original image, which is achieved by increasing the gain, may cause the noise of the image to become increased. To alleviate this problem, the first image obtained after the application of the gain may be denoised by the denoising module in the Pre-ISP, to obtain the second image. Specifically, the first image may be denoised by the denoising module, based on the brightness parameter(s), to obtain the second image. The noise of the second image is less than the noise of the first image.

In the embodiment, the denoising module may determine a degree of denoising the first image, based on the magnitude of the gain applied by the gain module to the original image. The larger the magnitude of the gain applied to the original image, the greater the degree of the corresponding denoising. Specifically, the larger the magnitude of the gain applied to the original image, the greater the brightness of the obtained first image, and the greater the noise of the first image; in this case, the first image may be denoised at a relatively high degree of denoising, to make the noise of the second image effectively suppressed. The smaller the magnitude of the gain, the less the brightness of the obtained first image, and the smaller the noise of the first image; in this case, the first image may be denoised at a relatively low degree of denoising, to obtain a better image clarity, improving the quality of the second image.

At block S, the original image is fused with the second image, to obtain a fused image.

In the embodiment, the original image may be fused with the second image through the fusion module in the Pre-ISP, to obtain the fused image with increased brightness and weakened noise.

Optionally, the first image in the embodiment may include one frame or multiple frames (the specific number of the frames is not limited), and accordingly, the second image may also include one frame or multiple frames. In this case, the fusion module may fuse the second image of one frame with the original image to obtain the fused image, or may fuse the second image of multiple frames with the original image to obtain the fused image.

For example, in a specific application scenario, the original image output from the image sensor of the electronic device may be input to the Pre-ISP; after the gain module in the Pre-ISP applies a gain to the original image, one frame or two frames may be obtained; and the denoising module in the Pre-ISP may denoise the one frame or two frames. Then, the original image is fused with an image that is obtained after denoising the one frame, to obtain the fused image (which fused image in this way includes a total of two frames); alternatively, the original image is fused with an image that is obtained after denoising the two frames, to obtain the fused image (which fused image in this way includes a total of 3 frames image).

In the image processing method provided by the present disclosure, the image sensor acquires the original image; and the Pre-ISP applies a gain to the original image to obtain the first image, denoises the first image to obtain the second image, and fuses the original image with the second image to obtain the fused image. Different from the fused image which is obtained by a time-domain multi-frame technique and which has the problem of ghosting caused by the exposure time interval, this method obtains the fused image by directly fusing the original image with an image that is obtained after the original image is applied with a gain and denoised, this does not need multiple exposures to obtain the fused image, thereby avoiding the problem of ghosting in image caused by the multiple exposures. Moreover, by fusing the original image with the second image that is obtained after the original image is applied with a gain and denoised, the resulting fused image has a high dynamic range, which in turn renders the fused image to be an HDR image. In this way, it enables the electronic device to obtain an HDR image without using the HDR technique, reduces the dependence of the electronic device on image sensors supporting an HDR function, and reduces the image processing cost in the process of improving the image quality.

Referring to, another embodiment of the present disclosure provides an image processing method applicable to an electronic device. The electronic device includes an image sensor and a Pre-ISP. The Pre-ISP includes a gain module, a denoising module, and a fusion module. The method includes operations as follows.

At block S, the image sensor acquires an original image.

In the embodiment, if it is detected that the electronic device does not support a high dynamic range function, the image acquired by the image sensor may be used as the original image.

At block S, the Pre-ISP applies a gain to the original image, to obtain a first image.

At block S, the denoising module denoises the first image, to obtain a second image.

At block S, the fusion module determines, based on a brightness value of the second image and a brightness threshold, at least one to-be-fused image.

Optionally, when the second image is fused with the original image, since each of the second image and the original image includes multiple image regions with different brightness, and the overall brightness of the second image is higher than the overall brightness of the original image, in order to ensure the overall brightness-and-darkness of the fused image, the fusion module may determine at least one to-be-fused image, based on the brightness value of the second image and the brightness threshold. Among others, one to-be-fused image may be understood as an image corresponding to a part of the second image or corresponding to a part of the original image, and the to-be-fused image may come from the second image, or from the original image, or from the second image and the original image. It is worth noting that, the size of the second image is the same as the size of the original image. In the embodiment, the specific value of the brightness threshold may be set according to actual demand.

In an implementation, a part of the original image, which corresponds to a region of the second image where the brightness value is greater than a first brightness threshold, is determined as a first to-be-fused image; and a part of the second image, which corresponds to a region of the second image where the brightness value is less than a second brightness threshold, is determined as a second to-be-fused image, where the first brightness threshold is larger than the second brightness threshold. In addition, a part of the original image corresponding to a region of the second image where the brightness value is between the second brightness threshold and the first brightness threshold, and a part of the second image corresponding to the region of the second image where the brightness value is between the second brightness threshold and the first brightness threshold, are determined as a third to-be-fused image.

In the case where the brightness value of the second image is between the second brightness threshold and the first brightness threshold, a weight parameter is determined for the part of the original image corresponding to the region of the second image where the brightness value is between the second brightness threshold and the first brightness threshold, and a weight parameter is determined for the part of the second image corresponding to the region of the second image where the brightness value is between the second brightness threshold and the first brightness threshold. The closer the brightness value of the region is to the first brightness threshold, the larger the weight parameter for the original image may be, so as to avoid a phenomenon that whitening of the region is caused due to excessive brightness of the region after the image fusion. The closer the brightness value of the region is to the second brightness threshold, the larger the weight parameter for the second image may be, so as to expand the brightness range of the fused image and enhance the contrast of the image. In this way, the part of the original image and the part of the second image may be combined according to their respective weight parameters, to obtain an image as the third to-be-fused image.

For example, in a specific application scenario, the brightness threshold may be set as the first brightness threshold and the second brightness threshold. The first brightness threshold may be a bright threshold in the color gradation index, and the second brightness threshold may be a dark threshold in the color gradation index, and the value of the bright threshold is larger than the value of the dark threshold. Optionally, for a region of the second image where the brightness value is greater than the bright threshold, during the fusing processing, a part of the original image corresponding to the region is selected as the to-be-fused image, so as to avoid the phenomenon that whitening of the region is caused due to excessive brightness of the region after the image fusion. For a region of the second image where the brightness value is less than the dark threshold, during the fusing processing, a part of the second image corresponding to the region is selected as the to-be-fused image, so as to expand the brightness range of the fused image and enhance the contrast of the image. For a region of the second image where the brightness value is greater than the dark threshold and less than the bright threshold, during the fusing processing, corresponding weights may be respectively assigned to a part of the second image corresponding to the region and a part of the original image corresponding to the region, where the weights may be understood as a ratio for selecting the original image and the second image; and then, the part of the second image and the part of the original image, each of which corresponding to the region, are selected based on the weights, to serve as the to-be-fused image. Then, the to-be-fused images obtained in the above three cases are stitched together to obtain a target image.

Optionally, in the case where the brightness value of the second image is larger than the dark threshold and less than the bright threshold, in assigning the corresponding weights to the second image and the original image, the closer the brightness value of the region is to the bright threshold, the larger the weight for the original image and the less the weight for the second image; and the closer the brightness value of the region is to the dark threshold, the less the weight for the original image and the larger the weight for the second image. By targetedly selecting images of to-be-fused regions, the contrast of the fused image can be improved, which in turn improves the clarity and quality of the image.

The above application scenario is detailed below by takingas an example. It is assumed that() illustrates the original image and() illustrates the second image, and the original image includes the same number of image regions as the second image, that is, as shown in, the original image includes an image region A, an image region B, an image region C, and an image region D, and the second image includes an image region a, an image region b, an image region c, and an image region d. The image brightness of different image regions of the same image may be different, the image brightness of a same image region of different images may be different, and the image brightness of different image regions of different images may be different.

In this way, if the brightness value of the image region b of the second image is larger than the bright threshold, then an image in the image region B of the original image may be determined as the first to-be-fused image, so as to avoid overexposure of the fused image that is caused due to excessive brightness when an image in the image region b is selected for fusion. It is notable that, the position of the image region b in the second image corresponds to the position of the image region B in the original image (which may be understood as being located at the same position). If the brightness value of the image region c of the second image is less than the dark threshold, an image in the image region C of the original image may be determined as the second to-be-fused image, so that the brightness difference of the image is apparent at the time of image fusion, and thus the overall brightness-and-darkness of the fused image is ensured. If the brightness values of the image region a and the image region d of the second image are between the dark threshold and bright threshold, weights may be determined respectively for the image region A and the image region D of the original image, and weights may also be determined for the image region a and the image region d of the second image; and then, the third to-be-fused image is determined based on the respective weights of the image region A, the image region D, the image region a and the image region d.

For example, taking the image region A and the image region a as an example, if the brightness value of the image region A is close to the bright threshold, it may be determined that the weight of the image region A is 70% for fusion, and the weight of the image region a is 30%. Then, in the fusion, 70% of the image in the image region A is selected to be fused with 30% of the image in the image region a. And if the brightness value of the image region A is close to the dark threshold, it may be determined that the weight of the image region A is 15% for fusion, and the weight of image region a is 85%. Then, in the fusion, 15% of the image in the image region A is selected to be fused with 85% of the image in the image region a.

At block S, the at least one to-be-fused image is stitched, to obtain the fused image.