Performing denoising on an image

The present disclosure relates to the field of image processing, and in particular to the field of denoising images.

There is an increasing use of medical imaging modalities/processes to assess the condition of a patient during a medical procedure, e.g. during a diagnosis procedure or surgical procedure.

However, medical imaging modalities are prone to noise due to the statistical nature of signal formation and data acquisition. These noise sources impair the diagnostic value of medical image data. For this reason, image denoising is not only an inherent part of many general-purpose image processing tools, but is also extremely important in the medical domain to improve the image quality and thus the diagnostic value of acquired medical image data.

One approach to denoising (medical) images is to use a machine-learning method, such as a convolutional neural network, to perform denoising of a (seemingly) noisy image. However, for many real-world medical imaging applications, there is no suitable ground truth data for training a machine-learning method, since there are no fully noise-free images available. Moreover, it has been seen that, when applying machine-learning denoising methods to an image, the output commonly produces a result with a noise level well below the ground truth noise level (e.g. the noise level of an image obtained using a high-quality capture technique). This is an example for a regression-to-mean behavior, commonly occurring when optimizing machine-learning models for image-to-image regression problems using the mean squared error (MSE) as the loss function.

There is therefore an ongoing desire to improve the accuracy of denoising.

The invention is defined by the claims.

According to examples in accordance with an aspect of the invention, there is provided a computer-implemented method of generating a partially denoised image.

The computer-implemented method comprises: providing an image to an input of a convolutional neural network; processing the image using the convolutional neural network to generate a residual noise image representing the noise content of the image; weighting the values of the residual noise image with one or more predetermined weighting factors to generate a weighted residual noise image; and combining the image and the weighted residual noise image to thereby generate a partially denoised image.

The one or more predetermined weighting factors are determined by selecting one or more weighting factors such that the values of one or more noise characteristics of the partially denoised image are responsive to the values of the same one or more noise characteristics of a portion of a reference image. The portion of the reference image may include the entire reference image, or only a part of the reference image.

The proposed mechanism helps to overcome the regression-to-mean behavior, by only partially denoising an image.

The herein described approach of weighting a residual noise image, before combining the weighted residual noise image with the original image, provides a mechanism for generating a partially denoised image with reduced complexity, processing requirements and memory storage requirements. In particular, using the proposed approach, there is no requirement for generating a fully denoised image when generating the partially denoised image. This invention instead relies upon the use of a residual noise image to modify the (originally obtained) image.

Moreover, the proposed approach makes use of the original image when generating the partially denoised image, thereby mitigating any potential shortcomings of a given original denoising algorithm, by bringing back a guaranteed natural noise texture to the finalized output image. This reduces a potentially over-smoothing or partially degrading effect, which an imperfect algorithm might otherwise have in the denoising stage.

The suggested solution (which makes use of a convolutional neural network) is relatively simple and fast, meaning it can be employed with relatively low computational cost. This is advantageous when compared to other, more elaborate machine-learning approaches, such as generative models or models using perceptual image appearance concepts. Although these approaches may be capable of reaching a desired target noise level, they come at the cost of much higher algorithmic complexity and a significantly increased runtime overhead.

The portion of the reference image effectively acts as a “target” for the noise level of the partially denoised image. This means that the partially denoised image can more closely resemble a “real-life” or higher-quality image, acting as a reference image, without loss of potentially valuable information. Thus, use of a reference image to set the predetermined weighting factor(s) can help reduce or avoid an undesirable regression-to-mean behavior, i.e. over-denoising, during performance of a denoising procedure.

In some examples, the one or more predetermined weighting factors are determined by selecting one or more weighting factors such that the values of one or more noise characteristics of the partially denoised image match the values of the same one or more noise characteristics of a portion of a reference image.

In some examples, the one or more predetermined weighting factors are determined by selecting one or more weighting factors such that the values of one or more noise characteristics of the partially denoised image are proportional to the values of the same one or more noise characteristics of a portion of a reference image. The proportion may be defined by a user input or may be a preset value.

The convolutional neural network may be configured to receive, as input, the image and provide, as output, the residual noise image. That is, the convolutional neural network may directly output the residual noise image. This embodiment improves a simplicity (i.e. reduces processing complexity) of the proposed approach.

The image may comprise a medical image. The medical image may be, for example, a computed tomography (CT) image, a magnetic resonance (MR) image, an X-ray image, an ultrasound (US) image, a positron emission tomography (PET) image, or a single-photon emission computed tomography (SPECT) image.

It is particularly advantageous to use the proposed approach for low-dose computed tomography images (i.e. the image may be a low-dose computed tomography image). This is because such images have a significant amount of noise, and are particularly susceptible to the effect of regression-to-mean behavior.

The image may be two-dimensional or three-dimensional. The image may be one in a sequence of images (e.g. which may form a so-called “4D image”), each of which may be processed according to herein described methods.

The one or more noise characteristics may include one or more of: a signal-to-noise ratio; a contrast-to-noise ratio; a mean; a standard deviation; and an estimated uncertainty map (e.g. produced when generating the residual noise image for the image).

The one or more predetermined weighting factors may be determined by: obtaining the reference image; determining one or more noise characteristics of a portion of the reference image; selecting one or more predetermined weighting factors such that the corresponding noise characteristics of the partially denoised image are responsive to the determined one or more noise characteristics of the portion of the reference image.

In some examples, the step of weighting the values of the residual noise image with one or more predetermined weighting factors comprises weighting the residual noise image with the one or more predetermined weighting factors and one or more further predetermined weighting factors to generate the weighted residual noise image. The one or more further predetermined weighting factors may be determined responsive to a user input. In this way, the weighting factor(s) may be partially tuned (e.g. interactively and continuously) by the user and partially tuned by the portion of the reference image, allowing target noise level control for the partially denoised image.

The user input may comprise a user selection of one or more potential predetermined weighting factors. Thus, there may be a selection of two or more predetermined sets of one or more predetermined weighting factors from which the user can select. This facilitates ease of selecting appropriate predetermined weighting factors for the denoising process.

In some embodiments, each value of the residual noise image represents a noise content of a set of one or more pixels of the image, wherein each value represents a different set of one or pixels; and the one or more predetermined weighting factors comprise a predetermined weighting factor for each value of the residual noise image, which is dependent upon the spatial position of the set of one or more pixels of the image represented by the value of the residual noise image.

Thus, the weighting factor(s) may depend upon spatial information, and in particular upon a spatial position of part of the image with which a value of the residual noise image is associated (e.g. describes the noise content of that part of the image). This means that the amount of denoising performed on different parts, areas or volumes of the image can differ, thereby allowing greater control over the noise characteristics of the partially denoised image.

In some examples, the one or more predetermined weighting factors are dependent upon the time or relative time at which the image was captured and/or a position of the image within a sequence of images.

Thus, the weighting factor(s) may depend upon temporal information, and in particular to a time or relative time at which the image was captured and/or a position of the image within a sequence of image. The relative time may, for example, be a time since the first image in the sequence was captured or a time since an immediately previous image in the sequence was captured. Thus, different images within a sequence of images (such as frames of a video/(cine)loop) may have different sets of one or more predetermined weighting values. This can allow each frame of a video/(cine)loop to be treated independently, to improve the partial denoising of the overall video/(cine)loop.

The method may further comprise displaying, at a user interface, at least the partially denoised image. The method may further comprise displaying other images and/or values, such as the (originally obtained) image.

In some examples, the step of combining the image and the weighted residual noise image further comprises weighting the image using a second set of one or more predetermined weighting factors, and combining the weighted image and the weighted residual noise image. The second set of one or more predetermined weighted factors may be independent of the one or more predetermined weighted factors (used to weight the residual noise image) or may be derived therefrom.

There is also proposed a computer program product comprising instructions which, when executed by a suitable computer or processing system, cause the computer to carry out any herein described method.

There is also proposed a processing system for generating a partially denoised image.

The processing system is adapted to: provide an image to an input of a convolutional neural network; process the image using the convolutional neural network to generate a residual noise image representing the noise content of the image; weight the values of the residual noise image with one or more predetermined weighting factors to generate a weighted residual noise image; and combine the image and the weighted residual noise image to thereby generate a partially denoised image. The one or more predetermined weighting factors are determined by selecting () one or more weighting factors such that the values of one or more noise characteristics of the partially denoised image are responsive to the values of the same one or more noise characteristics of a portion of a reference image

The image may comprise a medical image, such as a computed tomography image (CT), a magnetic resonance (MR) image, an X-ray image, an ultrasound (US) image, a positron emission tomography (PET) image, or a single-photon emission computed tomography (SPECT) image.

In some examples, the processing system is configured to weight the values of the residual noise image with one or more predetermined weighting factors by performing a process comprising weighting the residual noise image with the one or more predetermined weighting factors and one or more further predetermined weighting factors to generate the weighted residual noise image; and the one or more further predetermined weighting factors are determined responsive to a user input.

These and other aspects of the invention will be apparent from and elucidated with reference to the embodiment(s) described hereinafter.

The invention will be described with reference to the Figures.

It should be understood that the detailed description and specific examples, while indicating exemplary embodiments of the apparatus, systems and methods, are intended for purposes of illustration only and are not intended to limit the scope of the invention. These and other features, aspects, and advantages of the apparatus, systems and methods of the present invention will become better understood from the following description, appended claims, and accompanying drawings.

It should be understood that the Figures are merely schematic and are not drawn to scale. It should also be understood that the same reference numerals are used throughout the Figures to indicate the same or similar parts.

The invention provides a mechanism for generating a partially denoised image. A residual noise image, obtained by processing an image using a convolutional neural network, is weighted. The blending or combination of the weighted residual noise image and the (original) image generates the partially denoised image. The blending process may include a step of weighting the original image before combining the weighed residual noise image and the weighted (original) image.

Embodiments may be employed in any suitable image denoising environment, but are particularly advantageous when employed to denoise medical images.

Generally, the present disclosure provides a mechanism for generating a partially denoised image. An image and a weighted noise residuum (or residual noise image) of the image are blended together to produce a final output.

is a schematic drawing illustrating a workflow processapplied to an imageto generate a partially denoised image. The imagemay be two-dimensional or three-dimensional, and comprises a plurality of pixels. The imageis occasionally referred to in this disclosure as the “original image”.

For the sake of conciseness, the present disclosure uses the generic term “pixel” to refer to both pixels (of a 2D image) and voxels (of a 3D image). Each pixel of an image comprises one or more values, e.g. representing one or more channels of the image (e.g. each representing a particular color, such as in the RGB protocol). In some examples, each pixel has only a single value (e.g. representing light intensity alone).

The image may comprise a medical image. Suitable examples of a medical image include a computed tomography (CT) image, a magnetic resonance (MR) image, an X-ray image, an ultrasound (US) image, a positron emission tomography (PET) image, or a single-photon emission computed tomography (SPECT) image.

The image is processed, during a process, using a convolutional neural network. The convolutional neural networkprocesses the (values of the) imageto generate a residual noise imagerepresenting the noise content of the image. Processes for generating a residual noise image (sometimes called a “noise residuum”) using a convolutional neural network would be readily apparent to the skilled person.

One suitable approach for generating a residual noise image is disclosed by Zhang, K., Zuo, W., Chen, Y., Meng, D. and Zhang, L., 2017. Beyond a gaussian denoiser: Residual learning of deep cnn for image denoising. IEEE Transactions on Image Processing, 26(7), pp. 3142-3155. Other approaches would be known to the skilled person.

Each value of the residual noise image may be associated with a “pixel” of the residual noise image, and can therefore be associated with a particular x and y (and optionally z, for 3D images) position within the overall residual noise image.

In some examples, each value of the residual noise image represents a predicted noise level of a (value of a) respective, different pixel of the image. In other examples, values of the residual noise image represent the predicted noise level of (values of) different groups of pixels (e.g. representing an area or a volume) of the image. Put generally, each value of the residual noise image may represent a noise content of a set of one or more pixels of the image, wherein each value represents a different set of one or pixels.

For a multi-channel image, each value of the residual noise image may represent the predicted noise level of a different pixel or different group of pixels, of the image, across one or more channels.