Method For Performing Image Registration

The present application is the national stage entry of International Patent Application No. PCT/EP2023/081106, filed on Nov. 8, 2023, and claims priority to Application No. EP 22315272.9, filed on Nov. 10, 2022, the disclosures of which are incorporated herein by reference.

This specification relates to methods and systems for performing image registration of a plurality of images of a subject. The methods and systems may be used to register medical images, such as images of consecutive slices of a tissue sample obtained from an animal subject. However, the methods and systems may also be used to register other types of images of a subject.

Image registration may be used to overlay two or more images of the same subject taken at different times, from different viewpoints, and/or by different sensors. In the field of medical imaging, multiple images may be generated from a tissue sample of a subject such as a human in order to gain complimentary information regarding biological or medical insights about the subject. For example, a first image may be an immunohistochemistry (IHC) image of a first slice of the tissue sample while a second image may be an IHC image of a second slice of the tissue sample. In some cases the second slice may be consecutive to the first slice (i.e. the second slice was parallel and adjacent to the first slice in the tissue sample). The first image may be a CD8 (cluster of differentiation 8) stained IHC image while the second IHC image may be a pan-cytokeratin (Pan-CK) stained IHC image, for example. CD8 IHC images and Pan-CK IHC images may be used for the classification of patients into predefined immune phenotypes.

To facilitate analysis of images, they may need to be matched pixel-to-pixel through a process of image registration. Image registration may allow for a number of image analysis techniques to subsequently be performed using the registered images including, but not limited to, virtual staining, semantic image segmentation, cancer immune phenotyping, or the creation of patient models for a simulation.

At present, a human expert such as a pathologist can analyse the first and second images and attempt to manually perform registration of those images, however this can be a very time-consuming process and requires a high level of skill. Known computer-implemented registration methods may instead be used, however they can result in poor image alignment. Images which represent the same subject but which are captured by different imaging modalities or differ on a local level may be misaligned by existing image registration methods. For example, existing image registration methods do not work well for IHC images from consecutive slices of tumour biopsy samples.

There is therefore a need to provide an improved method of performing image registration of a plurality of images of a subject which may be less time consuming and may provide improved alignment.

According to a first aspect of this disclosure, there is provided a computer-implemented method for performing image registration of a plurality of images of a subject, the method comprising: receiving a first image of a subject and a second image of the subject; performing a first image registration to register the first image and the second image; receiving parameters of a bounding shape identifying a region of interest of the registered first image or the registered second image; generating a first cropped image of the registered first image and a second cropped image of the registered second image based on the parameters of the bounding shape; and performing a second image registration to register the first cropped image and the second cropped image. Performing the first image registration may comprise: segmenting the first image to generate a first segmented image and segmenting the second image to generate a second segmented image; generating a first distance map based on the first segmented image and generating a second distance map based on the second segmented image; registering the first distance map and the second distance map; and registering the first image and the second image based on the registration of the first distance map and the second distance map.

The first segmented image and the second segmented image may be binary images.

The first image registration may comprise a rigid transformation.

Performing the second image registration may comprise performing a normalised cross correlation based on the first cropped image and the second cropped image.

Performing the second image registration may further comprise using a Fast Fourier Transform (FFT) to transform the first cropped image and second cropped image from the spatial domain to the frequency domain, wherein the normalised cross correlation may be performed using the first cropped image in the frequency domain and the second cropped image in the frequency domain.

The subject may be an organism and the plurality of images may be a plurality of images of a tissue of the organism.

The first image may be an image of a first slice of the tissue of the organism and the second image may be an image of a second slice of the tissue of the organism.

The first image registration may be performed based on tissue-level morphology of the first image and the second image.

The second image registration may be performed based on cell-level morphology of the first cropped image and the second cropped image.

At least one of the first image or the second image may be an immunohistochemistry image or a hematoxylin and eosin stain image.

At least one of: the first cropped image may have a higher image resolution than the first image; or the second cropped image may have a higher image resolution than the second image.

The method may further comprise outputting for display the registered first cropped image and second cropped image.

According to a second aspect of this disclosure, there is provided a computer-implemented method for performing image registration of a plurality of images of a subject, the method comprising: receiving a first image of a subject and a second image of the subject; and performing a first image registration to register the first image and the second image; wherein performing the first image registration may comprise: segmenting the first image to generate a first segmented image and segmenting the second image to generate a second segmented image; generating a first distance map based on the first segmented image and generating a second distance map based on the second segmented image; registering the first distance map and the second distance map; and registering the first image and the second image based on the registration of the first distance map and the second distance map.

The method may further comprise receiving parameters of a bounding shape identifying a region of interest of the registered first image or the registered second image; generating a first cropped image of the registered first image and a second cropped image of the registered second image based on the parameters of the bounding shape; and performing a second image registration to register the first cropped image and the second cropped image.

The first segmented image and the second segmented image may be binary images.

The first image registration may comprise a rigid transformation.

Performing the second image registration may comprise performing a normalised cross correlation based on the first cropped image and the second cropped image.

Performing the second image registration may further comprise using a Fast Fourier Transform (FFT) to transform the first cropped image and second cropped image from the spatial domain to the frequency domain, wherein the normalised cross correlation may be performed using the first cropped image in the frequency domain and the second cropped image in the frequency domain.

The subject may be an organism and the plurality of images may be a plurality of images of a tissue of the organism.

The first image may be an image of a first slice of the tissue of the organism and the second image may be an image of a second slice of the tissue of the organism.

The first image registration may be performed based on tissue-level morphology of the first image and the second image.

The second image registration may be performed based on cell-level morphology of the first cropped image and the second cropped image.

At least one of the first image or the second image may be an immunohistochemistry image or a hematoxylin and eosin stain image.

At least one of: the first cropped image may have a higher image resolution than the first image; or the second cropped image may have a higher image resolution than the second image.

The method may further comprise outputting for display the registered first cropped image and second cropped image.

According to a third aspect of the present disclosure, there is provided a computer program product comprising computer-readable code that, when executed by a computing system, causes the computing system to perform a method according to any preceding method.

According to a fourth aspect of the present disclosure, there is provided system comprising one or more processors and a memory, the memory storing computer readable instructions that, when executed by the one or more processors, causes the system to perform any preceding method.

As discussed, known computer-implemented registration methods may result in poor image alignment, particularly of medical images obtained from a tissue sample. For example, existing registration methods do not work well for IHC images from consecutive slices of tumour biopsy samples.

1 FIG. 1 FIG. 102 104 102 104 shows a schematic overview of an example known method for performing image registration.shows a first IHC imageand a second IHC imageprior to registration of the images by the known registration method. The first IHC imagein this example is a CD8 stained IHC image while the second IHC imageis a Pan-CK stained IHC image.

102 104 102 106 108 104 106 108 104 106 108 102 106 108 104 106 108 102 102 104 a a b b a a b b a a 1 FIG. The first IHC imageand second IHC imageare of consecutive slices of a tumour biopsy sample. The first IHC imagein this example includes at least an image representation of a first tissue structureand an image representation of a second tissue structure. Being an image of a consecutive slice of a tumour biopsy sample, the second IHC imagein this example also contains image representations of the first and second tissue structures, however it can be seen inthat the image representation of the first tissue structureand the image representation of the second tissue structurein the second IHC imageboth differ visually from the corresponding image representation of the first tissue structureand image representation of the second tissue structurein the first IHC image. For example, the image representation of the first tissue structureand the image representation of the second tissue structureas shown in the second IHC imageare in different locations relative to the image representation of the first tissue structureand the image representation of the second tissue structurein the first IHC image. Furthermore, while the tissue structures of the first IHC imageand second imagemay appear similar on a global, tissue-level scale, the similarities may be less visible on a local (e.g. cell-level) scale.

106 108 102 106 108 104 102 104 102 104 a a b b The image representation of the first tissue structureand image representation of the second tissue structurein the first IHC imagemay differ from the image representation of the first tissue structureand image representation of the second tissue structurein the second IHC imagefor a number of different reasons. For example, the use of different staining techniques (CD8 versus Pan-CK) to generate the first IHC imageand second IHC imagemay account for some of the differences. Furthermore, where the first IHC imageand the second IHC imageare obtained from different slices of the same tissue sample, each slice will be similar to the previous or next consecutive slice but not identical, due to each slice containing different cells.

2 FIG. 202 102 204 104 202 108 102 204 108 104 202 204 202 204 102 104 a b Consecutive slices may therefore have very similar morphology on the tissue level, but their morphology may be less similar on the cellular level. This is illustrated by, which shows a first zoomed portionof the first IHC imageand a second zoomed portionof the second IHC image. The first zoomed portioncontains part of the image representation of the second tissue structureobtained from the first IHC imagewhile the second zoomed portioncontains part of the image representation of the second tissue structureobtained from the second IHC image. While the first zoomed portionand second zoomed portionappear similar at a global level (e.g. have similar outlines of the tissue structure), it can be seen on closer inspection that they differ at the cellular level (indicted by the dots in the first zoomed potionand second zoomed portion). This may be due to different cell counts and cell morphology between the slice of tissue sample used for the first IHC imageand the slice of tissue used for the second IHC image.

102 104 110 110 102 104 110 102 104 1 FIG. Known computer-implemented image registration methods for registering the first IHC imageand the second IHC imagemay produce a sub-optimal registration, as illustrated by the composite IHC imageshown in. Composite IHC imagehas been generated after registering the first IHC imageand the second IHC imageusing a known image registration method. The composite IHC imagemay have been generated by overlaying the first IHC imageand second IHC imageafter they have been aligned using the known image registration method.

110 106 106 108 108 103 104 110 106 106 108 108 102 104 102 104 110 102 104 a b a b a b a b 1 FIG. The composite IHC imageshows the image representations,,,of the first tissue structure and second tissue structure discussed previously in relation to the first IHC imageand the second IHC image. However, it can be seen from the composite IHC imagethat the image representations,,,of the first tissue structure and second tissue structure are poorly aligned, and therefore that the first IHC imageand second IHC imageas a whole as a whole are poorly aligned. This poor alignment may be due to the known registration method not being able to successfully handle the differences between the first IHC imageand the second IHC imageon the cellular level. Such poor alignment can make analysis of the composite IHC imageby a medical professional difficult. Whilehas been discussed in relation to IHC images, the discussion may equally apply to other types of medical image, or indeed types of non-medical image. For example, at least one of the first IHC imageor the second IHC imagemay be a different type of IHC image, or be a hematoxylin and eosin (H&E) stained image rather than an IHC image. In other examples, one of the images may be an image obtained by magnetic resonance imaging (MRI) while the other image may be obtained from a computed tomography (CT) scan.

Aspects of the present disclosure may provide an improved method of image registration. Aspects of the present disclosure may be of particular relevance for registering a plurality of images that have greater similarity on a global scale than they do on a local scale.

3 FIG. 6 FIG. 300 300 shows a schematic overview of a computer-implemented methodfor performing image registration of a plurality of images of a subject according to aspects of the present specification. The methodmay be performed by a computing system, such as the system described in relation to.

302 304 302 304 102 104 3 FIG. 1 FIG. A first imageof a subject and a second imageof a subject are received. The first imageand the second imageshown inare identical to the first IHC imageand the second IHC imagediscussed in relation to, however in other examples different types of images may be received such as H&E stain images.

302 304 302 304 In this example, the subject is a human patient and the first imageand second imageare images of consecutive slices of a tumour biopsy sample taken from the patient. The first imageand second imagecontain corresponding features such as tissue structures that are to be analysed after the images have been registered. However, the techniques described herein are applicable in many other scenarios and in a wide range of organisms, such as plants, animals, fungus, bacteria and viruses.

302 304 302 304 302 304 302 304 A first image registration process is performed using the first imageand the second imageto register (i.e. align) the first imageand the second image. Image registration may be used to estimate a transform between the first imageand the second image. The estimated transform may be applied to the first imageor the second imageto align each image.

302 304 302 304 The first image registration is a global registration and aligns the first imageand the second imagebased on tissue-level morphology of the first imageand the second image. The first image registration may be performed using any suitable image registration method disclosed in Barbara Zitová, Jan Flusser, Image registration methods: a survey, Image and Vision Computing, Volume 21, Issue 11, 2003, Pages 977-1000, ISSN 0262-8856, https://doi.org/10.1016/S0262-8856(03)00137-9, for example.

3 FIG. 302 306 304 308 In the example of, performing the first image registration comprises segmenting the first imageto generate a first segmented imageand segmenting the second imageto generate a second segmented image. The segmentation may be performed using a suitable thresholding segmentation algorithm known in the art, although other types of segmentation algorithm known in the art may be used instead. The segmentation may result in the creation of a binary image.

An example of a suitable segmentation algorithm is Otsu's algorithm, for example as disclosed at N. Otsu, “A Threshold Selection Method from Gray-Level Histograms,” in IEEE Transactions on Systems, Man, and Cybernetics, vol. 9, no. 1, pp. 62-66, Jan. 1979, doi: 10.1109/TSMC.1979.4310076 (https://ieeexplore.ieee.org/document/4310076).

Another example of a segmentation algorithm may use deep learning based methods, for example using a UNET convolutional neural network, for example as disclosed by Ronneberger, O., Fischer, P., Brox, T. (2015). U-Net: Convolutional Networks for Biomedical Image Segmentation. In: Navab, N., Hornegger, J., Wells, W., Frangi, A. (eds) Medical Image Computing and Computer-Assisted Intervention-MICCAI 2015. MICCAI 2015.

306 310 308 312 The first segmented image(which may be a binary image) is then converted to a first distance mapand the second segmented image(which may also be a binary image) is converted to a second distance map. A distance map is also known as a distance transform. The conversion to a distance map be performed using a suitable algorithm, for example as disclosed at C. R. Maurer, Jr., R. Qi, and V. Raghavan, “A Linear Time Algorithm for Computing Exact Euclidean Distance Transforms of Binary Images in Arbitrary Dimensions”, IEEE-Transactions on Pattern Analysis and Machine Intelligence, 25(2): 265-270, 2003.

310 312 306 308 The value of each pixel in the distance map,corresponds to the distance to a closest contour point in the corresponding segmented image,. Binary images do not provide strong gradients, which is unhelpful for optimization when aligning the images during image registration. Distance map representation provides better gradient signals, and so can lead to improved optimization results when aligning the images.

302 304 310 312 310 312 302 304 The first imagecan then be registered to the second imageby performing image registration on the first distance mapand second distance mapto obtain an estimated transform between the first distance mapand second distance map. The first imagecan be registered to the second imageby applying the estimated transform.

302 304 The first image registration may comprise a rigid image registration involving a rigid transformation, that is a transformation by rotation and/or translation. Based on the assumption that the slices of tissue shown in the first imageand the second imagewill be very similar morphologically on the tissue level, a rigid transformation should be sufficient for the first image registration. However, in other examples a different transformation such as an affine or non-rigid transformation could be used.

In this example, a gradient descent algorithm is used for the first image registration, which finds the best parameters given a rigid transformation, however in other examples the first image registration may be performed using any suitable algorithm known in the art. To use the gradient descent algorithm, we define a cost function which is the value that will be optimized. In this case we use the Mean Square Error as the cost function, but other metrics such as Mutual Information or L1 difference could be used in other examples.

3 FIG. 3 FIG. 1 FIG. 314 302 304 314 302 304 314 110 shows a composite imagegenerated from the registered first imageand second image. The composite imagemay be generated by overlaying the first imageand second imageafter they have been registered using the first image registration. Comparing the composite imageshown into the composite IHC imageshown in, it can be seen that the global registration method described so far may already provide improved image registration compared to previous methods of image registration, for example previous methods of image registration that do not involve segmenting the images and converting them into distance maps before performing registration.

316 314 316 302 304 1 FIG. 2 FIG. Imageshows a portion of the composite imagegenerally corresponding to the second tissue structure previously discussed in relation toand. It can be seen from imagethat the first imageand second imageare already well-aligned.

314 316 302 304 3 FIG. It should be noted that in some examples, composite imageand/or imageare not actually generated. They are provided into illustrate the degree of alignment of the first imageand the second imageby the first image registration.

302 304 302 304 The first, global image registration may allow the first imageand second imageto be aligned to a sufficient accuracy. Nevertheless, in some examples further accuracy in the registration of the first imageand second imagemay be achieved by also performing a second, local image registration, as discussed below.

302 304 318 302 304 314 302 304 Once the first imageand second imagehave been aligned using the first registration, a region of interest (ROI)is identified in at least one of the first imageor the second image, or in some examples the composite image. The ROI may be identified by a human user such as a pathologist, but in other examples it may be identified by a computer-implemented process, for example using a machine learning algorithm trained to identify a particular ROI. The ROI may contain a feature in the first imageand/or second imagethat is of interest to a pathologist for further analysis.

320 302 304 320 302 304 320 302 304 314 6 FIG. Parameters of a bounding shapeidentifying the ROI of the first imageor the second imageare received. The parameters may comprise coordinates of the bounding shape, for example coordinates corresponding to the position of the bounding shape in the first imageor second image. The parameters may have been determined based on an input provided by the user. For example, the user may draw the bounding shapeon a displayed version of the first image, second imageor composite image, using a user input interface such as a touchscreen or mouse coupled to the computing system described in relation to.

320 322 302 324 304 320 318 302 304 320 302 320 304 320 304 320 302 The parameters (such as co-ordinates) of the bounding shapemay be used to generate a pair of cropped images, in particular a first cropped imageof the first imageand a second cropped imageof the second image. More specifically, the parameters and the transform estimated by the first image registration process are used to identify the bounding shape(and therefore ROI) in both the registered first imageand second image. For example, if the parameters relate to a bounding shapeidentified at the first image, the corresponding bounding shapemay be determined for the second imageby applying the transform to the parameters. Similarly, if the parameters relate to a bounding shapeidentified at the second image, the corresponding bounding shapemay be determined for the first imageby applying the transform to the parameters.

322 302 318 320 322 320 322 318 320 322 318 302 The first cropped imagemay be cropped from the first imagebased on the parameters so as to contain at least the ROIidentified by the bounding shape. The first cropped imagemay be generated to include a margin around the bounding shapeso that the first cropped imageis larger than, and includes, the ROIand bounding shape. For example, the first cropped imagemay have a width and a height that are greater than the maximum width and height of the ROIin the first image, in some examples by a predetermined amount.

322 324 304 318 320 324 320 324 318 320 324 318 304 In a similar manner to the first cropped image, the second cropped imagemay be cropped from the second imagebased on the parameters so as to contain at least the ROIidentified by the bounding shape. The second cropped imagemay also be generated to include a margin around the bounding shapeso that the second cropped imageis larger than, and includes, the ROIand bounding shape. For example, the second cropped imagemay also have a width and a height that are greater than the maximum width and height of the ROIin the second image, in some examples by a predetermined amount.

3 FIG. 318 320 324 shows the ROIand bounding shapelocated on the second cropped image.

322 324 322 324 302 304 322 324 A second, local image registration process is then performed to register the first cropped imageand the second cropped image. The second image registration may involve registering the first cropped imageand the second cropped imageon a smaller level than the first registration of the first imageto the second image. That is, the second image registration may involve registering the first cropped imageand the second cropped imagebased on a cell-level morphology shown in the images rather than a tissue-level morphology in the images.

322 324 Performing the second image registration may comprise performing normalised cross-correlation (NCC) to register the first cropped imageand second cropped image. As an example, an NCC method as disclosed in D. Padfield, “Masked object registration in the Fourier domain” IEEE Transactions on Image Processing (2012). DOI:10.1109/TIP.2011.2181402 may be used, which is incorporated herein by reference. NCC is advantageous in that it is very fast to compute in the Fourier domain. However, in other examples the second image registration may be performed using a different suitable registration algorithm, for example an algorithm described in relation to the first image registration.

322 324 322 324 322 324 So that NCC may be performed, a Fast Fourier Transform (FFT) may be used to transform the first cropped imageand second cropped imagefrom a spatial domain to a frequency domain, with NCC then being performed on the transformed first cropped imageand second cropped imagein the frequency (Fourier) domain. A translation transformation to align the first cropped imageand second cropped imagemay be determined based on the result of the NCC.

322 324 302 304 The translation transformation may then be used to register the first cropped imageand second cropped image. Additionally or alternatively, the translation transformation may be used to register the first imageand second image.

322 324 6 FIG. The registered first cropped imageand second cropped image, may be output for display, for example by a display coupled to the computing system described in relation to.

3 FIG. 3 FIG. 326 322 324 326 322 324 318 326 As shown in, a composite imagebased on the registered first cropped imageand second cropped imagemay be generated and output for display. The composite imagemay comprise the registered first cropped imageoverlayed with the registered second cropped image.also shows the ROIhighlighted on the composite image, however this is optional.

322 324 In some implementations, the cropped images,are resized/rescaled to a pre-determined size prior to performing the second image registration. Interpolation techniques may be used to perform the rescaling to fill in any missing pixel data.

322 302 302 322 324 304 The first cropped imagemay have a higher resolution than the first image. For example, the first imagemay be a relatively low resolution version of an image whereas the first cropped imagemay be a cropped portion of a relatively high resolution version of the same image. Similarly, the second cropped imagemay have a higher resolution than the second image. This may improve the efficiency of the overall image registration method, by performing the global registration on large, but relatively low resolution images, then performing the local registration on a relatively smaller, but higher resolution cropped images.

4 FIG. 6 FIG. 3 FIG. 400 400 400 shows a flowchart of an example methodfor performing image registration of a plurality of images of a subject according to aspects of the present specification. The methodmay be performed by a computing system, such as the system described in relation to. The methodmay share one or more aspects with the method described previously in relation to.

402 302 304 302 304 302 304 302 304 At stepa first imageof a subject and a second imageof the subject are received. As an example, the subject may be an organism such as an animal, in particular a human. The first imagemay be an image of a first slice of tissue of the organism and the second imagemay be an image of a second slice of tissue. The first slice and second slice may be consecutive or near-consecutive slices of the same tissue. The first imageand the second imagemay have been obtained using different methods of image preparation. For example, at least one of the first imageor the second imagemay be an immunohistochemistry (IHC) image or a hematoxylin and eosin (H&E) stained image.

404 302 304 302 304 At step, a first image registration process to register the first imageand the second imageis performed. The first image registration may be a global registration and may be performed based on tissue-level morphology of the first imageand the second image. The first image registration may involve a rigid transform, although in other examples an affine or non-rigid transform may be used.

The first image registration may be performed using a gradient descent algorithm, with the mean square error selected as the cost function to optimize. However, in other examples a different cost function such as mutual information or L1 difference may be used, and/or a different algorithm to gradient descent may be used, as discussed previously.

406 320 318 302 304 320 At step, parameters of a bounding shapeidentifying a region of interestof the first imageor the second imageare received. The parameters may comprise co-ordinates, and may have been input by a user, for example by drawing the bounding shapeon a representation of the first image or second image output on a display.

408 322 302 324 304 320 302 304 318 320 302 304 404 320 302 320 304 320 302 304 322 324 302 304 322 324 320 322 324 302 304 At step, a first cropped imageof the first imageand a second cropped imageof the second imageare generated based on the parameters of the bounding shape. The first imageand second imageare both cropped to contain the ROIidentified by the bounding shape. The first imageand second imagehave been registered using the first image registration process in stepand so the location of the bounding shapein the first imagewill have a correspondence to the location of the bounding shapein the second image, based on the rigid transformation estimated during the first image registration. The parameters (e.g. coordinates) of the bounding shapemay therefore be applied to both the first imageand the second imageto produce the first cropped imageand second cropped image. This step limits the areas of the first imageand second imagesubjected to the second image registration process. The first cropped imageand/or second cropped imagemay be generated to include a margin around the bounding shape. The first cropped imageand/or second cropped imagemay have a higher resolution than the corresponding first imageor second image.

410 322 324 322 324 At step, a second image registration process to register the first cropped imageand the second cropped imageis performed. The second image registration may involve registering the first cropped imageand the second cropped imagebased on a cell-level morphology rather than a tissue-level morphology.

322 324 Performing the second image registration may comprise performing normalised cross-correlation (NCC) on the first cropped imageand second cropped imageto register the images, as discussed previously, although in other examples the second image registration may be performed using a different suitable registration algorithm.

322 324 322 324 326 322 324 Once the first cropped imageand the second cropped imagehave been registered, the registered first cropped imageand second cropped imagemay be output for display, for example on a computer display. A composite imagecomprising the registered first cropped imageand second cropped image(or at least portions of the registered first cropped image and second cropped image) may be output for display.

5 FIG. 6 FIG. 5 FIG. 5 FIG. 500 500 500 shows a flowchart of another example methodfor performing image registration of a plurality of images of a subject. The methodmay be performed by a computing system, such as the system described in relation to. The methodillustrated inis similar to the method illustrated in, but with additional steps.

502 302 304 402 4 FIG. At step, a first imageof a subject and a second imageof the subject are received, in some examples in a similar manner as previously described in relation to stepof.

504 302 304 404 504 506 508 510 4 FIG. At step, a first image registration process to register the first imageand the second imageis performed. This may be in a similar manner as previously described in relation to stepof, however stepcomprises a number of additional steps,,.

506 302 306 304 308 3 FIG. At step, the first imageis segmented to generate a first segmented imageand the second imageis segmented to generate a second segmented image. This may be performed as previously described in relation to, for example.

508 310 306 312 308 3 FIG. At step, a first distance mapis generated based on the first segmented imageand a second distance mapis generated based on the second segmented image. This may be performed as previously described in relation to, for example.

510 310 312 302 304 310 312 310 312 302 304 302 304 3 FIG. At step, the first distance mapis registered to the second distance map. This may be performed using a registration algorithm, for example as previously discussed in relation to. The first imageand the second imagemay then be registered based on the registration of the first distance mapand the second distance map. For example, a transform estimated from the registration of the first distance mapand the second distance mapmay be applied to the first imageand/or the second imageto align the first imageand the second image.

512 320 318 302 304 512 406 At step, parameters of a bounding shapeidentifying a region of interestof the first imageor the second imageare received. Stepmay correspond to one or more aspects of stepas discussed previously, and which shall not be repeated for brevity.

514 322 302 324 304 320 514 408 At step, a first cropped imageof the first imageand a second cropped imageof the second imageare generated based on the parameters of the bounding shape. Stepmay correspond to one or more aspects of stepas discussed previously, and which shall not be repeated for brevity.

516 322 324 516 410 At step, a second image registration process to register the first cropped imageand the second cropped imageis performed. Stepmay correspond to one or more aspects of stepas discussed previously, and which shall not be repeated for brevity.

322 324 322 324 326 322 324 Once the first cropped imageand the second cropped imagehave been registered, the registered first cropped imageand second cropped imagemay be output for display, for example on a computer display. A composite imagecomprising the registered first cropped imageand second cropped image(or at least portions of the registered first cropped image and second cropped image) may be output for display.

6 FIG. 600 shows a schematic example of a system/apparatusfor performing any of the methods described herein. The system/apparatus shown is an example of a computing device. It will be appreciated by the skilled person that other types of computing devices/systems may alternatively be used to implement the methods described herein, such as a distributed computing system.

600 602 600 602 602 602 702 The apparatus (or system)comprises one or more processors. The one or more processors control operation of other components of the system/apparatus. The one or more processorsmay, for example, comprise a general-purpose processor. The one or more processorsmay be a single core device or a multiple core device. The one or more processorsmay comprise a Central Processing Unit (CPU) or a graphical processing unit (GPU). Alternatively, the one or more processorsmay comprise specialised processing hardware, for instance a RISC processor or programmable hardware with embedded firmware. Multiple processors may be included.

604 704 604 The system/apparatus comprises a working or volatile memory. The one or more processors may access the volatile memoryin order to process data and may control the storage of data in memory. The volatile memorymay comprise RAM of any type, for example, Static RAM (SRAM), Dynamic RAM (DRAM), or it may comprise Flash memory, such as an SD-Card.

606 606 608 602 606 The system/apparatus comprises a non-volatile memory. The non-volatile memorystores a set of operation instructionsfor controlling the operation of the processorsin the form of computer readable instructions. The non-volatile memorymay be a memory of any kind such as a Read Only Memory (ROM), a Flash memory or a magnetic drive memory.

602 608 608 600 600 602 608 606 604 608 The one or more processorsare configured to execute operating instructionsto cause the system/apparatus to perform any of the methods described herein. The operating instructionsmay comprise code (i.e. drivers) relating to the hardware components of the system/apparatus, as well as code relating to the basic operation of the system/apparatus. Generally speaking, the one or more processorsexecute one or more instructions of the operating instructions, which are stored permanently or semi-permanently in the non-volatile memory, using the volatile memoryto store temporarily data generated during execution of said operating instructions.

6 FIG. Implementations of the methods described herein may be realised as in digital electronic circuitry, integrated circuitry, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and/or combinations thereof. These may include computer program products (such as software stored on e.g. magnetic discs, optical disks, memory, Programmable Logic Devices) comprising computer readable instructions that, when executed by a computer, such as that described in relation to, cause the computer to perform one or more of the methods described herein.

Any system feature as described herein may also be provided as a method feature, and vice versa. As used herein, means plus function features may be expressed alternatively in terms of their corresponding structure. In particular, method aspects may be applied to system aspects, and vice versa.

Furthermore, any, some and/or all features in one aspect can be applied to any, some and/or all features in any other aspect, in any appropriate combination. It should also be appreciated that particular combinations of the various features described and defined in any aspects of the specification can be implemented and/or supplied and/or used independently.

Although several embodiments have been shown and described, it would be appreciated by those skilled in the art that changes may be made in these embodiments without departing from the principles of this disclosure, the scope of which is defined in the claims and their equivalents.

The terms “drug” or “medicament” are used synonymously herein and describe a pharmaceutical formulation containing one or more active pharmaceutical ingredients or pharmaceutically acceptable salts or solvates thereof, and optionally a pharmaceutically acceptable carrier. An active pharmaceutical ingredient (“API”), in the broadest terms, is a chemical structure that has a biological effect on humans or animals. In pharmacology, a drug or medicament is used in the treatment, cure, prevention, or diagnosis of disease or used to otherwise enhance physical or mental well-being. A drug or medicament may be used for a limited duration, or on a regular basis for chronic disorders.

As described below, a drug or medicament can include at least one API, or combinations thereof, in various types of formulations, for the treatment of one or more diseases. Examples of API may include small molecules having a molecular weight of 500 Da or less; polypeptides, peptides and proteins (e.g., hormones, growth factors, antibodies, antibody fragments, and enzymes); carbohydrates and polysaccharides; and nucleic acids, double or single stranded DNA (including naked and cDNA), RNA, antisense nucleic acids such as antisense DNA and RNA, small interfering RNA (siRNA), ribozymes, genes, and oligonucleotides. Nucleic acids may be incorporated into molecular delivery systems such as vectors, plasmids, or liposomes. Mixtures of one or more drugs are also contemplated.

The drug or medicament may be contained in a primary package or “drug container” adapted for use with a drug delivery device. The drug container may be, e.g., a cartridge, syringe, reservoir, or other solid or flexible vessel configured to provide a suitable chamber for storage (e.g., short-or long-term storage) of one or more drugs. For example, in some instances, the chamber may be designed to store a drug for at least one day (e.g., 1 to at least 30 days). In some instances, the chamber may be designed to store a drug for about 1 month to about 2 years. Storage may occur at room temperature (e.g., about 20° C.), or refrigerated temperatures (e.g., from about −4° C. to about 4° C.). In some instances, the drug container may be or may include a dual-chamber cartridge configured to store two or more components of the pharmaceutical formulation to-be-administered (e.g., an API and a diluent, or two different drugs) separately, one in each chamber. In such instances, the two chambers of the dual-chamber cartridge may be configured to allow mixing between the two or more components prior to and/or during dispensing into the human or animal body.

For example, the two chambers may be configured such that they are in fluid communication with each other (e.g., by way of a conduit between the two chambers) and allow mixing of the two components when desired by a user prior to dispensing. Alternatively or in addition, the two chambers may be configured to allow mixing as the components are being dispensed into the human or animal body.

The drugs or medicaments contained in the drug delivery devices as described herein can be used for the treatment and/or prophylaxis of many different types of medical disorders. Examples of disorders include, e.g., diabetes mellitus or complications associated with diabetes mellitus such as diabetic retinopathy, thromboembolism disorders such as deep vein or pulmonary thromboembolism. Further examples of disorders are acute coronary syndrome (ACS), angina, myocardial infarction, cancer, macular degeneration, inflammation, hay fever, atherosclerosis and/or rheumatoid arthritis. Examples of APIs and drugs are those as described in handbooks such as Rote Liste 2014, for example, without limitation, main groups 12 (anti-diabetic drugs) or 86 (oncology drugs), and Merck Index, 15th edition.

Examples of APIs for the treatment and/or prophylaxis of type 1 or type 2 diabetes mellitus or complications associated with type 1 or type 2 diabetes mellitus include an insulin, e.g., human insulin, or a human insulin analogue or derivative, a glucagon-like peptide (GLP-1), GLP-1 analogues or GLP-1 receptor agonists, or an analogue or derivative thereof, a dipeptidyl peptidase-4 (DPP4) inhibitor, or a pharmaceutically acceptable salt or solvate thereof, or any mixture thereof. As used herein, the terms “analogue” and “derivative” refers to a polypeptide which has a molecular structure which formally can be derived from the structure of a naturally occurring peptide, for example that of human insulin, by deleting and/or exchanging at least one amino acid residue occurring in the naturally occurring peptide and/or by adding at least one amino acid residue. The added and/or exchanged amino acid residue can either be codeable amino acid residues or other naturally occurring residues or purely synthetic amino acid residues. Insulin analogues are also referred to as “insulin receptor ligands”. In particular, the term “derivative” refers to a polypeptide which has a molecular structure which formally can be derived from the structure of a naturally occurring peptide, for example that of human insulin, in which one or more organic substituent (e.g. a fatty acid) is bound to one or more of the amino acids. Optionally, one or more amino acids occurring in the naturally occurring peptide may have been deleted and/or replaced by other amino acids, including non-codeable amino acids, or amino acids, including non-codeable, have been added to the naturally occurring peptide.

Examples of insulin analogues are Gly(A21), Arg(B31), Arg(B32) human insulin (insulin glargine); Lys(B3), Glu(B29) human insulin (insulin glulisine); Lys(B28), Pro(B29) human insulin (insulin lispro); Asp(B28) human insulin (insulin aspart); human insulin, wherein proline in position B28 is replaced by Asp, Lys, Leu, Val or Ala and wherein in position B29 Lys may be replaced by Pro; Ala(B26) human insulin; Des(B28-B30) human insulin; Des(B27) human insulin and Des(B30) human insulin. Examples of insulin derivatives are, for example, B29-N-myristoyl-des(B30) human insulin, Lys(B29) (N-tetradecanoyl)-des(B30) human insulin (insulin detemir, Levemir®); B29-N-palmitoyl-des(B30) human insulin; B29-N-myristoyl human insulin; B29-N-palmitoyl human insulin; B28-N-myristoyl LysB28ProB29 human insulin; B28-N-palmitoyl-LysB28ProB29 human insulin; B30-N-myristoyl-ThrB29LysB30 human insulin; B30-N-palmitoyl-ThrB29LysB30 human insulin; B29-N-(N-palmitoyl-gamma-glutamyl)-des(B30) human insulin, B29-N-omega-carboxypentadecanoyl-gamma-L-glutamyl-des(B30) human insulin (insulin degludec, Tresiba®); B29-N-(N-lithocholyl-gamma-glutamyl)-des(B30) human insulin; B29-N-(ω-carboxyheptadecanoyl)-des(B30) human insulin and B29-N-(ω-carboxyheptadecanoyl) human insulin.

Examples of GLP-1, GLP-1 analogues and GLP-1 receptor agonists are, for example, Lixisenatide (Lyxumia®), Exenatide (Exendin-4, Byetta®, Bydureon®, a 39 amino acid peptide which is produced by the salivary glands of the Gila monster), Liraglutide (Victoza®), Semaglutide, Taspoglutide, Albiglutide (Syncria®), Dulaglutide (Trulicity®), rExendin-4, CJC-1134-PC, PB-1023, TTP-054, Langlenatide/HM-11260C (Efpeglenatide), HM-15211, CM-3, GLP-1 Eligen, ORMD-0901, NN-9423, NN-9709, NN-9924, NN-9926, NN-9927, Nodexen, Viador-GLP-1, CVX-096, ZYOG-1, ZYD-1, GSK-2374697, DA-3091, MAR-701, MAR709, ZP-2929, ZP-3022, ZP-DI-70, TT-401 (Pegapamodtide), BHM-034. MOD-6030, CAM-2036, DA-15864, ARI-2651, ARI-2255, Tirzepatide (LY3298176), Bamadutide (SAR425899), Exenatide-XTEN and Glucagon-Xten.

An example of an oligonucleotide is, for example: mipomersen sodium (Kynamro®), a cholesterol-reducing antisense therapeutic for the treatment of familial hypercholesterolemia or RG012 for the treatment of Alport syndrome. Examples of DPP4 inhibitors are Linagliptin, Vildagliptin, Sitagliptin, Denagliptin, Saxagliptin, Berberine.

Examples of hormones include hypophysis hormones or hypothalamus hormones or regulatory active peptides and their antagonists, such as Gonadotropine (Follitropin, Lutropin, Choriongonadotropin, Menotropin), Somatropine (Somatropin), Desmopressin, Terlipressin, Gonadorelin, Triptorelin, Leuprorelin, Buserelin, Nafarelin, and Goserelin.

Examples of polysaccharides include a glucosaminoglycane, a hyaluronic acid, a heparin, a low molecular weight heparin or an ultra-low molecular weight heparin or a derivative thereof, or a sulphated polysaccharide, e.g. a poly-sulphated form of the above-mentioned polysaccharides, and/or a pharmaceutically acceptable salt thereof. An example of a pharmaceutically acceptable salt of a poly-sulphated low molecular weight heparin is enoxaparin sodium. An example of a hyaluronic acid derivative is Hylan G-F 20 (Synvisc®), a sodium hyaluronate.

The term “antibody”, as used herein, refers to an immunoglobulin molecule or an antigen-binding portion thereof. Examples of antigen-binding portions of immunoglobulin molecules include F(ab) and F(ab′)2 fragments, which retain the ability to bind antigen. The antibody can be polyclonal, monoclonal, recombinant, chimeric, de-immunized or humanized, fully human, non-human, (e.g., murine), or single chain antibody. In some embodiments, the antibody has effector function and can fix complement. In some embodiments, the antibody has reduced or no ability to bind an Fc receptor. For example, the antibody can be an isotype or subtype, an antibody fragment or mutant, which does not support binding to an Fc receptor, e.g., it has a mutagenized or deleted Fc receptor binding region. The term antibody also includes an antigen-binding molecule based on tetravalent bispecific tandem immunoglobulins (TBTI) and/or a dual variable region antibody-like binding protein having cross-over binding region orientation (CODV).

The terms “fragment” or “antibody fragment” refer to a polypeptide derived from an antibody polypeptide molecule (e.g., an antibody heavy and/or light chain polypeptide) that does not comprise a full-length antibody polypeptide, but that still comprises at least a portion of a full-length antibody polypeptide that is capable of binding to an antigen. Antibody fragments can comprise a cleaved portion of a full length antibody polypeptide, although the term is not limited to such cleaved fragments. Antibody fragments that are useful in the present specification include, for example, Fab fragments, F(ab′)2 fragments, scFv (single-chain Fv) fragments, linear antibodies, monospecific or multispecific antibody fragments such as bispecific, trispecific, tetraspecific and multispecific antibodies (e.g., diabodies, triabodies, tetrabodies), monovalent or multivalent antibody fragments such as bivalent, trivalent, tetravalent and multivalent antibodies, minibodies, chelating recombinant antibodies, tribodies or bibodies, intrabodies, nanobodies, small modular immunopharmaceuticals (SMIP), binding-domain immunoglobulin fusion proteins, camelized antibodies, and VHH containing antibodies. Additional examples of antigen-binding antibody fragments are known in the art.

The terms “Complementarity-determining region” or “CDR” refer to short polypeptide sequences within the variable region of both heavy and light chain polypeptides that are primarily responsible for mediating specific antigen recognition. The term “framework region” refers to amino acid sequences within the variable region of both heavy and light chain polypeptides that are not CDR sequences, and are primarily responsible for maintaining correct positioning of the CDR sequences to permit antigen binding. Although the framework regions themselves typically do not directly participate in antigen binding, as is known in the art, certain residues within the framework regions of certain antibodies can directly participate in antigen binding or can affect the ability of one or more amino acids in CDRs to interact with antigen. Examples of antibodies are anti PCSK-9 mAb (e.g., Alirocumab), anti IL-6 mAb (e.g., Sarilumab), and anti IL-4 mAb (e.g., Dupilumab).

Pharmaceutically acceptable salts of any API described herein are also contemplated for use in a drug or medicament in a drug delivery device. Pharmaceutically acceptable salts are for example acid addition salts and basic salts.

Those of skill in the art will understand that modifications (additions and/or removals) of various components of the APIs, formulations, apparatuses, methods, systems and embodiments described herein may be made without departing from the full scope and spirit of the present specification, which encompass such modifications and any and all equivalents thereof.

An example drug delivery device may involve a needle-based injection system as described in Table 1 of section 5.2 of ISO 11608-1:2014(E). As described in ISO 11608-1:2014(E), needle-based injection systems may be broadly distinguished into multi-dose container systems and single-dose (with partial or full evacuation) container systems. The container may be a replaceable container or an integrated non-replaceable container.

As further described in ISO 11608-1:2014(E), a multi-dose container system may involve a needle-based injection device with a replaceable container. In such a system, each container holds multiple doses, the size of which may be fixed or variable (pre-set by the user). Another multi-dose container system may involve a needle-based injection device with an integrated non-replaceable container. In such a system, each container holds multiple doses, the size of which may be fixed or variable (pre-set by the user).

As further described in ISO 11608-1:2014(E), a single-dose container system may involve a needle-based injection device with a replaceable container. In one example for such a system, each container holds a single dose, whereby the entire deliverable volume is expelled (full evacuation). In a further example, each container holds a single dose, whereby a portion of the deliverable volume is expelled (partial evacuation). As also described in ISO 11608-1:2014(E), a single-dose container system may involve a needle-based injection device with an integrated non-replaceable container. In one example for such a system, each container holds a single dose, whereby the entire deliverable volume is expelled (full evacuation). In a further example, each container holds a single dose, whereby a portion of the deliverable volume is expelled (partial evacuation).