Systems and methods for aligning digital slide images

This application claims the benefit of U.S. Provisional Patent Application No. 63/256,575, filed Oct. 16, 2021, the disclosure of which is incorporated, in its entirety, by this reference.

The accompanying drawings illustrate a number of example embodiments and are a part of the specification. Together with the following description, these drawings demonstrate and explain various principles of the instant disclosure.

is a block diagram of an example system for aligning digital slide images.

is a block diagram of an example implementation of a system for aligning digital slide images.

is a flow diagram of an example method for aligning digital slide images.

is a plan view andis a perspective view of a sectionable fiducial marker in accordance with some embodiments described herein.

is a top view of a first tissue sample slide,is a top view of a second tissue sample slide, andis a top view of a third tissue sample slide, in accordance with some embodiments described herein.

includes a view of tissue samples and/or tissue sample slides aligned in a common orientation in accordance with some embodiments described herein.

includes a view of a graphical user interface that shows sections of a tissue sample oriented within the graphical user interface in accordance with some embodiments described herein.

Throughout the drawings, identical reference characters and descriptions indicate similar, but not necessarily identical, elements. While the example embodiments described herein are susceptible to various modifications and alternative forms, specific embodiments have been shown by way of example in the drawings and will be described in detail herein. However, the example embodiments described herein are not intended to be limited to the particular forms disclosed. Rather, the instant disclosure covers all modifications, equivalents, and alternatives falling within the scope of the appended claims.

In preparing a patient tissue sample for microscopic analysis, a pathologist may cause the tissue sample to be sectioned into multiple slices, also referred to as sections or levels. Each section or level of a tissue sample may be disposed upon a carrier medium, such as a transparent (e.g., glass) slide.

When a pathologist microscopically analyzes a tissue sample, the pathologist may find it useful to review multiple portions or sections of the sampled tissue to ensure nothing is overlooked. Some processes may require a detailed microscopic review of multiple portions of the same specimen taken from various sectioned slices or levels of the specimen. Additionally, when special stains are ordered for application to a tissue sample, all stained tissue may be reviewed in detail. Such a review may include multiple sections or levels of the tissue sample.

Furthermore, during analysis of a tissue sample, when a pathologist identifies an area of interest in one level, the pathologist may find it desirable to scrutinize corresponding areas of other levels and/or stains. Unfortunately, morphology presented by a tissue sample may vary significantly from level to level, sometimes making it difficult to visually locate corresponding points or locations across different levels. Conventional processes for identifying an area of interest at a first level, switching views to a second level, and especially identifying a location on the second level corresponding to the area of interest on the first level, may be tedious and/or prone to failure.

Additionally, to aid in and/or to facilitate some analytical processes, when using transparent (e.g., glass) slides, a pathologist may choose to overlay at least two slides and physically manipulate the slides until their corresponding sections or levels of the tissue sample are aligned (e.g., positionally and/or optically aligned) with one another. The pathologist may then mark the slides to identify areas of interest. Conventional methods of manipulation of transparent slides bearing tissue samples may be cumbersome, inconvenient, and/or prone to failure. Hence, the present application identifies and addresses a need for improved systems, methods, and apparatuses for aligning of tissue sample slide images to aid in analysis of such tissue samples.

The present disclosure generally relates to methods and systems for aligning (e.g., orienting, scaling, stretching, shrinking, resizing, keystone correction, etc.) digital slide images of a sectioned tissue sample. In some embodiments, the disclosed systems and methods may utilize at least one sectionable fiducial marker that is present in each section of the sectioned tissue sample (and therefore in each digital slide image of the respective sections of the tissue sample). Embedding sectionable fiducial markers along with the specimen (e.g., into a paraffin wax block that includes the specimen) in such a way as to ensure that the markers will consistently appear in the same position across all levels may create and/or enable common points of reference for alignment.

Embodiments of the systems and methods described herein may identify one or more sections of the sectionable fiducial marker(s) in each image to determine how to align the images to each other, including any appropriate reorienting, scaling, stretching, shrinking, etc., of the images. In some embodiments, in the aligned images, the parts of the sectioned tissue sample that originated from adjacent portions of the tissue (e.g., at each respective level) may be positioned (e.g., over each other, adjacent to each other, etc.) for simultaneous and/or overlapping viewing.

Accordingly, the methods and systems of the present disclosure may facilitate and/or improve the viewing of digital slide images of a sectioned tissue sample, such as for identification and diagnosis of abnormal tissue structures.

The following will provide, with reference to, detailed descriptions of systems for aligning digital slide images. Detailed descriptions of corresponding computer-implemented methods will also be provided in connection with.

is a block diagram of an example systemfor aligning digital slide images. As illustrated in this figure, example systemmay include one or more modulesfor performing one or more tasks. As will be explained in greater detail below, modulesmay include a receiving modulethat may be configured to receive an image of a section of a sectionable tissue sample block. The image may include image data of, embedded into the section of the sectionable tissue sample block, (1) a section of a tissue sample, and (2) a section of a sectionable fiducial marker. As further shown in, modulesmay also include a determining moduleconfigured to determine an attribute of the section of the sectionable fiducial marker and/or an attribute of the section of the tissue sample from the received image of the section of the sectionable tissue sample block. Additionally, modulesmay also include an executing moduleconfigured to execute a tissue sample management action based on the determined attribute of the section of the sectionable fiducial marker.

As further illustrated in, example systemmay also include one or more memory devices, such as memory. Memorygenerally represents any type or form of volatile or non-volatile storage device or medium capable of storing data and/or computer-readable instructions. In one example, memorymay store, load, and/or maintain one or more of modules. Examples of memoryinclude, without limitation, Random Access Memory (RAM), Read Only Memory (ROM), flash memory, Hard Disk Drives (HDDs), Solid-State Drives (SSDs), optical disk drives, caches, variations or combinations of one or more of the same, or any other suitable storage memory.

As also shown in, example systemmay also include one or more physical processors, such as physical processor. Physical processorgenerally represents any type or form of hardware-implemented processing unit capable of interpreting and/or executing computer-readable instructions. In one example, physical processormay access and/or modify one or more of modulesstored in memory. Additionally or alternatively, physical processormay execute one or more of modulesto facilitate aligning of digital slide images in accordance with the systems and methods described herein. Examples of physical processorinclude, without limitation, microprocessors, microcontrollers, central processing units (CPUs), Field-Programmable Gate Arrays (FPGAs) that implement softcore processors, Application-Specific Integrated Circuits (ASICs), portions of one or more of the same, variations or combinations of one or more of the same, or any other suitable physical processor.

As also shown in, example systemmay further include one or more data stores, such as data store, that may receive, store, and/or maintain data. Data storemay represent portions of a single data store or computing device or a plurality of data stores or computing devices. In some embodiments, data storemay be a logical container for data and may be implemented in various forms (e.g., a database, a file, a file system, a data structure, etc.). Examples of data storemay include, without limitation, files, file systems, data stores, databases, and/or database management systems such as an operational datastore (ODS), a relational database, a No SQL database, a NewSQL database, and/or any other suitable organized collection of data.

In at least one example, data storemay include (e.g., store, host, access, maintain, etc.) image data. As will be explained in greater detail below, in some examples, image datamay include and/or represent any image data that may include and/or be associated with, without limitation, one or more tissue samples, one or more tissue sample blocks, one or more sections of one or more sectionable tissue sample blocks, one or more tissue sample slides, and so forth.

As further shown in, example systemmay optionally include a machine learning model. A machine learning model may be a data analysis tool that may find patterns within and/or make predictions regarding a previously unseen dataset. In some examples, machine learning modelmay include any suitable model and/or algorithm including, without limitation, a linear regression algorithm, a logistic regression algorithm, a support-vector machine (SVM), a nearest neighbor classifier, a principal component analysis algorithm, a decision tree, a naïve Bayes classifier, a k-means clustering algorithm, and so forth.

One machine learning model that may be particularly useful for analyzing unstructured datasets such as image data, and hence may be included as part of machine learning model, may be an artificial neural network. Artificial neural networks are computing systems inspired by biological neural networks. Artificial neural networks may “learn” to perform tasks by processing example or training data, often without being pre-programmed with task-specific rules. An effectively trained artificial neural network can be a powerful tool to aid in modern computing tasks such as pattern recognition, process control, data analysis, social filtering, and so forth.

An example of training of an artificial neural network from a given example may include determining a difference (e.g., error) between a processed output of the artificial neural network (e.g., a predicted result) and a target output. A training system may then adjust internal probability-weighted associations of the artificial neural network according to a learning rule and the difference between the processed output and the target output. Successive adjustments may cause the artificial neural network to produce output (e.g., one or more predictions) that is increasingly similar to the target output.

In some embodiments, an artificial neural network may include any software and/or hardware composed of interconnected processing nodes. These processing nodes, which may be referred to as “artificial neurons,” may receive inputs and pass outputs to other artificial neurons. The output of each artificial neuron may be determined by a non-linear function combination of each input to the artificial neuron, and each connection between artificial neurons may be assigned a “weight” that determines the degree to which a particular connection contributes to the output of the destination neuron(s).

Artificial neural networks may be used in a variety of contexts, including, without limitation, computer vision (e.g., image recognition and object detection), natural language processing (e.g., translation and speech recognition), medical diagnosis and recommendation systems.

Artificial neural networks may be implemented in a variety of ways. In some embodiments, an artificial neural network may be implemented as software programs and/or any other suitable form of computer-readable instructions that are executed on one or more physical processors. In further embodiments, an artificial neural network may be implemented in physical hardware, such as a series of interconnected physical processors with each processor unit acting as an artificial neuron. Hence, although some examples described herein may explain and/or illustrate machine learning modelin the context of a software-implemented artificial neural network, machine learning modelmay, in some examples, be implemented in any suitable physical hardware.

Example systeminmay be implemented in a variety of ways. For example, all or a portion of example systemmay represent portions of an example system(“system”) in. As shown in, systemmay include a computing device. In at least one example, computing devicemay be programmed with one or more of modules.

In at least one embodiment, one or more modulesfrommay, when executed by computing device, enable computing deviceto perform one or more operations to align digital slide images. For example, as will be described in greater detail below, receiving modulemay, when executed by computing device, cause computing deviceto receive an image (e.g., image) of a section of a sectionable tissue sample block. The received image may include image data of a section of a tissue sample embedded into a tissue sample block (e.g., tissue sample image) and a section of a sectionable fiducial marker embedded into the tissue sample block (e.g., sectionable fiducial marker image).

Additionally, determining modulemay, when executed by computing device, cause computing deviceto determine an attribute of the section of the sectionable fiducial marker (e.g., attribute) from the received imageof the section of the sectionable tissue sample block. The attribute may include any suitable attribute including, without limitation, at least one wavelength of light (e.g., color) reflected by the section of the sectionable fiducial marker, a shape of the section of the sectionable fiducial marker, an orientation of the section of the sectionable fiducial marker, or a position of the section of the sectionable fiducial marker within the section of the sectionable tissue sample block.

Determining modulemay also, when executed by computing device, cause computing deviceto determine an attribute of the section of the tissue sample from the received image. The attribute may include any suitable attribute including, without limitation, morphology, phenotype, shape, size, at least one wavelength of reflected light (e.g., color), etc. of the tissue sample.

In some implementations, tissue sample imagemay include an image of a tissue array, including multiple sections of tissue from one or more patients. Determining modulemay, when executed by computing device, cause computing deviceto determine a location of each section of tissue in the tissue array relative to the section of the sectionable fiducial marker in image. This location information may facilitate the automatic identification of each section of tissue in the tissue array and its origin (e.g., the originating patient). In addition, the location information can facilitate the presentation of a single image or multiple images of tissue samples from an individual patient or from multiple patients for review by a user (e.g., a pathologist).

In some examples, as further shown in, one or more of the systems described herein may optionally include and/or perform operations involving an image of an additional section of the sectionable tissue block (e.g., additional image). For example, in some embodiments, receiving modulemay optionally receive an image of an additional section of the sectionable tissue block (e.g., additional image) that may include an image of an additional section of the tissue sample (e.g., additional tissue sample image) and an image of an additional section of the sectionable fiducial marker (e.g., additional sectionable fiducial marker image).

Furthermore, executing modulemay, when executed by computing device, cause computing deviceto execute a tissue sample management action (e.g., tissue sample management action) based on the determined attribute of the section of the sectionable fiducial marker. For example, executing modulemay determine an orientation of the sectionable fiducial marker (e.g., orientation) based on the attribute of the sectionable fiducial marker (e.g., a shape of the sectionable fiducial marker, a gradient of wavelengths of light reflected by the section of the sectionable fiducial marker, etc.). In some examples, one or more of modules(e.g., executing module) may further present one or more images to a user via a suitable graphical user interface (e.g., graphical user interface). In some examples, graphical user interfacemay be separate from computing deviceand in communication (e.g., wireless and/or wired communication) with computing device, such that the user can remotely access information (e.g., imageand/or additional image) from computing device.

Computing devicegenerally represents any type or form of computing device capable of reading and/or executing computer-executable instructions and/or hosting executables. Examples of computing deviceinclude, without limitation, application servers, storage servers, database servers, web servers, and/or any other suitable computing device configured to run certain software applications and/or provide various application, storage, and/or database services.

In at least one example, computing devicemay be a computing device programmed with one or more of modules. All or a portion of the functionality of modulesmay be performed by computing deviceand/or any other suitable computing system. As will be described in greater detail below, one or more of modulesfrommay, when executed by at least one processor of computing device, may enable computing deviceto align digital slide images (e.g., imageand additional image) in any of the ways described herein.

Many other devices or subsystems may be connected to systeminand/or systemin. Conversely, all of the components and devices illustrated inneed not be present to practice the embodiments described and/or illustrated herein. The devices and subsystems referenced above may also be interconnected in different ways from those shown in. Systemsandmay also employ any number of software, firmware, and/or hardware configurations. For example, one or more of the example embodiments disclosed herein may be encoded as a computer program (also referred to as computer software, software applications, computer-readable instructions, and/or computer control logic) on a computer-readable medium.

is a flow diagram of an example computer-implemented methodfor aligning digital slide images. The steps shown inmay be performed by any suitable computer-executable code and/or computing system, including systemin, systemin, and/or variations or combinations of one or more of the same. In one example, each of the steps shown inmay represent an algorithm whose structure includes and/or is represented by multiple sub-steps, some examples of which will be provided in greater detail below.

As illustrated in, at step, one or more of the systems described herein may receive an image of a section of a sectionable tissue sample block. The image may include image data of a section of a tissue sample and a section of a sectionable fiducial marker. For example, receiving modulemay, as part of computing device, receive, from data store, an imagethat may include a tissue sample imageand a sectionable fiducial marker image. In some examples, receiving modulemay also receive an additional image, which may be adjacent to, separate from, or overlaid on image, as explained above.

The material forming the sectionable fiducial markers described herein may be embedded into a wax block and infused with paraffin, while not significantly changing in size or shape. The material may also be capable of being cleanly cut on a microtome without dulling the blade. By way of example and not limitation, the material forming the fiducial(s) may include a polymer material, a gelatin material, and/or a tissue mimetic material. For example, a tissue mimetic material may include biological and/or synthetic materials that mimic a biological sample. In some embodiments, the tissue mimetic material may include at least one biological cell, a protein material, and/or a lipid material. Tissue mimetic materials that are suitable for forming the sectionable fiducial(s) of this disclosure are disclosed in U.S. Pat. No. 9,851,349, titled “MATRIX FOR RECEIVING A TISSUE SAMPLE AND USE THEREOF,” issued Dec. 26, 2017, the entire disclosure of which is incorporated by reference herein. An example material that may be used to form the fiducial(s) may include at least one of: protein (e.g., animal protein), one or more lipids (e.g., animal fat, vegetable oil, etc.), glycerin, water, a gelling agent (e.g., an ionically gelled gelling agent), an inorganic buffer, an antifoaming agent, and/or a paraffin wax material.

In some examples, a sectionable fiducial marker may include a material or a gap in a material such that the fiducial markers are optically identifiable from surrounding material of the section (e.g., paraffin wax). For example, a sectionable fiducial marker may have a shape (e.g., rectangle, square, triangle, star-shaped, arrow-shaped, plus-shaped, circle, parallelogram, trapezoid, etc.), orientation, and/or size that can be identified. In some examples, multiple sectionable fiducial markers may be included in each section to improve and facilitate image alignment. In some embodiments, the shape, size, and orientation of the structure surrounding the tissue sample may itself be used as a fiducial for digital image alignment purposes.

By way of example and/or illustration,is a plan view of an example of a sectionable fiducial markerthat may be embedded within a tissue sample block in accordance with some embodiments described herein.shows a perspective view of sectionable fiducial marker. The sectionable fiducial markermay be formed from a material that may be included within and/or sectioned as part of a sectionable tissue sample block.

As shown, sectionable fiducial markermay have a distinct and/or identifiable shape or aspect that may indicate a particular direction and/or orientation. Note that the shape of the sectionable fiducial markeris provided as an example only and that a sectionable fiducial marker may have any suitable form, shape, and/or aspect.

As shown in, the sectionable fiducial markermay be sectioned (i.e., transversely sectioned) into multiple fiducial marker sections(e.g., fiducial marker section(), fiducial marker section(), and fiducial marker section()). When the sectionable fiducial markeris embedded within a tissue sample block as described herein, and when the tissue sample block is sectioned as described herein, each section of the sectioned tissue sample block may include a different fiducial marker section.

In additional embodiments, a tissue block may be formed, one or more holes may be formed in the tissue block, and the one or more holes may be filled with a sectionable material, which may be optically distinguishable from a surrounding material of the tissue block after sectioning is complete. By way of example, the sectionable material may include a colored paraffin wax material, a particulate material (e.g., a pigment), etc.

Using a sectionable structure to cradle or surround the tissue may provide containment that limits the area in which tissue may be found as well as a place to include markers that would correspond to the orientation of the tissue. Hence, in some embodiments, the tissue sample may be placed within a pre-formed structure including one or more sectionable fiducial markers therein. The tissue sample and structure may be processed together to form a tissue block. Hence, the sectionable fiducial marker and the tissue sample may be embedded within a sectionable tissue sample block. Sections of the tissue block, including of the tissue sample and the structure that includes the sectionable fiducial marker, may be taken (e.g., microtome sectioned) and placed on slides.

By way of illustration,show a set of tissue sample slides(e.g., tissue sample slide(), tissue sample slide(), tissue sample slide()) that each include a different section of a sectioned tissue sample block. As shown, each slide may include a structure (e.g., a rectangular structure) at least partially surrounding a respective section of the tissue sample. The structure may be a sectionable structure that may include one or more sectionable fiducial markers. The one or more sectionable fiducial markers, either alone or in combination with each other, may have an identifiable orientation to facilitate automatic orientation of corresponding digital slide images.

In some examples, digital images may be taken (e.g., via one or more digital imaging devices) of the tissue sample slides, hence generating one or more images of one or more sections of a sectioned tissue sample block. These images may be included as part of image data, with each image including image data of a section of a tissue sample and image data of a section of a sectionable fiducial marker. Receiving modulemay receive imagethat may include the tissue sample imageand the sectionable fiducial marker image. As will be described in greater detail below, one or more of modules(e.g., determining module) may identify fiducial markers in the images of the structure (e.g., a section of a sectionable fiducial marker), and one or more of modules(e.g., executing module) may align at least some of the images with each other based on the location, size, and orientation of the respective fiducial markers.