Multimodal Deep Learning to Differentiate Tumor Recurrence from Treatment Effect in Human Glioblastoma

This application claims priority to and the benefit of U.S. provisional patent application No. 63/557,699, filed on Feb. 26, 2024, and titled Method for Optimizing Insulin Dosing Parameters in Computer Aided Dosing System, the disclosure of which is hereby incorporated by reference herein in its entirety.

None.

In some embodiments, artificial intelligence and machine learning techniques may be used in optional embodiments of this disclosure. Machine Learning (ML) and Artificial Intelligence (AI) systems are in widespread use in customer service, marketing, and other industries, including medicine and science. Machine learning is considered a subset of more general artificial intelligence operations, and AI endeavors may utilize numerous instances of machine learning to make decisions, predict outputs, and perform human-like intelligent operations. Machine learning protocols typically involve programming a model that instantiates an appropriate algorithm for a given computing environment and training the model on a particular data set or domain with known historical results. The results are generally known outputs of many combinations of parameter values that the algorithm accesses during training. The model uses numerous statistical and mathematical operations to learn how to make logical decisions and generate new outputs based on the historical training data. Machine learning (ML) includes, but is not limited to, a number of models such as neural networks, deep learning algorithms, support vector machines, data clustering, regression models, and Monte Carlo simulations. Other models may utilize linear regression, logistic regression, support vector machines, K-means clustering, classification models such as a binary classifier or a multi-class classifier, clustering models, anomaly detection, other supervised learning models, and even combinations of one or more machine language model types. Most of these take vectors of data as inputs.

The term “artificial intelligence,” therefore, includes any technique that enables one or more computing devices or comping systems (i.e., a machine) to mimic human intelligence. Artificial intelligence (AI) includes, but is not limited to, knowledge bases, machine learning, representation learning, and deep learning. The term “machine learning” is generally a subset of AI that enables a machine to acquire knowledge by extracting patterns from raw data.

The term “representation learning” may be used as a subset of machine learning that enables a machine to automatically discover representations needed for feature detection, prediction, or classification from raw data. Representation learning techniques include, but are not limited to, autoencoders.

The term “deep learning” may also be considered a subset of machine learning that enables a machine to automatically discover representations needed for feature detection, prediction, classification, etc. using layers of processing. Deep learning techniques include, but are not limited to, artificial neural network or multilayer perceptron (MLP).

Machine learning models include supervised, semi-supervised, and unsupervised learning models. In a supervised learning model, the model learns a function that maps an input (also known as feature or features) to an output (also known as target or target) during training with a labeled data set (or dataset). In an unsupervised learning model, the model learns a function that maps an input (also known as feature or features) to an output (also known as target or target) during training with an unlabeled data set. In a semi-supervised model, the model learns a function that maps an input (also known as feature or features) to an output (also known as target or target) during training with both labeled and unlabeled data.

Some machine learning models are designed for a specific data set or domain and are highly expert at handling the nuances within that narrow domain. It is with respect to these and other considerations that the various aspects of the present disclosure as described below are presented.

This disclosure combines algorithms deciphered by artificial intelligence and machine learning with currently known systems and models that gather data from a patient on a real time basis. Accordingly this disclosure can utilize sensors and medical equipment that improve a system's ability to diagnose and treat a patient.

Brackets with numerals therein refer to references cited in this disclosure.

In an embodiment, a computer implemented method of distinguishing tumor progression from treatment-related necrosis in digital images of a subject includes using a computer having a processor connected to computer memory storing software that, when executed, performs computer instruction steps of a machine learning architecture by collecting magnetic resonance image (MRI) data of a selected anatomical portion of the subject and collecting dynamic positron emission tomography (dPET) data for the subject with a tracer applied to the anatomical portion of the subject. The MRI data frames and dPET data frames are co-registered and the computer stores a co-registered dPET volume of frames in the computer memory. Using the co-registered dPET volume, the method calculates and saves a tracer influx constant (Ki) map for the tracer at the anatomical portion of the subject. The method continues by segmenting respective tumor data from the MRI data and the Ki maps on a frame by frame basis and applying segmented MRI data and segmented Ki maps to respective sections of a dual encoder convolutional neural network (CNN). Concatenating respective output latent feature vectors from the respective sections of the dual encoder allows for feeding concatenated feature vectors to fully connected layers of the machine learning architecture to distinguish tumor progression from treatment-related necrosis of the anatomical portion.

In an embodiment, the method further includes injecting the tracer into the subject to model glucose transport to the anatomical portion of the subject.

In an embodiment, the method includes injecting Fluorine-18 fluorodeoxyglucose (18F-FDG) as a surrogate marker for glucose metabolism.

In an embodiment, the method includes training the CNN utilizing a supervised transfer learning procedure.

In an embodiment, the method includes storing three dimensional (3D) co-registered dPET tumor volumes in the computer memory.

In another implementation, a computer implemented method includes distinguishing tumor progression from treatment-related necrosis in digital images of a subject, and the method uses a computer having a processor connected to computer memory storing software that, when executed, performs computer instruction steps of a multimodal deep learning architecture to collect magnetic resonance image (MRI) data of a selected anatomical portion of the subject and further collect dynamic positron emission tomography (dPET) data for the subject with a tracer applied to the anatomical portion of the subject. The method continues by co-registering MRI data frames with dPET data frames and storing multi-channel parametric positron emission tomography (PET) volumes as three dimensional (3D) parametric PET maps in the computer memory. Using the 3D parametric PET maps, the method can identify and store multi-modal image features in the computer memory and continue by segmenting respective tumor data from both the MRI data and the 3D parametric PET maps on a frame by frame basis. The method includes applying segmented MRI data and selected multi-modal image features from the segmented 3D parametric PET maps to respective sections of at least one convolutional neural network (CNN) and concatenating respective output latent feature vectors from the respective sections of the at least one CNN. The method allows for feeding concatenated feature vectors to fully connected layers of the multimodal architecture to distinguish tumor progression from treatment-related necrosis of the anatomical portion of the subject.

In an embodiment, prior to storing the 3D parametric PET maps, the method includes performing a step of calculating an image derived blood input function (IDIF) that identifies an amount of tracer in the blood available for the anatomical portion to use.

In an embodiment, calculating the IDIF includes segmenting internal carotid arteries (ICA) of the subject from the 3D parametric PET maps co-registered with the MRI data frames.

In an embodiment, calculating the IDIF includes correcting the IDIF with multi-parameter modeling correcting for partial volume (PV) effects and spill over (SP) contamination to store a model corrected blood input function (MCIF) in the computer memory.

In an embodiment, the method further includes feeding the MCIF and the dPET data for the subject into a graphical Patlak model that performs a voxel-wise linear regression on the data to derive a rate of tracer uptake, Ki, as a slope.

In an embodiment, the method further utilizes the MCIF to compute voxel by voxel parametric maps of tracer kinetic rate constants and tracer influx constant.

In an embodiment, the method further includes convolving an ICA segmentation to compute an average blood time-activity curve across all time frames to produce a tracer time activity curve as an initial value for the IDIF.

In an embodiment, the method further includes the anatomical portion being a brain of a subject with a tumor therein, and the method further includes collecting radiomics features from respective voxels of the 3D parametric PET maps and/or radiomics from corresponding images of MRI tumor volumes, wherein the radiomics features comprise at least one of first-order statistics, 2D and 3D shape-descriptors, or texture level features.

In an embodiment, the method further includes concatenating respective output latent feature vectors further comprises adding to a concatenated feature vector with multimodal image features from the 3D parametric PET maps, wherein the multimodal features comprise metabolic uptake rate Ki, individual rate constants K1 to K3, total blood volume, tumor time-activity curves (TAC), or standardized uptake values (SUV).

In an embodiment, the method further includes collecting magnetic resonance image (MRI) data further comprises collecting multi-channel MRI data comprising T1 image data, T1c image data, t2/FLAIR image data, perfusion imaging, and diffusor tensor imaging (DTI).

In an embodiment, the method further includes concatenating respective output latent feature vectors further comprises adding to a concatenated feature vector with the multi-channel MRI data.

In an embodiment, the method further includes collecting static PET data for the anatomical portion of the subject and wherein concatenating respective output latent feature vectors further comprises the static PET data.

In an embodiment, the method further includes retrieving demographic data regarding the subject and wherein concatenating respective output latent feature vectors further includes the demographic data.

In another implementation, a system includes a computer having a processor connected to computer memory and in communication with an MRI imaging device and a PET scanning device, wherein the computer memory stores software that implements the multimodal deep learning architecture of this disclosure.

In an embodiment, the anatomical feature of the subject is a brain having a tumor.

In some aspects, the disclosed technology relates to systems, methods, and computer-readable medium improving insulin therapy dosing. Although example embodiments of the disclosed technology are explained in detail herein, it is to be understood that other embodiments are contemplated. Accordingly, it is not intended that the disclosed technology be limited in its scope to the details of construction and arrangement of components set forth in the following description or illustrated in the drawings. The disclosed technology is capable of other embodiments and of being practiced or carried out in various ways.

It must also be noted that, as used in the specification and the appended claims, the singular forms “a,” “an” and “the” include plural referents unless the context clearly dictates otherwise. Ranges may be expressed herein as from “about” or “approximately” one particular value and/or to “about” or “approximately” another particular value. When such a range is expressed, other exemplary embodiments include from the one particular value and/or to the other particular value.

By “comprising” or “containing” or “including” is meant that at least the named compound, element, particle, or method step is present in the composition or article or method, but does not exclude the presence of other compounds, materials, particles, method steps, even if the other such compounds, material, particles, method steps have the same function as what is named.

In describing example embodiments, terminology will be resorted to for the sake of clarity. It is intended that each term contemplates its broadest meaning as understood by those skilled in the art and includes all technical equivalents that operate in a similar manner to accomplish a similar purpose. It is also to be understood that the mention of one or more steps of a method does not preclude the presence of additional method steps or intervening method steps between those steps expressly identified. Steps of a method may be performed in a different order than those described herein without departing from the scope of the disclosed technology. Similarly, it is also to be understood that the mention of one or more components in a device or system does not preclude the presence of additional components or intervening components between those components expressly identified.

As discussed herein, a “subject” (or “patient”) may be any applicable human, animal, or other organism, living or dead, or other biological or molecular structure or chemical environment, and may relate to particular components of the subject, for instance specific organs, tissues, or fluids of a subject, may be in a particular location of the subject, referred to herein as an “area of interest” or a “region of interest.”

A detailed description of aspects of the disclosed technology, in accordance with various example embodiments, will now be provided with reference to the accompanying drawings. The drawings form a part hereof and show, by way of illustration, specific embodiments and examples. In referring to the drawings, like numerals represent like elements throughout the several figures.

An aspect of an embodiment of the present disclosure provides, among other things, a system, method and computer readable medium for providing deep learning methods and multimodal deep learning architectures to distinguish tumor progression in a patient's brain from tissue necrosis caused by tumor treatments.

In an embodiment shown in, a computer implemented method of distinguishing tumor progression from treatment-related necrosis in digital images of a subject includes using a computer having a processor connected to computer memory storing software that, when executed, performs computer instruction steps of a machine learning architecture by collectingmagnetic resonance image (MRI) data of a selected anatomical portion of the subject and collectingdynamic positron emission tomography (dPET) data for the subject with a tracer applied to the anatomical portion of the subject. The MRI data frames and dPET data frames are co-registeredand the computer stores a co-registered dPET volume of frames in the computer memory. Using the co-registered dPET volume, the method calculates and saves a tracer influx constant (Ki) mapfor the tracer at the anatomical portion of the subject. The method continues by segmentingrespective tumor data from the MRI data and the Ki maps on a frame by frame basis and applying segmented MRI data and segmented Ki maps to respective sections of a dual encoder convolutional neural network (CNN). Concatenating respective output latent feature vectors from the respective sections of the dual encoderallows for feeding concatenated feature vectors to fully connected layersof the machine learning architecture to distinguish tumor progression from treatment-related necrosis of the anatomical portion.

In another implementation shown in, a computer implemented method includes distinguishing tumor progression from treatment-related necrosis in digital images of a subject, and the method uses a computer having a processor connected to computer memory storing software that, when executed, performs computer instruction steps of a multimodal deep learning architecture to collect magnetic resonance image (MRI) dataof a selected anatomical portion of the subject and further collect dynamic positron emission tomography (dPET) datafor the subject with a tracer applied to the anatomical portion of the subject. The method continues by co-registering MRI data frames with dPET data framesand storing multi-channel parametric positron emission tomography (PET) volumes as three dimensional (3D) parametric PET maps in the computer memory. Using the 3D parametric PET maps, the method can identify and store multi-modal image featuresin the computer memory and continue by segmenting respective tumor data from both the MRI data and the 3D parametric PET maps on a frame by frame basis. The method includes applying segmented MRI data and selected multi-modal image features from the segmented 3D parametric PET maps to respective sections of at least one convolutional neural network (CNN)and concatenating respective output latent feature vectorsfrom the respective sections of the at least one CNN. The method allows for feeding concatenated feature vectors to fully connected layers of the multimodal architecture to distinguish tumor progression from treatment-related necrosis of the anatomical portion of the subject.

is a high level functional block diagram of an embodiment of the present disclosure, or an aspect of an embodiment of the present disclosure.

As shown in, a processor or controllercommunicates with the glucose monitor or device, and optionally the insulin device. The glucose monitor or devicecommunicates with the subjectto monitor glucose levels of the subject. The processor or controlleris configured to perform the required calculations. Optionally, the insulin devicecommunicates with the subjectto deliver insulin to the subject. The processor or controlleris configured to perform the required calculations. The glucose monitorand the insulin devicemay be implemented as a separate device or as a single device. The processorcan be implemented locally in the glucose monitor, the insulin device, or a standalone device (or in any combination of two or more of the glucose monitor, insulin device, or a stand along device). The processoror a portion of the system can be located remotely such that the device is operated as a telemedicine device.also illustrates sensors and detectors that can be used to gather field data measurements for a subject, in real time or from samples, from the patient's blood. These kinds of sensors and detectors may be stand alone equipment or incorporated into an insulin delivery device or pump.

Referring to, in its most basic configuration, computing devicetypically includes at least one processing unitand memory. Depending on the exact configuration and type of computing device, memorycan be volatile (such as RAM), non-volatile (such as ROM, flash memory, etc.) or some combination of the two.

Additionally, devicemay also have other features and/or functionality. For example, the device could also include additional removable and/or non-removable storage including, but not limited to, magnetic or optical disks or tape, as well as writable electrical storage media. Such additional storage is the figure by removable storageand non-removable storage. Computer storage media includes volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. The memory, the removable storage and the non-removable storage are all examples of computer storage media. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology CDROM, digital versatile disks (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can accessed by the device. Any such computer storage media may be part of, or used in conjunction with, the device.

The device may also contain one or more communications connectionsthat allow the device to communicate with other devices (e.g. other computing devices). The communications connections carry information in a communication media. Communication media typically embodies computer readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism and includes any information delivery media. The term “modulated data signal” means a signal that has one or more of its characteristics set or changed in such a manner as to encode, execute, or process information in the signal. By way of example, and not limitation, communication medium includes wired media such as a wired network or direct-wired connection, and wireless media such as radio, RF, infrared and other wireless media. As discussed above, the term computer readable media as used herein includes both storage media and communication media.

In addition to a stand-alone computing machine, embodiments of the disclosure can also be implemented on a network system comprising a plurality of computing devices that are in communication with a networking means, such as a network with an infrastructure or an ad hoc network. The network connection can be wired connections or wireless connections. As a way of example,illustrates a network system in which embodiments of the disclosure can be implemented. In this example, the network system comprises computer(e.g. a network server), network connection means(e.g. wired and/or wireless connections), computer terminal, and PDA (e.g. a smart-phone)(or other handheld or portable device, such as a cell phone, laptop computer, tablet computer, GPS receiver, mp3 player, handheld video player, pocket projector, etc. or handheld devices (or non portable devices) with combinations of such features). In an embodiment, it should be appreciated that the module listed asmay be glucose monitor device. In an embodiment, it should be appreciated that the module listed asmay be a glucose monitor device, artificial pancreas, and/or an insulin device (or other interventional or diagnostic device). Any of the components shown or discussed withmay be multiple in number. The embodiments of the disclosure can be implemented in anyone of the devices of the system. For example, execution of the instructions or other desired processing can be performed on the same computing device that is anyone of,, and. Alternatively, an embodiment of the disclosure can be performed on different computing devices of the network system. For example, certain desired or required processing or execution can be performed on one of the computing devices of the network (e.g. serverand/or glucose monitor device), whereas other processing and execution of the instruction can be performed at another computing device (e.g. terminal) of the network system, or vice versa. In fact, certain processing or execution can be performed at one computing device (e.g. serverand/or insulin device, artificial pancreas, or glucose monitor device (or other interventional or diagnostic device)); and the other processing or execution of the instructions can be performed at different computing devices that may or may not be networked. For example, the certain processing can be performed at terminal, while the other processing or instructions are passed to devicewhere the instructions are executed. This scenario may be of particular value especially when the PDAdevice, for example, accesses to the network through computer terminal(or an access point in an ad hoc network). For another example, software to be protected can be executed, encoded or processed with one or more embodiments of the disclosure. The processed, encoded or executed software can then be distributed to customers. The distribution can be in a form of storage media (e.g. disk) or electronic copy.

is a block diagram that illustrates a systemincluding a computer systemand the associated Internetconnection upon which an embodiment may be implemented. Such configuration is typically used for computers (hosts) connected to the Internetand executing a server or a client (or a combination) software. A source computer such as laptop, an ultimate destination computer and relay servers, for example, as well as any computer or processor described herein, may use the computer system configuration and the Internet connection shown in. The systemmay be used as a portable electronic device such as a notebook/laptop computer, a media player (e.g., MP3 based or video player), a cellular phone, a Personal Digital Assistant (PDA), a glucose monitor device, an artificial pancreas, an insulin delivery device (or other interventional or diagnostic device), an image processing device (e.g., a digital camera or video recorder), and/or any other handheld computing devices, or a combination of any of these devices. Note that whileillustrates various components of a computer system, it is not intended to represent any particular architecture or manner of interconnecting the components; as such details are not germane to the present disclosure. It will also be appreciated that network computers, handheld computers, cell phones and other data processing systems which have fewer components or perhaps more components may also be used. The computer system ofmay, for example, be an Apple Macintosh computer or Power Book, or an IBM compatible PC. Computer systemincludes a bus, an interconnect, or other communication mechanism for communicating information, and a processor, commonly in the form of an integrated circuit, coupled with busfor processing information and for executing the computer executable instructions. Computer systemalso includes a main memory, such as a Random Access Memory (RAM) or other dynamic storage device, coupled to busfor storing information and instructions to be executed by processor.

Main memoryalso may be used for storing temporary variables or other intermediate information during execution of instructions to be executed by processor. Computer systemfurther includes a Read Only Memory (ROM)(or other non-volatile memory) or other static storage device coupled to busfor storing static information and instructions for processor. A storage device, such as a magnetic disk or optical disk, a hard disk drive for reading from and writing to a hard disk, a magnetic disk drive for reading from and writing to a magnetic disk, and/or an optical disk drive (such as DVD) for reading from and writing to a removable optical disk, is coupled to busfor storing information and instructions. The hard disk drive, magnetic disk drive, and optical disk drive may be connected to the system bus by a hard disk drive interface, a magnetic disk drive interface, and an optical disk drive interface, respectively. The drives and their associated computer-readable media provide non-volatile storage of computer readable instructions, data structures, program modules and other data for the general purpose computing devices. Typically computer systemincludes an Operating System (OS) stored in a non-volatile storage for managing the computer resources and provides the applications and programs with an access to the computer resources and interfaces. An operating system commonly processes system data and user input, and responds by allocating and managing tasks and internal system resources, such as controlling and allocating memory, prioritizing system requests, controlling input and output devices, facilitating networking and managing files. Non-limiting examples of operating systems are Microsoft Windows, Mac OS X, and Linux.

The term “processor” is meant to include any integrated circuit or other electronic device (or collection of devices) capable of performing an operation on at least one instruction including, without limitation, Reduced Instruction Set Core (RISC) processors, CISC microprocessors, Microcontroller Units (MCUs), CISC-based Central Processing Units (CPUs), and Digital Signal Processors (DSPs). The hardware of such devices may be integrated onto a single substrate (e.g., silicon “die”), or distributed among two or more substrates. Furthermore, various functional aspects of the processor may be implemented solely as software or firmware associated with the processor.

Computer systemmay be coupled via busto a display, such as a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), a flat screen monitor, a touch screen monitor or similar means for displaying text and graphical data to a user. The display may be connected via a video adapter for supporting the display. The display allows a user to view, enter, and/or edit information that is relevant to the operation of the system. An input device, including alphanumeric and other keys, is coupled to busfor communicating information and command selections to processor. Another type of user input device is cursor control, such as a mouse, a trackball, or cursor direction keys for communicating direction information and command selections to processorand for controlling cursor movement on display. This input device typically has two degrees of freedom in two axes, a first axis (e.g., x) and a second axis (e.g., y), that allows the device to specify positions in a plane.

The computer systemmay be used for implementing the methods and techniques described herein. According to one embodiment, those methods and techniques are performed by computer systemin response to processorexecuting one or more sequences of one or more instructions contained in main memory. Such instructions may be read into main memoryfrom another computer-readable medium, such as storage device. Execution of the sequences of instructions contained in main memorycauses processorto perform the process steps described herein. In alternative embodiments, hard-wired circuitry may be used in place of or in combination with software instructions to implement the arrangement. Thus, embodiments of the disclosure are not limited to any specific combination of hardware circuitry and software.