Systems and methods for processing electronic images for health monitoring and forecasting

This application is a continuation of and claims the benefit of priority to U.S. application Ser. No. 17/107,121, filed on Nov. 30, 2020, which in turn claims priority to U.S. Provisional Application No. 62/951,491 filed Dec. 20, 2019, the entire disclosures of which are hereby incorporated herein by reference in their entirety.

Various embodiments of the present disclosure pertain generally to population health monitoring and forecasting by processing electronic images. More specifically, particular embodiments of the present disclosure relate to systems and methods for identifying or detecting patient health trends in a specific geographic region. The present disclosure further provides systems and methods for automatically detecting and forecasting population health trends based on one or more patient health variables.

Global population health monitoring is useful for monitoring the spread of existing diseases, the occurrences of new diseases, and determining if specific disease rates are changing. However, this information is hard to monitor and organize for numerous reasons, ranging from a lack of personnel, inadequate communication among hospitals, and inadequate data collection and analysis tools. Population health information is critical for agencies operating at the global, national, state and local levels to make informed decisions and to be aware of emerging threats to communities across these scales. Even when information is available, it may be flawed. Poor epidemiological information can lead to poor decision making and the misallocation of resources.

In one method, epidemiologists and other disease researchers might track an increase in a particular known disease in an area with information obtained by pathologists or other individuals associated with a diagnosis. This information collection process may be very slow and encounter substantial amounts of time lag due to the amount of information involved.

The foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure. The background description provided herein is for the purpose of generally presenting the context of the disclosure. Unless otherwise indicated herein, the materials described in this section are not prior art to the claims in this application and are not admitted to be prior art, or suggestions of the prior art, by inclusion in this section.

According to certain aspects of the present disclosure, systems and methods are disclosed for monitoring a population health using artificial intelligence (AI).

A method for monitoring a population health using AI includes: determining at least one relationship between at least one geographic region of a plurality of geographic regions, at least one data variable, and/or at least one health variable; estimating a current prevalence of a data variable in a geographic region of the plurality of geographic region of the plurality of geographic regions; determining a trend in a relationship between the data variable and the geographic region at a current time; determining a second trend in the relationship between the data variable and the geographic region at least one prior point in time; determining if the trend in the relationship is irregular within a predetermined threshold, generating an alert.

A system for monitoring a population health using AI includes a memory storing instructions; and at least one processor executing the instructions to perform a process including determining at least one relationship between at least one geographic region of a plurality of geographic regions, at least one data variable, and/or at least one health variable; estimating a current prevalence of a data variable, and/or at least one health variable; estimating a current prevalence of a data variable in a geographic region of the plurality of geographic region of the plurality of geographic regions; determining a trend in a relationship between the data variable and the geographic region at a current time; determining a second trend in the relationship between the data variable and the geographic region at at least one prior point in time; determining if the trend in the relationship is irregular within a predetermined threshold, generating an alert.

A non-transitory computer-readable medium storing instructions that, when executed by a processor, cause the processor to perform a method for monitoring a population health, the method including determining at least one relationship between at least one geographic region of a plurality of geographic regions, at least one data variable, and/or at least one health variable; estimating a current prevalence of a data variable, and/or at least one health variable; estimating a current prevalence of a data variable in a geographic region of the plurality of geographic region of the plurality of geographic regions; determining a trend in a relationship between the data variable and the geographic region at a current time; determining a second trend in the relationship between the data variable and the geographic region at at least one prior point in time; determining if the trend in the relationship is irregular within a predetermined threshold, generating an alert.

It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosed embodiments, as claimed.

Reference will now be made in detail to the exemplary embodiments of the present disclosure, examples of which are illustrated in the accompanying drawings. Wherever possible, the same reference numbers will be used throughout the drawings to refer to the same or like parts.

The systems, devices, and methods disclosed herein are described in detail by way of examples and with reference to the figures. The examples discussed herein are examples only and are provided to assist in the explanation of the apparatuses, devices, systems, and methods described herein. None of the features or components shown in the drawings or discussed below should be taken as mandatory for any specific implementation of any of these devices, systems, or methods unless specifically designated as mandatory.

Also, for any methods described, regardless of whether the method is described in conjunction with a flow diagram, it should be understood that unless otherwise specified or required by context, any explicit or implicit ordering of steps performed in the execution of a method does not imply that those steps must be performed in the order presented but instead may be performed in a different order or in parallel.

As used herein, the term “exemplary” is used in the sense of “example,” rather than “ideal.” Moreover, the terms “a” and “an” herein do not denote a limitation of quantity, but rather denote the presence of one or more of the referenced items.

Pathology refers to the study of diseases, as well as the causes and effects of disease. More specifically, pathology refers to performing tests and analysis that are used to diagnose diseases. For example, tissue samples may be placed onto slides to be viewed under a microscope by a pathologist (e.g., a physician that is an expert at analyzing tissue samples to determine whether any abnormalities exist). That is, pathology specimens may be cut into multiple sections, stained, and prepared as slides for a pathologist to examine and render a diagnosis.

The process of using computers to assist pathologists is known as computational pathology. Computing methods used for computational pathology may include, but are not limited to, statistical analysis, autonomous or machine learning, and AI. AI may include, but is not limited to, deep learning, neural networks, classifications, clustering, and regression algorithms. By using computational pathology, lives may be saved by helping pathologists improve their diagnostic accuracy, reliability, efficiency, and accessibility. For example, computational pathology may be used to assist with detecting slides suspicious for cancer, thereby allowing pathologists to check and confirm their initial assessments before rendering a final diagnosis.

As described above, computational pathology processes and devices of the present disclosure may provide an integrated platform allowing a fully automated process including data ingestion, processing and viewing of digital pathology images via a web-browser or other user interface, while integrating with a laboratory information system (LIS). Further, clinical information may be aggregated using cloud-based data analysis of patient data. The data may come from hospitals, clinics, field researchers, etc., and may be analyzed by machine learning, computer vision, natural language processing, and/or statistical algorithms to do real-time monitoring and forecasting of health patterns at multiple geographic specificity levels.

Population monitoring may be used to monitor the spread of existing disease, the occurrence of new disease, or to determine if specific disease rates are changing. However, this information may be hard to monitor and organize for numerous reasons, which may range from a lack of personnel, inadequate communication among hospitals, to inadequate data collection and analysis tools.

The present systems and methods address this problem by aggregating clinical information using cloud-based data analysis of patient data. This data can come from hospitals, clinics, field researchers, and others. It may then be analyzed by machine learning, computer vision, natural language processing, and statistical algorithms to perform real-time monitoring and forecasting of health patterns at multiple geographic specificity levels. By centralizing the information pertaining to population disease rates, epidemiologists may identify trends sooner, so mitigation efforts may be put in place more quickly. Secondly, the utilization of AI to scan and diagnose pathology images from pathology slides removes possibility of human error in diagnosis or treatment in this process.

The process of using computers to assist in population monitoring may help in monitoring and organizing information pertaining to population health monitoring. A primary embodiment of the present disclosure involves aggregating information from sources that may potentially be located throughout the world. Machine learning may help to organize and find relationships within the provided information. Machine learning may also predict future relationships between selected variables, to estimate future rates of a disease or other health variable within a geographic region.

The below embodiments describe various machine learning algorithm training methods and implementations. These embodiments are merely exemplary. Any training methodologies could be used to train a machine learning model and/or system for the specific purpose of detecting external contaminants in a pathology slide. Below, some exemplary terms are described.

An input health variable may comprise a disease or other health related factor (e.g., cholesterol level, vitamin D, pathogens, cancers, etc.), as well as an input diagnostic slide. A training dataset may include a set of whole slide images (WSI) and/or additional diagnostic data from a set of cases used for training the machine learning (ML) algorithm. A validation dataset may include a set of WSIs and/or additional diagnostic data from a set of cases used for validating the generalizability of the ML algorithm. A set of labels may be used for each instance in the training data that contain information that an algorithm is being trained to predict (e.g., what disease is being monitored, etc.). A convolutional neural network (CNN) may refer to an architecture that may be built that can scan over the pathology slide. One embodiment may include training this CNN, using the training labels, to make one prediction per pathology slide about whether a disease is present. A CNN+Aggregator may refer to an architecture that may be built to incorporate information from a CNN that is executed over multiple localized regions of a pathology slide. One embodiment may include training this CNN, using the training labels, to make predictions for each region in the pathology slide about the likelihood that a disease is present in a specimen or scanned region. In some embodiments, a second model may take individual predictions over tissue/specimen/image regions as inputs and predict the likelihood that the pathology slide may contain a disease. Model Uncertainty may refer to a machine learning model that may be trained to predict any parameter about, or related to, a pathology slide, e.g., detection of the presence of a disease. The level of uncertainty the machine learning model has about specific predictions could be computed using a variety of methods, e.g., identifying an ambiguous range of the probability values such as those close to the threshold, using out-of-distribution techniques (Out-of-Distribution detector for Neural Networks (ODIN), tempered mix-up, Mahalanobis distance on the embedding space), etc. This uncertainty could be used to estimate the likelihood a slide may contain a disease.

According to one embodiment, a machine learning model may be trained to predict the relationship between a plurality of selected health or data variables in a geographic region, or in a selected sub-image of the geographic region. The output prediction from this model may then be used to determine whether to continue monitoring the health of the population.

illustrates an exemplary block diagram of a system and network for determining a relationship between a health variable or data variable and a geographic region, using machine learning, according to an exemplary embodiment of the present disclosure.

Specifically,illustrates an electronic networkthat may be connected to servers at hospitals, laboratories and/or doctor's offices, etc. For example, physician servers, hospital servers, clinical trial servers, research lab servers, and/or laboratory information systems, etc., may each be connected to an electronic network, such as the Internet, through one or more computers, servers and/or handheld mobile devices. According to an exemplary embodiment of the present application, the electronic networkmay also be connected to server systems, which may include processing devices that are configured to implement a disease detection platform, which includes a geographic location analysis toolfor determining specimen property or image property information pertaining to digital pathology image(s), and using machine learning to determine whether a disease or infectious agent is present, according to an exemplary embodiment of the present disclosure. The geographic location analysis toolmay also predict future relationships.

The physician servers, hospital servers, clinical trial servers, research lab serversand/or laboratory information systemsmay create or otherwise obtain images of one or more patients' cytology specimen(s), histopathology specimen(s), slide(s) of the cytology specimen(s), digitized images of the slide(s) of the histopathology specimen(s), or any combination thereof. The physician servers, hospital servers, clinical trial servers, research lab serversand/or laboratory information systemsmay also obtain any combination of patient-specific information, such as age, medical history, cancer treatment history, family history, past biopsy or cytology information, etc. The physician servers, hospital servers, clinical trial servers, research lab serversand/or laboratory information systemsmay transmit digitized slide images and/or patient-specific information to server systemsover the electronic network. Server system(s)may include one or more storage devicesfor storing images and data received from at least one of the physician servers, hospital servers, clinical trial servers, research lab servers, and/or laboratory information systems. Server systemsmay also include processing devices for processing images and data stored in the storage devices. Server systemsmay further include one or more machine learning tool(s) or capabilities. For example, the processing devices may include a machine learning tool for a disease detection platform, according to one embodiment. Alternatively or in addition, the present disclosure (or portions of the system and methods of the present disclosure) may be performed on a local processing device (e.g., a laptop).

The physician servers, hospital servers, clinical trial servers, research lab serversand/or laboratory information systemsrefer to systems used by pathologists for reviewing the images of the slides. In hospital settings, tissue type information may be stored in a laboratory information system.

illustrates an exemplary block diagram of a disease detection platformfor determining specimen property or image property information pertaining to digital pathology image(s), using machine learning. The disease detection platformmay include a geographic location analysis tool, a data ingestion tool, a slide intake tool, a slide scanner, a slide manager, a storage, a laboratory information systemand a viewing application tool.

The geographic location analysis tool, as described below, refers to a process and system for determining data variable property or health variable property information pertaining to digital pathology image(s). Machine learning may be used to classify an image, according to an exemplary embodiment. The geographic location analysis toolmay also predict future relationships, as described in the embodiments below.

The data ingestion toolmay facilitate a transfer of the digital pathology images to the various tools, modules, components, and devices that are used for classifying and processing the digital pathology images, according to an exemplary embodiment.

The slide intake toolmay scan pathology images and convert them into a digital form, according to an exemplary embodiment. The slides may be scanned with slide scanner, and the slide managermay process the images on the slides into digitized pathology images and store the digitized images in storage.

The viewing application toolmay provide a user with a specimen property or image property information pertaining to digital pathology image(s), according to an exemplary embodiment. The information may be provided through various output interfaces (e.g., a screen, a monitor, a storage device and/or a web browser, etc.).

The geographic location analysis tool, and one or more of its components, may transmit and/or receive digitized slide images and/or patient information to server systems, physician servers, hospital servers, clinical trial servers, research lab servers, and/or laboratory information systemsover a network. Further, server systemsmay include storage devices for storing images and data received from at least one of the geographic location analysis tool, the data ingestion tool, the slide intake tool, the slide scanner, the slide manager, and viewing application tool. Server systemsmay also include processing devices for processing images and data stored in the storage devices. Server systemsmay further include one or more machine learning tool(s) or capabilities, e.g., due to the processing devices. Alternatively, or in addition, the present disclosure (or portions of the system and methods of the present disclosure) may be performed on a local processing device (e.g., a laptop).

Any of the above devices, tools and modules may be located on a device that may be connected to an electronic network such as the Internet or a cloud service provider, through one or more computers, servers and/or handheld mobile devices.

illustrates an exemplary block diagram of a geographic location analysis tool, according to an exemplary embodiment of the present disclosure. The geographic location analysis toolmay include a training data platformand/or a target data platform.

According to one embodiment, the training data platformmay include a training data intake module, a data analysis module, and a relationship identification module.

The training data platform, according to one embodiment, may create or receive training images that are used to train a machine learning model to effectively analyze and classify digital pathology images. For example, the training images may be received from any one or any combination of the server systems, physician servers, hospital servers, clinical trial servers, research lab servers, and/or laboratory information systems. Images used for training may come from real sources (e.g., humans, animals, etc.) or may come from synthetic sources (e.g., graphics rendering engines, 3D models, etc.). Examples of digital pathology images may include (a) digitized slides stained with a variety of stains, such as (but not limited to) H&E, Hematoxylin alone, IHC, molecular pathology, etc.; and/or (b) digitized tissue samples from a 3D imaging device, such as microCT.

The training data intake modulemay create or receive a dataset comprising one or more training datasets corresponding to one or more health variables and/or one or more data variables. For example, the training datasets may be received from any one or any combination of the server systems, physician servers, hospital servers, clinical trial servers, research lab servers, and/or laboratory information systems. This dataset may be kept on a digital storage device. The data analysis modulemay identify quality control (QC) issues (e.g., imperfections) for the training datasets at a global or local level that may greatly affect the usability of a dataset. For example, the quality score determiner module may use information about an entire dataset, e.g., the dataset type, the overall quality of the cut of the specimen, the overall quality of the dataset itself, or pathology slide characteristics, and determine an overall quality score for the dataset. The relationship identification modulemay analyze health variables and/or data variables and determine whether a determined relationship has an irregular trend. It is useful to identify whether a relationship has an irregular trend, as trends may be used for future relationship predictions, and may trigger an alert to a user.

According to one embodiment, the target data platformmay include a target data intake module, a relationship analysis module, and an output interface. The target data platformmay receive a target image and apply the machine learning model to the received target image to determine a characteristic of a target data set. For example, the target data may be received from any one or any combination of the server systems, physician servers, hospital servers, clinical trial servers, research lab servers, and/or laboratory information systems. The target data intake modulemay receive a target dataset corresponding to a target health variable or a data variable. The relationship analysis modulemay apply the machine learning model to the target dataset to determine a characteristic of the target health variable or a data variable. For example, the relationship analysis modulemay detect a trend of the target relationship. The relationship analysis modulemay also apply the machine learning model to the target dataset to determine a quality score for the target dataset. Further, the relationship analysis modulemay apply the machine learning model to the target dataset to determine whether the target health variable or a data variable is present in a determined relationship.

The output interfacemay be used to output information about the target data and the determined relationship. (e.g., to a screen, monitor, storage device, web browser, etc.).

is a flowchart illustrating an exemplary method for monitoring population health, using machine learning, according to an exemplary embodiment of the present disclosure. For example, an exemplary method(e.g., steps-) may be performed by geographic location analysis toolautomatically or in response to a request from a user.

According to one embodiment, the exemplary methodfor monitoring a population health may include one or more of the following steps. In step, the method may include determining at least one relationship between at least one geographic region of a plurality of geographic regions, at least one data variable, and/or at least one health variable. For example, a health variable may comprise a disease or other health-related factor (e.g., cholesterol level, vitamin D level, pathogens, cancers, etc.), and a data variable may comprise an attribute of an individual (e.g., age, race, ethnicity, gender, income level, BMI, etc.). The health variable, data variable, and/or geographic region may be received from any one or any combination of the server systems, physician servers, hospital servers, clinical trial servers, research lab servers, and/or laboratory information systems.

In step, the method may include estimating the current prevalence of a data variable in a geographic region of the plurality of geographic regions. The geographic region may comprise a specific city, a county, a state, nationally, globally, etc.

In step, the method may include determining a trend in a relationship between the data variable and the geographic region at a current time. The trend may be determined via many means, e.g., a correlation machine learning method, geographic visual overlay, etc. For clarity, the trend may be determined between a geographic region(s) and a data variable(s), a geographic region(s) and health variables, or data variables and health variables within a geographic region.

In step, the method may include determining a second trend in the relationship between the data variable and the geographic region at at least one prior point in time.

In step, the method may include determining if the trend in the relationship is irregular within a predetermined threshold with respect to the second trend from the at least one prior point in time.

In step, the method may include, upon determining that the trend in the relationship is irregular within a predetermined threshold, generating an alert. This alert may comprise a visual display, a sound, or any other suitable alarm. The alert may be triggered if there is any irregularity detected in the determined relationship associated with the trend or the second trend of the determined relationship between the same variables within the same geographic region associated with a previous point in time.

is a flowchart illustrating an exemplary method of applying a machine learning model to output a determined relationship between a health variable(s), a data variable(s), and/or a geographic region(s), according to an exemplary embodiment of the present disclosure. For example, an exemplary method(e.g., steps-) may be performed by a geographic location analysis toolautomatically or in response to a request from a user.

According to one embodiment, the exemplary methodfor determining a relationship between variables and geographic region may include one or more of the following steps. In a step, the method may include receiving a plurality of time-stamped patient data from a specific geographical location. Patient data may include, but is not limited to, digital images of a pathology specimen (e.g., histology, cytology, etc.), genetic data, pathogenic data, clinical notes, health variables and/or data variables for the patient, test results, MRI scans, CT scans, pathology images, etc. Data may be received via networking or some other means. Patient data may be stored into a digital storage device, such as a hard drive, a network drive, a cloud storage, a RAM, etc.

In a step, the method may include training a machine learning model to predict a future relationship over time in a geographic region at the specific location. Additional information about the received geographic region or location may be included (e.g., context regarding environmental factors, current disease rates, information from neighboring geographical regions, past historical trends, etc.). The training algorithm may be implemented in a number of ways, including but not limited to, a convolutional neural network; a graph convolutional network, e.g., a node could be a feature associated with a geographic region; an autoregressive model; a recurrent neural network; and/or a capsule network.