Systems and methods for automatic benchmarking for radiology

Examples of the subject matter disclosed herein relate to automatic imaging protocol parameter adjustment based on radiation analytics.

In some radiation dose management systems, a user may explore how factors impacting a radiation dose received by a patient compare between protocols, devices, operators, and so on. However, if the user wants to compare only similar studies (e.g., studies captured using the same or similar study variables), the user has to manually sort through data based on chosen stratifications of the studies (e.g., by BMI range, by age range). Additionally, in conventional dose management solutions, a user may manually search for parameters which could explain differences in received radiation dose among studies. Attempts to adjust a radiation dose received by a patient may include adjusting one or more parameters which may influence the radiation dose, and repeating the scan.

The current disclosure at least partially addresses one or more of the above identified issues by a medical system which may be used for radiation dose management, the medical system comprising a client device having a graphical user interface (GUI) and a display device, the client device operably coupled to a network, and an analytics tool configured with instructions stored on a memory and executable by a processor to receive a first user input via the client device, identify values of at least one variable of interest (VOI) from a subject of a benchmark target variable which differ by at least a first threshold amount from other values of the same VOI from other subjects of the benchmark target variable within a benchmarking context, generate and output for display on the display device a causal graph illustrating influences of different study variables on an identified VOI value, and automatically update the causal graph and display thereof based on a second user input to adjust at least one study variable.

A method for the analytics tool may include comparing a VOI value from a first subject of a benchmark target variable with values of the same VOI from other subjects of the same benchmark target variable within a benchmarking context, identifying a VOI value which differs from a comparable VOI value by at least a first threshold, generating at least one candidate explanation for the difference in VOI values, generating a causal graph illustrating at least one candidate explanation for the difference in VOI values, and automatically updating the causal graph to incorporate adjustments to nodes of the causal graph to simulate a radiation dose delivery.

The current disclosure further at least partially addresses one or more of the above identified issues using a method, comprising receiving a first user input, including selection of at least one variable of interest (VOI), a benchmark target variable, and a benchmarking context, for the at least one VOI, comparing values of the VOI for a first subject of the benchmark target variable with values of the same VOI from other subjects of the benchmark target variable within the benchmarking context to identify VOI values which differ by at least a first threshold amount from other VOI values of the same VOI, for the VOI values identified as differing by at least the first threshold amount, generating and outputting for display an output including a subject identifier and at least one candidate explanation for the identified VOI value, receiving a second user input selecting a subject from the output by selecting the respective subject identifier, comparing study variables used to generate the identified VOI value to average values of respective study variables of comparable VOI values, and generating and outputting for display a causal graph indicating at least one study variable of the study variables used to generate the identified VOI which differs by at least a second threshold amount, receiving a third user input including an adjustment to at least one study variable of the causal graph, wherein adjustment includes adding and/or removing at least one study variable node, and/or adjusting a value of the at least one study variable node, updating the causal graph based on adjustments included in the third user input, and outputting the updated causal graph for display on a display device.

It should be understood that the summary above is provided to introduce in simplified form a selection of concepts that are further described in the detailed description. It is not meant to identify key or essential features of the claimed subject matter, the scope of which is defined uniquely by the claims that follow the detailed description. Furthermore, the claimed subject matter is not limited to implementations that solve any disadvantages noted above or in any part of this disclosure.

Described herein are systems and methods for automating radiation dose management using a tool which performs an assessment of received radiation dose among study parameters. Study parameters may be selected by a user and the tool may automatically compare radiation dose values within the selected study parameters. For example, the tool compares values of a variable of interest (VOI) for a first subject of a benchmark target variable with values of the same VOI from other subjects of the benchmark target variable within a benchmarking context to identify VOI values which differ by at least a first threshold amount from other VOI values of the same VOI. The VOI is representative of a radiation dose (e.g., a radiation dose metric), and may be a dose length product (DLP), a computed tomography dose index (CTDI), a size specific dose estimate (SSDE), and/or an image quality related to a radiation dose. The benchmark target variable may define a population whose VOI values are compared. For example, the benchmark target variable may be operators, hospitals, devices and so on, and a benchmark target variable subject is an individual entity of the benchmark target variable (e.g., a first operator, a second operator, and so on). When the benchmark target variable is “operators”, VOI values (e.g., DLP values) from scans performed by different operators within a given context may be compared, as further described herein. The benchmarking context defines a target set of historical studies to be compared. The benchmarking context may be defined as a timeframe (e.g., within the past three years), as a distance (e.g., healthcare providers within a 50-mile radius), an amount of locations and/or devices (e.g., three hospitals) and so on. For example, VOI values (e.g., DLP values) from different benchmark target variable subjects (e.g., from scans performed by different operators) within the past year may be compared to identify instances of high radiation doses. The tool enables automatic determination of which one or more subjects of the benchmark target variable deliver a greater radiation dose to patients, compared to peer subjects of the same benchmark target variable. The tool further provides some candidate explanations for the difference and visual depictions of relationships among study variables which may influence differences in radiation dose among benchmark target variable subjects.

Dose index registries including radiation dose data are typically held at a national level and allow for comparison of dose averages for comparable studies between hospitals. The systems and methods described herein adds to current uses of dose registries a versatility of benchmark target variables, automatic matching of similar studies, and the possibility to use the same method for a single hospital for internal benchmarking.

In this way, medical providers may automatically evaluate where adjustments may be made to radiology practices, which may reduce a workload and/or time spent on dose-level inspections, compared to a manual comparison of the devices, operators, protocols, and so on. Also, when the tool is used to compare among hospitals/sites (e.g., automatically pinpoint areas of difference), the customer (e.g., hospital/site) may compare its practices against other hospitals, when protocol normalization is included. The tool may thus assist a healthcare provider in identifying best-practices (e.g., practices which result in a desired radiation dose level). Benchmarking on operators, devices, protocols, and so on cases personnel training. In this way, identification of study variables which influence radiation dose levels among different operators, devices, protocols, and so on may highlight opportunities where practices may be adjusted so a desirable dose is delivered to a patient.

shows a block diagram illustrating a systemfor medical data transmission and analytics, including an analytics tool which may store and execute instructions for a method for identifying instances of high radiation dose in radiation dose data. A high-level example of the method for identifying instances of high radiation dose, as well as generating candidate explanations and a display illustrating influences of study variables on radiation dose is shown in. A detailed method for identifying instances of high radiation dose, identifying candidate explanations, and generating and automatically updating causal graphs is illustrated in.shows a first example strategy for comparing variable of interest (VOI) values. Information regarding benchmark target variable subjects which are identified as generating VOI values which differ from VOI values of peer subjects of the same benchmark target variable by at least a first threshold amount is output for display, an example of which is shown in. The display may be used to select a benchmark target variable subject to further analyze.shows a causal graph illustrating influence of study variables on VOI values, for example, of a selected benchmark target variable subject. The causal graph ofmay be manipulated to simulate effects of different study variables on a VOI value, as described in the high-level method of.shows a first example updated causal graph based on removal of a study variable from the causal graph of,shows a second example updated causal graph based on addition of a study variable to the causal graph of, andshows a third example updated causal graph based on adjustment to a study parameter value of the causal graph of.

shows a block diagram illustrating a systemfor medical data transmission and analytics. The systemmay be used to transmit medical data, collected by a hospital modality and stored in a medical database, to an analytics tool. In the examples described herein, medical data which is transmitted by the medical data transmission system may be related to a radiation procedure performed by hospital modalities such as imaging systems, however the methods and systems described herein may also be applied for other types of medical data collected by other modalities.

An imaging devicemay be a hospital modality, such as a computed tomography (CT) system or an x-ray angiography (XA) system. The imaging devicemay perform imaging procedures, such as radiation observations in which a patient is exposed to a radiation dose. The imaging devicemay share data collected by a respective modality to a storage device via at least one of multiple connectivity methods. The multiple connectivity methods may include (e.g., Digital Imaging and Communications in Medicine (DICOM®) images, DICOM® SRs, Health Level Seven (HL7®), proprietary logs associated with a manufacturer or operating system, and so on. The imaging devicemay be operably coupled to a medical database and/or storage device via a wired connection, a wireless connection, and/or any method for communicably connecting systems. As described herein “operably coupled” is to be understood as coupling of elements via connectivity methods (e.g., wired connection, wireless connection, and so on) which enable transfer of data, signals, requests, and/or other information among the operably coupled elements.

In the example described herein, the medical database to which the imaging deviceis operably coupled is a radiation dose database (RDD)of a radiation dose management system (RMS). The RDDis configured to receive radiation dose data from at least one imaging deviceand store the radiation dose data. The RDDmay be an external database or a local database (e.g., housed on a device of the system) of the RMS. The RDDmay be a database stored in a mass storage device configured to communicate with secure channels (e.g., HTTPS and TLS), and store data in encrypted form. As further described herein, the imaging devicemay share radiation dose data to the RMS, and the RMSmay enrich the RDDwith data collected from exams performed by the respective hospital modality (e.g., radiation dose data), as well as other exam and/or hospital modality data (e.g., hospital modality manufacturer, hospital modality operator, and so on). The RMSmay be further configured with instructions stored on memory and executable by a processor to enable tracking of radiation doses received by a patient over time.

The RMS, and specifically the RDD, is communicably coupled to an analytics toolto allow transfer of radiation dose data therebetween. The radiation dose data may include a summary of an imaging procedure (e.g., study variables), radiation dose administered to a patient, and so on. In some examples, the analytics toolmay be integrated in the RMS. The RMSmay be configured to expose defined resources, thus exposing radiation dose data through the analytics tool, where the analytics toolmay enhance third-party usage of data.

The RMSmay be configured to receive and store radiation dose data from at least one imaging device. For example, the RMSmay be configured to implement resources including: a CT radiation dose summary, an X-ray radiation dose summary, a radiopharmaceutical radiation dose summary, a CT irradiation event summary, a practitioner, a modality device, an imaging study, and a patient. Other additional or alternative resources may be included, such as patient body mass index (BMI) and patient weight observation. In some examples, the analytics toolmay be configured to collect radiation information following requests of the consuming applications. Radiation information may be distributed among different resources of the analytics tool.

As described herein, the analytics toolis configured to use defined resources (e.g., radiation dose data) for macroscopic analysis of radiation dose data. This may be used for comparison between devices, cohorts of patients, or comparison between hospitals regarding national radiation dose levels, for example. For example, the analytics toolmay be configured to enable visualization, analysis, analytics, and transformation of data. Data may be collected from multiple sources and aggregated and presented for query and analysis. The analytics toolmay enable creation of dashboards based on collected data, which allow users to select metrics and cohort levels to aggregate data in selected cohorts and groups, providing a macroscopic view of the data for analysis, such as detection of anomalies, as described herein with respect to.

The analytics toolmay be communicably coupled to one or more client devicesfor display, manipulation, and visualization of analytics on the radiation dose data. Each client device may include a processor, memory, communication module, user input device, display (e.g., screen or monitor), and/or other subsystems and may be in the form of a desktop computing device, a laptop computing device, a tablet, a smart phone, or other device. The client devices may be located locally at the hospital (e.g., as part of hospital administration) and/or remotely from the hospital (such as a user's mobile device). In some examples, the analytics toolmay be integrated in the client device. When requested, a report generated by the analytics toolmay be output for display on a display device of a client device as one or more graphical user interfaces. For example, the client devicemay be configured with a graphical user interface (GUI)and a display device. As an example, the client devicemay store one or more GUI templates in memory that include placeholders for relevant information transmitted to the client devicefrom the analytics tool.

The analytics toolmay be configured with resources including a processor, a memory, and a communication modulethat may be allocated to store and execute one or more of the methods described herein. Elements of the systemmay communicate with each other via a network, which may be a suitable wired and/or wireless network. One or more of the devices described herein may be implemented over a cloud or other computer network.

Communication modulefacilitates transmission of electronic data within and/or among one or more systems. Communication via communication modulecan be implemented using one or more protocols. In some examples, communication via communication moduleoccurs according to one or more standards DICOM®, HL7®, ANSI X12N, and so on. Communication modulecan be a wired interface (e.g., a data bus, a Universal Serial Bus (USB) connection, etc.) and/or a wireless interface (e.g., radio frequency, infrared, near field communication (NFC), etc.). For example, communication modulemay communicate via wired local area network (LAN), wireless LAN, wide area network (WAN), etc. using any past, present, or future communication protocol (e.g., BLUETOOTH™, USB 2.0, USB 3.0, etc.). Memoryincludes one or more data storage structures, such as optical memory devices, magnetic memory devices, or solid-state memory devices, for storing programs and routines executed by processor(s)to carry out various functionalities disclosed herein. Memorymay include any desired type of volatile and/or non-volatile memory such as, for example, static random access memory (SRAM), dynamic random access memory (DRAM), flash memory, read-only memory (ROM), etc.

Processor(s)may be any suitable processor, processing unit, or microprocessor, for example. Processor(s)may be a multi-processor system, and, thus, may include one or more additional processors that are identical or similar to each other and that are communicatively coupled via an interconnection bus.

As used herein, the terms “sensor,” “system,” “unit,” or “module” may include a hardware and/or software system that operates to perform one or more functions. For example, a sensor, module, unit, or system may include a computer processor, controller, or other logic-based device that performs operations based on instructions stored on a tangible and non-transitory computer readable storage medium, such as a computer memory. Alternatively, a sensor, module, unit, or system may include a hard-wired device that performs operations based on hard-wired logic of the device. Various modules or units shown in the attached figures may represent the hardware that operates based on software or hardwired instructions, the software that directs hardware to perform the operations, or a combination thereof.

“Systems,” “units,” “sensors,” or “modules” may include or represent hardware and associated instructions (e.g., software stored on a tangible and non-transitory computer readable storage medium, such as a computer hard drive, ROM, RAM, or the like) that perform one or more operations described herein. The hardware may include electronic circuits that include and/or are connected to one or more logic-based devices, such as microprocessors, processors, controllers, or the like. These devices may be off-the-shelf devices that are appropriately programmed or instructed to perform operations described herein from the instructions described above. Additionally or alternatively, one or more of these devices may be hard-wired with logic circuits to perform these operations.

shows a high-level flow chart illustrating a methodfor identifying instances of high radiation dose, generating candidate explanations for potential causes of the instances of high radiation dose, and simulating instances which may alter radiation dose based on adjustment of study variables which may influence the radiation dose. Instances of high radiation dose as defined herein include instances where a first radiation dose value and a second radiation dose value differ by at least a first threshold amount. For example, a first threshold amount may be an amount of radiation which, when added to a desired radiation dose, increases a total radiation dose to above a desired level, where degradation may occur above the desired level. In some examples, the first threshold may represent a statistically significant difference in values. When the first radiation dose value and the second radiation dose value differ by at least the first threshold amount, the radiation dose having a higher value is identified as having high radiation, and is further analyzed according to the methods described herein. Instructions for the method, and the other methods described herein, may be stored in a memory of an analytics tool, such as the memoryof the analytics toolof, and executed by a processor thereof (e.g., the processor). The processormay receive inputs/requests/selections, as described herein, which may be input by a user via a GUI (e.g., the GUIof the client deviceof).

At, the methodincludes comparing values of a variable of interest (VOI) for a first subject of a benchmark target variable with values of the same VOI from other subjects of the benchmark target variable within a benchmarking context to identify VOI values which differ by at least a first threshold amount from other VOI values of the same VOI. For example, VOI values (e.g., DLP values) from different benchmark target variable subjects (e.g., from scans performed by different operators) within the past year may be compared to identify instances of high radiation doses. As another example, when the VOI is CTDI, the benchmark target variable is devices, and the benchmarking context is three hospitals, CTDI values from scans performed using different devices from three different hospitals may be compared to identify instances of high radiation doses. In some examples, a stratification variable may be selected by a user to further specify the benchmarking context, such that the methodis applied to each stratum of the benchmark target variable. For example, as further described with respect to, the methodmay be used to compare protocols (e.g., identified by the stratification variable) performed by operators (e.g., the benchmark target variable).

Comparing values of a VOI may be performed using matching of observational studies and statistical analysis, as further described with respect to. Briefly, comparing values of a VOI includes matching a first VOI value (e.g., DLP, CTDI, and so on) for a first subject of the benchmark target variable (e.g., an operator, a procedure, and so on) to comparable values, for example, a VOI value from a patient scan having similar study variables (e.g., patient age, size, gender, and so on) to the study variables of the first VOI value. Following matching of observational studies, statistical analysis such as a student's t-test is performed to identify VOI values which differ by at least the first threshold amount. Comparison of VOI values may be performed for each scan performed by each subject of the benchmark target variable within a benchmarking context.

The methodincludes identifying candidate explanations which may help explain why at least one VOI value from a benchmark target variable subject differs from other VOI values from other subjects of the benchmark target variable within a benchmarking context which have similar study variables. At, the methodincludes generating and outputting for display at least one candidate explanation per identified benchmark target variable subject. Candidate explanations may include analyses about parameters of the protocol used to capture the VOI value, characteristics of the device used to perform the protocol, and/or patient characteristics. For example, information regarding relationships among study variables of an imaging study and values of the study variables which result in an identified VOI value may be evaluated to identify one or more study variable values which may directly influence the VOI value. Each possible candidate explanation (e.g., identifying one or more study variables) may be examined, and matching may be applied so the effect of the study variable on the VOI may be evaluated by comparing values of one or more study variables to comparable studies. As further described herein, a study variable may be identified as the candidate explanation for a given VOI value when the study variable has a statistically significant effect on the VOI. For example, the study variable may differ from other comparable study variables by at least a second threshold amount. In another example, the statistically significant effect may be that the VOI value differs from other VOI values from comparable studies by at least a first threshold amount. An example display is shown and further described with respect to. The display may include some or all subjects of the benchmark target variable within a benchmarking context which have been identified as having differing VOI values.

One or more candidate explanations for a differing VOI value may be further analyzed using a causal graph, which may also be used to simulate an effect of adjusting study variables on the VOI value. At, the methodinclude generating and outputting for display a causal graph including a plurality of study variables used in the scan having the differing VOI value. Study variables of the causal graph may include patient characteristics (e.g., age, size, weight), device characteristics such as make and model, operator, protocol, and parameters of the protocol. When displayed, the causal graph may be manipulated, for example by a user. Nodes of the causal graph, each of which represent a factor of the plurality of study variables, may be added and/or removed, and the causal graph may be automatically updated to reflect the addition and/or removal of one or more nodes. In some examples, study variables of the causal graph may be adjusted without addition/removal, for example, adjusting parameter values. Automatic updating of the causal graph includes simulating a scan using the adjusted parameter values and nodes which are still included in the causal graph to generate a predicted VOI value based on the updated causal graph. An example causal graph is shown and further described with respect to. In other examples, a causal forest may be used to illustrate potential influences of study variables, where study variables which may directly influence the VOI value are at a top of a tree of the causal forest. The causal graph shows relationships among the study variables, and simulation of scans and dynamic updating of the causal graph may illustrate candidate explanations of differing VOI values, as well as potential study variables which may be implemented in real scans to achieve a desired VOI value.

Adjusted parameter values and study variables (e.g., addition/removal of nodes) of an updated causal graph may be used as parameters for an imaging study. At, the methodincludes transmitting the plurality of study variables from an updated causal graph to an imaging device. In some examples, the plurality of study variables may be transmitted to the imaging device via a memory or other storage, for example, the plurality of study variables may be stored in the RDDofand retrieved from the RDDby an imaging device of the imaging devices. The adjusted parameter values and study variables may be different from parameter values and study variables used to conduct the imaging scan of the identified VOI (e.g., which differs from other VOI values by at least the first threshold amount), therefore using the adjusted parameter values to conduct an imaging scan may result in an adjusted VOI (e.g., increased or decreased VOI value).

Details of the methodare now to be discussed in further detail. As described above, comparing values of a VOI may be performed using matching of observational studies and statistical analysis. Different methods of comparison may be used depending on selected VOI, benchmarking target variable, and benchmarking context. Matching of observational studies may include automatically correcting for study variables which may directly and/or indirectly influence the VOI values. For example, when comparing two devices with a given protocol (e.g., the benchmark target variable is “devices” and the benchmarking context is the two devices), each device may have a different patient population. Each of the patient populations may be comprised of a different number of patients and/or patients having different characteristics. To perform a comparison between comparable VOI values of the two devices, each patient (e.g., and associated VOI value) from one device may be matched to a comparable patient (e.g., and respective VOI value), where comparable patients have the same or similar study variables. Study variables may include patient characteristics (e.g., size, age, and weight), series type, device, and protocol.

In some embodiments, VOI values are compared by generating all one-to-one combinations of VOI values among benchmark target variable subjects within the benchmarking context and performing statistical analysis to identify differences in VOI values which differ by at least the first threshold amount. Each VOI value may be individually matched to a VOI value from another subject which has the same or similar study variables. VOI values which do not have a corresponding VOI value from another subject may not be compared to another VOI value and may not be included in the output indicating differing VOI values (e.g., as further described with respect to). Though two or more VOI values may be generated by a study which uses the same or similar study variables, the VOI values themselves may be similar or may be different.

In some embodiments, an average of VOI values for each subject of the benchmark target variable is compared to average VOI values of the other benchmark target variable subjects. Prior to averaging VOI values for each benchmarking target variable subject, each VOI value for a subject may be individually compared to all VOI values for other subjects of the benchmark target variable within the benchmarking context to determine if there is a comparable VOI value, as described above. If a VOI value of a subject is identified which does not have a comparable VOI value from other benchmark target variable subjects, the VOI value may be excluded from the average VOI value, in some examples. VOI values for each subject in the benchmarking context may be averaged, such that each subject has an associated average VOI. By averaging VOI values and removing VOI values generated from different study variables, comparisons of radiation dose among subjects may be made between comparable datasets.

For example, in one year, a first operator may have performed 300 studies. Matching of observational studies includes identifying 300 studies from the other operators in the one-year timeframe which are comparable (e.g., the same or similar study variable values) to the studies of the first operator. This is repeated for each operator in the benchmarking context (e.g., second operator, third operator, n−1 operator, n operator). Matching studies for the first operator (or second/third/etc.) are sourced from a pool of studies for all operators in the benchmarking context (e.g., stored in a radiation dose database, such as the RDDof) to have a greater likelihood of finding comparable studies. For example, an imaging device may have eleven operators in one year. Each of the operators may have performed 300 studies. Comparing studies of the first operator to the pool of studies for all operators includes finding a comparable study for each of the 300 studies of the first operator from the pool of 3000 studies from all operators (e.g., excluding the first operator). This ensures robustness of comparison, compared to doing a 1:1 comparison with another operator (e.g., matching the 300 studies of the first operator with the 300 studies of the second operator).

After identifying comparable VOI values, whether individual VOI values or averaged VOI values, statistical analysis is performed on comparable VOI values to identify unbalanced comparisons. For example, a student-t test may be applied to a dataset comprised of all VOI values, except those removed on the basis of being incomparable to other VOI values, to identify VOI values which may indicate high radiation dose (e.g., VOI values which differ from a mean VOI value range by at least the first threshold amount).

shows an example strategy 300 for comparison of VOI values. In the example of, the VOI is a DLP, the benchmark target variable is operators, and the benchmarking context is the past one year as of the date the method is being performed. The example ofis simplified to explain comparison of VOI values; as further described with respect to, more than one VOI may be compared at a time and additional selections may be made to further specify VOI values to be compared.

In the example of, n operatorsperformed procedures which captured DLP values within the past one year. It is to be understood that there may or may not be additional operators between operatorand operator n−1, in some examples. A number of scans performed by each operator may be independent of a number of scans performed by other operators. For example, operatorperformed four scans which captured DLP values and operatorperformed two scans which captured DLP values. Some operators may have performed the same amount of scans which captured DLP values, such as operatorand operator n−1. In some examples, operators may perform less than two and/or greater than four scans which capture DLP values. Each DLP value captured by a scan is illustrated inby a box to the right of the respective operator and labeled with a corresponding number (e.g., DLP 1A, DLP 1B, and so on correspond to operator), and a letter differentiating scans performed by the same operator (e.g., DLP 1A and DLP 1B are two different scans performed by operator). Lettering among operators is irrelevant to scan types or other scan parameters (e.g., DLP 2A has no inherent correlation to DLP 1A signified by “A”).

As described above, matching of observational studiesmay be performed prior to statistical analysis of VOI values to compare VOI values among comparable studies. Study variables for each DLP value of each operator are compared to study variables for all DLP values of all operators in the past year. For example, study variables used to capture DLP1A (e.g., from operator) are compared to study variables used to capture DLP2A, DLP2B, all DLP values from scans performed by operators between operatorand operator n−1, DLPn-1A, DLPn-1B, and so on until a final DLP value from a final operator in the benchmarking context (e.g., DLPnC). In some examples, study variables of DLP values may be compared within a set of operator studies (e.g., study variables of DLP1A may be compared to study variables of DLP1B) as well as values from other operator studies. Study variables of DLP values may be sequentially compared to other study variables of DLP values, and comparison may be halted when the same or similar (e.g., within 5% difference for numeric variables) study variables are identified. In some embodiments, study variables of DLP values may be compared all at once and groups of two or more comparable VOI values may be identified among cases having the same or similar (e.g., within 5% difference for numeric variables) study variables. In some instances, study variables used to capture a DLP value may not have a same or similar occurrence within the benchmarking context. In this case, the DLP value which does not have a comparable value may be omitted from statistical analysis. For example, study variables used to capture each of DLP1C and DLPnA may not have comparable study variables within the benchmarking context, thus DLP1C and DLPnA are shown with a line through them at, indicating that DLP1C and DLPnA will not be included in the statistical analysis. It is to be understood that all other DLP values within the benchmarking context are comparable with at least one other DLP value.

In some examples, statistical analysisincludes performing a one-to-one comparison of a DLP value with a comparable value (e.g., captured from a scan having similar or the same study variables, DLP from the same comparison pair). Each comparison pair may include two DLP values which are not included in other comparison pairs, in some examples. In other examples, some comparison pairs may include the same DLP value. For example, if the study variables of DLP1A, DLP2A, and DLPn-1B are the same as each other and no other cases, a first comparison pair may include DLP1A and DLP2A, and a second comparison pair may include DLP2A and DLPn-1B. Statistical analysismay include determining if a difference between DLP values of a comparison pair is at least the first threshold amount. If the DLP values of the comparison pair do not differ by at least the first threshold amount, then it may be determined that neither of the DLP values in the comparison pair are considerer to indicate a high radiation dose. If the DLP values of the comparison pair differ by at least the first threshold amount, it may be determined that the greater DLP value indicates a high radiation dose. The operator who performed the scan which generated the greater DLP may be considered an outlier operator. Information about the outlier operator is output for display on a display device (e.g., the displayof), and may be used to identify candidate explanations for the high radiation dose, as further described with respect to.

As a check to prevent two high radiation dose cases from being in a comparison pair and not being identified as high radiation dose cases (e.g., not being identified as outlier operators), in some examples all DLP values are compared for all operators in the past year. For example, DLP1A (e.g., from operator) is compared to DLP2A, DLP2B, all DLP values from scans performed by operators between operatorand operator n−1, DLPn-1A, DLPn-1B, and so on until DLPnC. In some examples, DLP values may be compared within a set of operator studies (e.g., DLP1A may be compared to DLP1B) as well as values from other operator studies. DLP values may be sequentially compared to other DLP values, and comparison may be halted when a first DLP value differs from a compared to DLP value by at least the first threshold amount in greater than a threshold number of instances. For example, the first DLP value may be compared to three other DLP values with similar study variables and, if it is determined that the difference between the first DLP value and each of the three other DLP values is at least the first threshold amount, the first DLP value may be identified as indicating a high radiation dose and the corresponding operator may be identified as an outlier, to be further discussed with respect to. In some examples, the threshold number of instances may be greater than or less than three instances. In some examples, scans may be performed using a standardized set of study variables, therefore there may be many DLP values which are captured using the same or similar standardized set of study variables. These DLP values may be grouped together to be compared by statistical analysis.

In some examples, statistical analysisis performed on average VOI values for each benchmark target variable subject. For example, following removal of DLP values which do not have a comparable value (e.g., based on matching of study variables) as indicated inby a line drawn through DLP values, the remaining DLP values for each operator may be averaged 308. Statistical analysismay then be performed on the averaged DLP values. For example, the averaged DLP values may be compared using a student-t test to identify a significance of a difference between means of data sets. Within the student-t test, a first dataset may be the DLP values of the first operator and a second dataset may be the DLP values of the second operator. Significant differences in DLP values may be identified where the difference is at least the first threshold amount. In some examples, there may be groupings of DLP values for each operator, where the DLP values are grouped based on similar study variables. In this case, DLP values may be statistically analyzed within each grouping for an operator. When other operators have groupings of DLP values based on similar study variables, DLP values may be statistically analyzed among operators for values captured using the same or similar study variables. As described above, the greater DLP value may indicate a high radiation dose, and associated operator data may be output for further analysis.

Following statistical analysis of VOI variables and identification of benchmark target variable subjects associated with high radiation doses (e.g., outlier operators), candidate explanations are generated for potential causes of the instances of high radiation dose. For example, for a VOI value identified as indicating a high radiation dose, study variables of the scan which generated the VOI value may be compared to standardized study variables to identify potential differences in study variables values which may result in high radiation dose. Further detail regarding candidate explanations and comparison of study variables is described with respect to.

Turning now to, an example display is shown including a tablewhich may be output for display on a display device (e.g., the display device) as a result of executing operationof the method. For example, the tablemay be generated following statistical analysisof, and information about the one or more outlier operatorsis included in the table. The tableshows results of a comparison of operator practices (e.g., “operators” is the benchmark target variable/population) for two protocols (e.g., “protocols” is the benchmarking context, specifically a first protocol and a second protocol). The tablefurther includes candidate explanations indicating why an operator was identified by the method as having a high radiation dose. The tableis an example of how information from the statistical analysis may be organized and presented for display on a display device, however other displays may be generated and displayed without departing from the scope of the present disclosure.

A first headerand a second headerinclude information about the benchmarking context. In the example of, the first headerincludes information about a first protocol, which is a standard brain scan performed in facility one by eight operators. The second headerincludes information about a second protocol, which is an axial brain scan performed in facility one by 281 operators. The 281 operators who performed the second protocol may or may not include at least one of the eight operators who performed the first protocol, and vice versa. In other examples, there may be more than or less than two headers, and each header may include information about different benchmarking context, compared to. For example, when the benchmarking context is studies from the past year and the benchmark target variable is hospitals, the first header may be a first month or a first quarter, the second header may be a second month or a second quarter, a third header may be a third month or a third quarter, and so on.

A first columnunder each of the first headerand the second headeridentifies at least one operatorwhich has been identified as having a high radiation dose (e.g., radiation dose which differs from a desired radiation dose by at least the first threshold amount). In the example of, the first columnshows an operator's name, however in other examples a different identifier may be used, such as an identification number or operator code. As described with respect to, operators who performed scans which had high radiation doses may be identified by comparing the variable of interest of a first operator to variables of interest of all other operators within the benchmarking context.

A second columnunder each of the first headerand the second headerindicates, for each operator identified in the first column, at least one candidate explanationfor the high radiation dose. Candidate explanations may include analyses about parameters of the protocol used to capture the VOI value, characteristics of the device used to perform the protocol, and/or patient characteristics. An up arrow may be positioned next to a parameter in the second columnto indicate the parameter may be high (e.g., total number of irradiations is higher than average total number of irradiations for protocols performed by other operators). In other examples, a down arrow may be positioned next to the potential parameter to indicate the parameter may be low. In some examples, the first indicator may be something other than an icon, for example, a text side, boldness, color, font, and so on may be adjusted to indicate the potential parameter state (e.g., high/low).

For the first protocol (e.g., under the first header), Operatoris the only operator of the eight operators who performed the first protocol to be identified as performing a scan which has a high radiation dose. For example, a first radiation dose received by a first patient as a result of the first procedure, performed by Wagner, may be greater than a second radiation dose received by a second patient as a result of either the first procedure or the second procedure, either of which performed by an operator, which may include Wagner. The first radiation dose (e.g., identified as a high radiation dose) may be greater than the compared to radiation dose by at least the first threshold amount. For example, the first radiation dose may be 10 mGy*cm greater than the second radiation dose. As shown in the second column, a candidate explanation for identifying Wagner's scan as having a high radiation dose may be that a study total DLP for Wagner's scan may be high to compare to study DLP values of other operators within the first protocol group and the second protocol group. As briefly described above and further described with respect to, more than one VOI may be selected when initiating the method. Thus, more than one VOI are shown in the second columnfor some of the operators in the first column. For example, additional candidate explanations for Operatorinclude a high total number of irradiations for the study, a high spiral total number of irradiations, and a high spiral CTDI.

For some potential parameters, such as is shown with the high spiral CTDI in the case of the first operator, a second indicator may be shown to indicate a potential adjustment which may be made to the potential parameter to adjust the outcome of future protocol executions (e.g., lower the dose). For example, the second indicator may be a wrench and may be positioned next to, in front of, or after the potential parameter and may be followed by text indicating an adjustment which may be made to adjust the potential parameter. For example, to decrease spiral CTDI for Operator, it is proposed that a pitch factor be adjusted, where the pitch factor for single slice CT systems is a distance that a table (e.g., of a CT system on which an imaging subject may be positioned) travels in one 360-degree gantry (e.g., of the CT system) rotation, divided by beam collimation. The pitch factor affects both image quality and radiation dose, where a pitch factor of greater than one decreases patient dose and may also decrease image quality, as fewer projections are obtained, and a pitch factor less than one may increase radiation dose and increase image quality, as more projections are obtained. Under the second headerfor Operator, adjusting an average x-ray tube current/exposure is provided as a potential adjustment which may address the sequenced CTDI being high.

To further analyze candidate explanations for identified high radiation doses and potential adjustments which may be made to reduce the radiation dose, a subject may be selected for further study. For example, a user may select data associated with Operatorfor further analysis by selecting at least one of the operator's name (e.g., from the first column) and/or a candidate explanation (e.g., from the second column). A causal graph may be generated which shows relationships among study variables used to generate the VOI value of interest. For example, values of study variables used to generate the VOI value of interest may be compared to standard study variable values used to perform the same scan. For example, standard study variable values may be industry standards, hospital/healthcare provider standards, device standards, and so on which may be a default value used for each scan. For some scans, such as when a patent is larger than an average patient size, study variable values may be adjusted from the standard. In some cases, this may result in an increase in radiation dose, as indicated by the identified VOI.

Turning to, a causal graphis shown. The causal graphis for a computed tomography (CT) case and includes study variables used to perform a CT scan. Directional connectors between labeled nodes for each study variable indicate relationships among the study variables. For example, in the causal graph, a patient age node, an operator node, and a BMI node all point to a protocol node. This indicates that a probability distribution of the protocol node (e.g., a child node) is dependent on values of the parent age node, the operator node, and the BMI node (e.g., parent nodes). Given values of the parent node(s), the child node is independent of non-descendants. For example, once values of the protocol node are known, values of a parameter node are influenced by values of the protocol node and not by values of the patient age node, the operator node, and/or the BMI node. Similarly, once values of the BMI node are known, values of the parameter node are not influenced by values of a patient size node or a patient weight node. The causal graph defines a general probability structure and is used to specify joint distribution.

Briefly returning to, selecting “spiral CTDI” in the second columnunder the first headerfor Operatormay generate the causal graph. Information regarding study variables used to capture the identified VOI may be retrieved from a radiation database, such as the RDDof the RMSof. Retrieved study variable information may be compared to standard study variable values to identify potential deviation from the standard. In some cases, there may be a study variable which is identified as differing from a standard study variable value by at least a second threshold amount. For example, the second threshold amount may be an amount which may increase a radiation dose. In other examples, the study parameters for the identified VOI value may be compared to average values of respective parameters for comparable VOI values.

Following analysis of study variables used to generate the CTDI, generation and output of the causal graphmay indicate that the CTDI (e.g., indicated by a CTDI node) is influenced by parameters and a device model. As shown in, nodes of study variables which are determined to directly influence the CTDI may be directly connected to the CTDI node. In some examples, study variables determined to directly influence the CTDI may additionally be indicated on the causal graph, for example, by a color, pattern, and/or other visual indicator. The parameters are influenced by the device model, patient BMI, the protocol, and the operator. Thus, adjustments to any nodes which directly influence the CTDI, and/or adjustments to nodes which influence nodes that directly influence the CTDI may result in adjustment to the CTDI. In this way, the causal graph is an example of an exploratory medium which may be modified to simulate possible adjustments to study variables without performing a simulated scan using an imaging device. In some examples, as further described herein, study variable values for one or more imaging studies may be automatically updated based on an updated causal graph which has been updated to simulate potential adjustments which may change (e.g., increase or decrease) a respective VOI value.