Facilitating interpretability of classification model

This application is a U.S. national phase application of International Application No. PCT/EP2020/074171 filed Aug. 31, 2020, which claims the benefit of EP Application Serial No. 19194471.9 filed Aug. 29, 2019 and is incorporated herein by reference.

The invention relates to a system, such as a clinical decision support system, which is configured to apply a classification model to clinical data, such as patient data, and to a computer-implemented method for applying a classification model to clinical data. The invention further relates to a computer-readable medium comprising instructions for causing a processor system to perform the method.

Clinical decision support systems are increasingly used in clinical practice, for example when prioritizing emergency room episodes or when predicting a treatment outcome for a given patient. The input for such clinical decision support systems is typically clinical data, such as patient data. A clinical decision support system may be configured to infer clinically relevant information from the clinical data. For that purpose, the clinical decision support system may apply a classification model to the clinical data which may provide a classification of the clinical data and may thereby implement at least part of a clinical decision-making process. The classification model may in many cases be a machine-learned (‘trained’) classification model, such as a trained neural network, Support-Vector Machine (SVN) model, etc. Such types of clinical decision support systems are also referred to as being ‘data-driven’ as they are not fully defined by experts anymore.

In the design of such data-driven clinical decision support systems, it is a concern to provide the decision-making entity (e.g., the physician) with trust in the clinical decision support information provided by the classification model, for example by providing the physician with an understanding of how this clinical information is calculated to establish sufficient trust in the clinical information.

An additional challenge in data-driven clinical decision support systems is that new data may be added, e.g., by retraining the classification model, in which case the physician may wish to understand how this affects the classification model's performance, not only in terms of technical metrics, but also in terms of how the decision-making process of the classification model is affected by the new data. For example, while it is possible that the model accuracy remains stable, the new decision-making process after re-training with the new data may be misleading. A typical example may be the classic “Anscombe's Quartet”, where all datasets show the same statistics, but in reality, the data distributions are clearly different.

The difficulties in bringing transparency to the decision-making process are related to the fact that a classification model often resembles an opaque black-box where the user is only able to obtain insights in the model's input, output and the technical characteristics of the model, such as the accuracy or recall or other metrics such as Receiver Operating Characteristic (ROC). However, typically no insights are provided into the model's internal decision-making process. This is typically due to the model's complexity and the multi-dimensional aspects, which are hard to interpret by humans. Different approaches have been proposed to overcome this, such as Bayesian networks, which provide a visual graph of the feature relationships, or data visualization approaches allowing the user to interpret the data in a human interpretable way. However, these techniques are typically specific to a type of classification model and cannot be generalized to different types of classification models. This severely limits the applicability of such techniques.

It may be desirable to be able to facilitate the interpretability of a classification model's decision-making process in a more model agnostic manner.

In accordance with a first aspect of the invention, a system is provided which is configured to apply a classification model to clinical data.

clinical data comprising data instances which are each representable as a feature vector in a multi-dimensional feature space; a classification model configured to be applied to the feature vector to provide a classification of the respective data instance; a data interface for accessing: apply a non-linear and manifold-preserving dimensionality reduction technique to all or a subset of the feature vectors to obtain a plurality of clinical data points in a lower-dimensional space; create synthetic data points in the lower-dimensional space and determine feature vectors for the synthetic data points by applying an interpolation technique to the feature vectors of the clinical data points, thereby obtaining an interpolated feature vector for each of the synthetic clinical data points; for each synthetic clinical data point: apply the classification model to the respective interpolated feature vector to obtain a classification for the synthetic clinical data point, and determine a classification uncertainty of the classification; and generate a visualization of the lower-dimensional space for display to a user, wherein the visualization comprises a visualization of the classification uncertainty in visual relation to the synthetic clinical data points. a processor subsystem configured to: The system comprises:

In accordance with a further aspect of the invention, a computer-implemented method is provided for applying a classification model to clinical data.

clinical data comprising data instances which are each representable as a feature vector in a multi-dimensional feature space; a classification model configured to be applied to the feature vector to provide a classification of the respective data instance; accessing: applying a non-linear and manifold-preserving dimensionality reduction technique to all or a subset of the feature vectors to obtain a plurality of clinical data points in a lower-dimensional space; creating synthetic data points in the lower-dimensional space and determining feature vectors for the synthetic data points by applying an interpolation technique to the feature vectors of the clinical data points, thereby obtaining an interpolated feature vector for each of the synthetic clinical data points; applying the classification model to the respective interpolated feature vector to obtain a classification for the synthetic clinical data point, and determining a classification uncertainty of the classification; and for each synthetic clinical data point: generating a visualization of the lower-dimensional space for display to a user, wherein the visualization comprises a visualization of the classification uncertainty in visual relation to the synthetic clinical data points. The method comprises:

In accordance with a further aspect of the invention, a computer-readable medium is provided comprising transitory or non-transitory data representing a computer program, the computer program comprising instructions for causing a processor system to perform the computer-implemented method.

The above measures involve accessing clinical data which may comprise several data instances which are each representable as a feature vector in a multi-dimensional feature space. For example, such clinical data may be patient data, and each data instance may relate to a different patient. In this example, the clinical data of a particular patient may form a feature vector in the multi-dimensional feature space. For example, if a particular data instance contains 33 values, such as sex, weight, height, blood type, etc., the data instance may be representable as a data point in a 33-dimensional feature space, with the coordinates of the data point representing the feature's values, e.g., ‘F’, ‘60 kg’, ‘170 cm’, ‘O-negative’, etc. Such representation of data as feature vectors is known per se in data classification.

Furthermore, a classification model may be accessed, e.g., in the form of a data representation of the classification model, e.g., as classification model data. The classification model, which may be a machine-learned classification model such as a neural network, SVN or the like, may be configured to be applied to the feature vector to provide a classification of the respective data instance. Such a classification may in general be an inference, e.g., a prediction, for example of a clinical diagnosis, and may in the context of clinical decision support constitute clinical decision support information which may support a user's decision-making.

The above measures further involve applying a non-linear and manifold-preserving dimensionality reduction technique to all or a subset of the feature vectors to obtain a plurality of clinical data points in a lower-dimensional space. Such non-linear and manifold-preserving dimensionality reduction techniques are known per se, and are based on the manifold assumption that states that (high-dimensional) data typically lies at least roughly on a low-dimensional manifold, with this being also the underlying assumption in various machine learning based techniques. A non-limiting example of such a non-linear and manifold-preserving dimensionality reduction technique is the t-distributed Stochastic Neighbor Embedding (t-SNE) algorithm. As a result of applying this technique, respective clinical data points are obtained in the lower-dimensional space. Here, ‘lower-dimensional’ refers to a dimensionality which is lower, and in some cases much lower, than the dimensionality of the original multi-dimensional feature space. In some examples, the dimensionality reducing technique may be a non-linear or manifold-preserving dimensionality reduction technique. Other examples of suitable techniques include but are not limited to UMAP, ISOMAP, HSNE and A-tSNE, each being known per se in the art of dimensionality reduction of multi-dimensional data.

As a result of the above, the lower-dimensional space, which may also be referred to as an ‘embedding space’, may now contain clinical data points which each have an associated higher-dimensional feature vector. Feature vectors for other data points in the lower-dimensional space may be obtained by applying an interpolation technique to the feature vectors of the clinical data points. For example, such an interpolation technique may involve applying a weighted average to the feature vectors of clinical data points in a neighborhood of the ‘other’ data point, in which the weighting is inversely proportionate to the distance to the respective clinical data points in the lower-dimensional space.

As a result, synthetic clinical data points may be obtained in the lower-dimensional space, referring to coordinates in the lower-dimensional space for which an interpolated feature vector has been determined in the above-described manner. For each of these synthetic clinical data points, a classification may be obtained by applying the classification model to the respective interpolated feature vector, and a classification uncertainty of the classification may be determined. Such a classification uncertainty may be determined in various known ways, and may in general depend on the type of classification model, as also elucidated elsewhere.

The above measures are based on the insight that the non-linear and manifold-preserving dimensionality reduction technique allows the feature space to be represented in a lower-dimensional manner in which the variance in the clinical data is preserved to at least a substantial degree. Such a lower-dimensional space is much easier to interpret for a human observer than the higher-dimensional feature space. For example, if the classification of the clinical data is plotted in such a lower-dimensional space, e.g., as a different visual representation overlaying each clinical data point, a user may more easily see decision boundaries in the classification by the classification model than in the original higher-dimensional feature space.

However, the clinical data which has been used as input to the dimensionality reduction technique may yield clinical data points which are distributed non-uniformly and/or sparsely across the lower-dimensional space.

It may be of interest to obtain visual feedback on the performance of the classification model also in other areas of the lower-dimensional space, e.g., in areas which do not contain any or a sufficient number of original clinical data points, as the classification model may later be applied to clinical data of which the corresponding clinical data points lie in such areas of the lower-dimensional space. This is addressed by generating the aforementioned synthetic clinical data points, which may for example be determined in a regular grid in the lower-dimensional space and which in general provide more data points in the lower-dimensional space and thereby increase the density of data points in the lower-dimensional space. Such increased density may greatly improve the interpretability of the visual feedback, particularly if the original clinical data points are only sparsely distributed.

The lower-dimensional space may then be visualized, e.g., as a 2D or 3D image in case of a 2D or 3D space, and the classification uncertainty associated with the interpolated feature vectors of the synthetic clinical data points may be visualized in visual relation to the synthetic data points. For example, the pixels or voxels representing the synthetic data points may be assigned a saturation or an intensity which represents the uncertainty. In some embodiments, the classification uncertainty of all clinical data points, i.e., original and synthetic, may be visualized.

Advantageously, the classification uncertainty across the lower-dimensional space may be shown to the user, which may indicate areas in which the uncertainty is particularly high (or the certainty particularly low). This may indicate a need for adjusting the classification model, e.g., by parameter tuning or otherwise, or if the classification model is a trained classification model, the need for more training data which comprises data instances in the particular area, or in general a need for the user to treat the classification by the classification model in this area cautiously.

Advantageously, the above measures provide a visualization of the classification uncertainty of a classification model across the lower-dimensional space while considering the classification model as a ‘black box’ by not being dependent on internal parameters of the classification model. Rather, the visualization is provided based on input (feature vectors) and output (classification) of the classification model and derived parameters (classification uncertainty). Advantageously, the above measures may facilitate the interpretability of a classification model's decision-making process in a more model agnostic manner.

Optionally, the processor subsystem is further configured to generate, in the visualization of the lower-dimensional space, a visualization of the classification by the classification model. In addition to the classification uncertainty, also the classification itself may be visualized. For example, the pixels or voxels in the 2D or 3D image may be assigned a saturation or an intensity which represents the classification uncertainty and a hue which represents the classification. This may enable a user to perceive classification boundaries, which may elsewhere also be referred to as ‘decision boundaries’, and in particular to perceive complex classifications boundaries which may indicate a poor generalization of the classification model to such areas.

Optionally, the system comprises a user interface subsystem comprising a display output for displaying said visualizations and a user input interface for receiving user input data from a user input device operable by a user, and the processor subsystem is configured to, via the user interface subsystem, enable a user to select a synthetic clinical data point, and in response to said selection, provide a visualization of the respective interpolated feature vector. This user interface functionality may enable a user to easily see the interpolated feature vector of a selected synthetic clinical data point, e.g., as a visualization of each feature vector component, which may in turn enable a user to draw conclusions on the relation between i) the classification and/or classification certainty and ii) the feature(s) on which the classification is based.

Optionally, the processor subsystem is configured to, via the user interface subsystem, enable the user to select two synthetic clinical data points, and in response to said selection, provide a visualization of a difference between the respective interpolated feature vectors. This user interface functionality may enable a user to easily see the differences in interpolated feature vectors between selected synthetic clinical data points, which may be particularly useful near classification boundaries, allowing a user to draw conclusions on the relation between a change in classification and the feature vector differences.

Optionally, the classification model is trained on training data, and the clinical data for which the visualization is provided is the training data of the classification model. The above measures may be applied to the training data itself, which may allow a user to obtain feedback on the classification and classification certainty with respect to the training data. This may for example indicate a need for more and/or a different type of training data.

Optionally, all or a subset of the data instance of the training data comprise or are associated with a respective ground truth classification, and the processor subsystem is configured to generate a visualization of the ground truth classification in visual relation with the clinical data points in the visualization of the lower-dimensional space. By visualizing the ground truth, a difference between the ground truth and the classification by the classification model may be made visible, which may indicate misclassifications or other problems.

generate further clinical data points representing the further clinical data in the lower-dimensional space; and visualize the further clinical data points in the visualization of the lower-dimensional space. Optionally, the data interface is configured to access further clinical data, and the processor subsystem is configured to:

Such further clinical data point(s) may represent new input data after the training. By plotting such further clinical data point(s) in the lower-dimensional space, the spatial relation between the further clinical data point(s) and the original clinical data points may be made visible. For example, if both types of data points form separate clusters in the lower-dimensional space, this may, in case of a trained classification model, indicate that the classification model may be insufficiently generalized to classify the new input data. Furthermore, such visualization may allow a user to visually relate the new input data to the classification and classification certainty of the classification model.

Optionally, the processor subsystem is configured to determine the classification and the classification uncertainty, and to visualize the classification uncertainty, for a regular grid of synthetic clinical data points in the lower-dimensional space. The interpolated feature vectors, and in turn the classification and classification uncertainty relating to said interpolated feature vectors, may be determined for data points in a regular grid. For example, if the lower-dimensional space is visualized as a 2D image, the classification and classification uncertainty may be determined for each pixel of the 2D image.

Optionally, the non-linear and manifold-preserving dimensionality reduction technique is a t-distributed Stochastic Neighbor Embedding (t-SNE) algorithm. Alternative algorithms include, but are not limited to UMAP, ISOMAP, HSNE and A-tSNE. Optionally, applying the interpolation technique comprises using a KD-tree algorithm to search for clinical data points to be used in the interpolation. The KD-tree algorithm may be used to find a set of K-Nearest Neighbor (KNN) clinical data points for interpolation. Alternatively, any other algorithm may be used for KNN computation. Examples of such algorithms include but are not limited to approximate KD-Trees and hashing techniques.

Optionally, the system is part of a workstation or imaging apparatus.

It will be appreciated by those skilled in the art that two or more of the above-mentioned embodiments, implementations, and/or optional aspects of the invention may be combined in any way deemed useful.

Modifications and variations of a system, computer-implemented method and/or any computer program product, which correspond to the described modifications and variations of another one of said entities, can be carried out by a person skilled in the art on the basis of the present description.

It should be noted that the figures are purely diagrammatic and not drawn to scale. In the figures, elements which correspond to elements already described may have the same reference numerals.

20 data storage 30 clinical data 40 model data 50 visualization data 60 user input device 62 user input data 80 display 100 system for applying classification model to clinical data 120 data interface 122 external data communication 124 internal data communication 140 processor subsystem 142 internal data communication 160 user interface subsystem 170 user input interface 180 display output 182 display data 200 lower-dimensional space 210 t-SNE-1 dimension 220 t-SNE-2 dimension 230 clinical data point with ground-truth classification 300 visualization of classification and classification uncertainty of synthetic clinical data points as 2D image 310 312 ,complex decision boundaries in classification 320 misclassification in ground-truth 330 areas with low classification confidence 340 areas with high classification confidence 350 selection of synthetic clinical data point 360 selection of two synthetic clinical data points 400 visualization of interpolated feature vector 410 visualization of difference in interpolated feature vectors 420 feature value axis 422 feature value difference axis 430 feature component axis 500 method of applying classification model to clinical data 510 accessing clinical data 520 accessing classification model 530 applying dimensionality reduction technique 540 determining feature vectors for other data points 550 determining classification and classification uncertainty 560 generating visualization of classification uncertainty 600 computer-readable medium 610 non-transitory data The following list of reference numbers is provided for facilitating the interpretation of the drawings and shall not be construed as limiting the claims.

1 FIG. 100 shows a systemfor applying a classification model to clinical data and which may be configured to generate a visualization of a classification uncertainty of the classification model and to display said visualization.

100 120 30 30 120 122 20 30 20 100 120 100 30 120 20 1 FIG. The systemis shown to comprise a data interfacefor accessing clinical datacomprising data instances which are each representable as a feature vector in a multi-dimensional feature space. For example, the clinical datamay comprise data records for a plurality of patients, with each data record representing a data instance. For example, as also shown in, the data interfacemay provide data accessto an external data storagewhich may comprise said clinical data. The data storagemay, for example be constituted by, or be part of, a Picture Archiving and Communication System (PACS) or an Electronic Medical Record (EMR) database of a Hospital Information System (HIS) to which the systemmay be connected or comprised in. Alternatively, the data interfacemay provide data access to an internal data storage which is part of the system. Alternatively, the clinical datamay be accessed via a network. In general, the data interfacemay take various forms, such as a network interface to a local or wide area network, e.g., the Internet, a storage interface to an internal or external data storage, etc. The data storagemay take any known form, such as a hard drive or an array of hard drives or an SSD or an array of SSDs.

20 40 20 30 40 30 40 120 120 The data storageis further shown to comprise model datadefining a classification model for being applied to the feature vector to provide a classification of the respective data instance. Depending on the embodiment, the data storagemay comprise one or both types of data,. In some embodiments, the clinical dataand the model datamay each be accessed from a different data storage, e.g., via a different subsystem of data interface. Each subsystem may be of a type as described above for the data interface.

100 140 120 124 140 100 140 20 50 The systemis further shown to comprise a processor subsystemwhich may internally communicate with the data interfacevia data communication. The processor subsystemmay be configured to, during the operation of the system, apply a non-linear and manifold-preserving dimensionality reduction technique to all or a subset of the feature vectors to obtain a plurality of clinical data points in a lower-dimensional space, determine feature vectors for other data points in the lower-dimensional space by applying an interpolation technique to the feature vectors of the clinical data points, thereby obtaining synthetic clinical data points in the lower-dimensional space each having an interpolated feature vector, and for each synthetic clinical data point, apply the classification model to the respective interpolated feature vector to obtain a classification for the synthetic clinical data point, and determine a classification uncertainty of the classification. The processor subsystemmay be further configured to generate a visualization of the lower-dimensional space for display to a user, wherein the visualization comprises a visualization of the classification uncertainty in visual relation to the synthetic clinical data points. Such visualization may be stored in the data storage, e.g., in the form of visualization data.

100 2 4 FIGS.-C It is noted that the operation of the system, including various optional aspects thereof, will be further explained with reference to.

100 160 140 160 142 160 100 100 160 170 62 60 60 60 170 60 60 160 180 182 80 100 80 1 FIG. 1 FIG. As an optional component, the systemis shown to comprise a user interface subsystem. The processor subsystemmay communicate with the user interface subsystemvia internal data communication. The user interface subsystemmay be configured to, during operation of the system, enable a user to interact with the system, for example using a graphical user interface. The user interface subsystemis shown to comprise a user input interfaceconfigured to receive user input datafrom a user input deviceoperable by the user. The user input devicemay take various forms, including but not limited to a computer mouse, touch screen, keyboard, microphone, etc.shows the user input device to be a computer mouse. In general, the user input interfacemay be of a type which corresponds to the type of user input device, i.e., it may be a thereto corresponding type of user device interface. The user interface subsystemis further shown to comprise a display outputconfigured to provide display datato a displayto visualize output of the system, such as the aforementioned visualization of the lower-dimensional space and other types of visualizations. In the example of, the display is an external display. Alternatively, the display may be an internal display.

100 100 In general, the systemmay be embodied as, or in, a single device or apparatus, such as a workstation, e.g., laptop or desktop-based, or a server. The device or apparatus may comprise one or more microprocessors which execute appropriate software. For example, the processor subsystem may be embodied by a single Central Processing Unit (CPU), but also by a combination or system of such CPUs and/or other types of processing units. The software may have been downloaded and/or stored in a corresponding memory, e.g., a volatile memory such as RAM or a non-volatile memory such as Flash. Alternatively, the functional units of the system, e.g., the data interface and the processor subsystem, may be implemented in the device or apparatus in the form of programmable logic, e.g., as a Field-Programmable Gate Array (FPGA). In general, each functional unit of the system may be implemented in the form of a circuit. It is noted that the systemmay also be implemented in a distributed manner, e.g., involving different devices or apparatuses, such as distributed servers, e.g., in the form of cloud computing.

2 FIG. illustrates a result of a dimensionality reduction applied to clinical data. In this example, the clinical data comprises individual data instances, for example representing data of respective patients or exams, which are each representable as a feature vector in a 33-dimensional feature space. A manifold is assumed to exist in the 33-dimensional feature space which sufficiently represents the clinical data. In other words, it is assumed that feature redundancy exists and that the variance in the clinical data is on a lower-dimensional structure embedded in the high-dimensional space. The data on such a lower-dimensional structure may be represented in a lower-dimensional space, such as a 2D space, using a non-linear and manifold preserving dimensionality reduction technique (also simply referred to as a non-linear projection technique), for example using a so-called t-SNE algorithm.

2 FIG. 200 210 220 230 230 200 The result of applying such a t-SNE algorithm to the clinical data is shown in, in which a visualization of the lower-dimensional spaceis shown having two dimensions,labelled tSNE-1 and tSNE-2. In addition, clinical data pointsare shown which represent the higher-dimensional feature vectors of the clinical data in the lower-dimensional space. The dimensionality reduction may be such that clinical data pointswhich are nearby to each other in the 2D spacehave similar feature vectors. It will be appreciated that many alternatives exist to t-SNE, such as UMAP, ISOMAP, HSNE and A-tSNE, all of which are known per se. In case t-SNE is used as a dimensionality reduction algorithm, the so-called approximated-tSNE implementation of tSNE may be used for desktop applications or TensorFlow.js tSNE for web applications.

2 4 FIGS.-C 2 4 FIGS.-C In the example of, the clinical data is the training data which has been used for training the classification model. Such training data may be labelled, in that there exists a ground truth for the classification by the classification model. As shown in, this labelling may be visualized, e.g., by distinguishing between two categories as either a darker square or a lighter circle.

3 FIG.A 1 FIG. 2 FIG. 300 300 100 shows a visualizationof the classification and the classification uncertainty of the classification model in the two-dimensional space. Such a visualizationmay have been generated by the systemofby, for a dense and regular grid of synthetic clinical data points in the two-dimensional space of, applying the classification model to the interpolated feature vector associated with a respective synthetic clinical data point to obtain a classification, and by determining the classification uncertainty. In determining the interpolated feature vectors, the so-called KD-Tree algorithm may be used to search for nearest datapoints to be used in the interpolation. The interpolation itself may be any suitable weighted or non-weighted interpolation technique which is applied to the set of datapoints which is found using the KD-Tree algorithm, e.g., the KNN datapoints.

300 This visualizationmay be termed a ‘classification landscape’, and may be generated in an output-driven manner, in that the dense and regular grid of synthetic clinical data points may correspond to the pixel grid of the output image containing the visualization. Alternatively, any other suitable regular grid may be used, or an irregular grid or any other set of synthetic clinical data points.

3 4 FIGS.A-A As is also shown in the examples of, the classification may be visualized by selecting a hue for a respective pixel, while the classification uncertainty may be visualized by selecting a color saturation for the respective pixel. For example, areas of high confidence (or high certainty or low uncertainty) may be visualized with a high color saturation while areas of low confidence (or low certainty or high uncertainty) may be visualized with a low color saturation. It will be appreciated that alternatively, any other type of visualization may be used, as are known per se from the field of data visualization, including but not limited to using patterns instead of hue/saturation, using heat maps, using contour lines, etc.

3 FIG.A 3 FIG.A 310 312 300 320 It can be seen inthat there exist complex decision boundaries,in the classification, in that the decision boundaries are very high-dimensional and poorly representable in lower-dimensional space, which may be apparent in the visualization in various ways, such as having decision boundaries in very close vicinity of each other. Such complex decision boundaries may indicate poor generalization of the classification model, and hence may require careful judgement by the clinician. From the visualization, the clinician may also detect possible misclassificationsas the ground truth classification may mismatch the classification by the classification model, the latter being represented inby the underlying hue. Such misclassification may be a misclassification in the ground truth, e.g., in the form of an outlier, but may also be a misclassification by the classification model.

3 FIG.B 3 FIG.A 300 330 340 shows the visualizationof, and shows areas where the classification model has low confidence, being areaswhich are visualized with a low color saturation, and where the classification model has high confidence, being areaswhich are visualized with a high color saturation. In general, the classification uncertainty may also be referred to as classification certainty, referring to the complement of the classification uncertainty, or classification confidence or the like. Such classification (un)certainty may be determined in various ways. For example, for support-vector machines, the classification certainty may be determined as the distance from the decision boundary, while for random-forest classifiers, the classification certainty may correspond to the percentage of trees agreeing on the predictions, while for deep-learning based methods, the classification certainty may be derived from the entropy of the probability vector. Determining such a classification (un)certainty or confidence is known per se in data classification.

300 320 300 300 300 3 3 FIGS.A andB In general, the classification landscapeofmay show a global behavior of the classification model. As indicated above, misclassified pointsmay be seen over the landscape, and the shape of the classification boundaries may be revealed. At the same time, the classification landscapemay be generated in a classification model agonistic manner, in that no internal parameters of the classification model may be needed to generate the classification landscape. In some embodiments, new clinical data may also be shown in the classification landscape, e.g., as new clinical data points obtained by the aforementioned dimensionality reduction. Depending on the location of the new clinical data points in the classification landscape, a clinician may determine whether the output of the classification model can be trusted or not.

In general, the classification landscape may be generated for any type of classification model, including but not limited to support-vector machines, decision trees, random forest classifier, or deep-learning based classification models.

4 FIG.A 1 FIG. 300 100 360 350 illustrates a user selecting synthetic clinical data points in the visualizationof the two-dimensional space, for example using the user interface subsystem of the systemofand a thereto connected mouse, touch screen or the like. Depending on whether a single synthetic clinical data pointor two of such data pointsare selected, a different visualization may be generated.

4 FIG.B 400 360 430 420 400 shows a visualizationof an interpolated feature vector which may be provided in response to a selection of the single synthetic clinical data point. Here, the vertical axismay list the various feature vector components, such as for example sex, weight, height, blood type, etc., while the horizontal axismay show the feature vector values, such as for example ‘F’, ‘60 kg’, ‘170 cm’, ‘O-negative’. In this example, the feature vector is shown to comprise 33 features, e.g., from 0 to 32. Such a visualizationmay enable a user to draw conclusions on the relation between on the one hand the classification and/or classification certainty and on the other hand the feature(s) on which the classification is based.

4 FIG.C 410 350 430 422 410 shows a visualizationof a difference between two interpolated feature vectors which may be provided in response to the selection of the two synthetic clinical data points, being in this example at opposite sides of a decision boundary. Here, the vertical axismay again list the various feature vector components, while the horizontal axismay show the feature vector value differences. Such a visualizationmay allow a user to draw conclusions on the relation between a change in classification and the feature vector differences.

5 FIG. 1 FIG. 500 500 100 500 shows a computer-implemented methodfor applying a classification model to clinical data. The methodmay correspond to an operation of the systemof. However, this is not a limitation, in that the methodmay also be performed using another system, apparatus or device.

500 510 500 520 500 530 500 540 500 550 550 500 560 The methodis shown to comprise, in a step titled “ACCESSING CLINICAL DATA”, accessingclinical data comprising data instances which are each representable as a feature vector in a multi-dimensional feature space. The methodis further shown to comprise, in a step titled “ACCESSING CLASSIFICATION MODEL”, accessinga classification model configured to be applied to the feature vector to provide a classification of the respective data instance. The methodis further shown to comprise, in a step titled “APPLYING DIMENSIONALITY REDUCTION TECHNIQUE”, applyinga non-linear and manifold-preserving dimensionality reduction technique to all or a subset of the feature vectors to obtain a plurality of clinical data points in a lower-dimensional space. The methodis further shown to comprise, in a step titled “DETERMINING FEATURE VECTORS FOR OTHER DATA POINTS”, determiningfeature vectors for other data points in the lower-dimensional space by applying an interpolation technique to the feature vectors of the clinical data points, thereby obtaining synthetic clinical data points in the lower-dimensional space each having an interpolated feature vector. The methodis further shown to comprise, in a step titled “DETERMINING CLASSIFICATION AND CLASSIFICATION UNCERTAINTY”, for each synthetic clinical data point, applyingthe classification model to the respective interpolated feature vector to obtain a classification for the synthetic clinical data point, and determininga classification uncertainty of the classification. The methodis further shown to comprise, in a step titled “GENERATING VISUALIZATION OF CLASSIFICATION UNCERTAINTY”, generatinga visualization of the lower-dimensional space for display to a user, wherein the visualization comprises a visualization of the classification uncertainty in visual relation to the synthetic clinical data points.

500 5 FIG. It will be appreciated that, in general, the operations of the computer-implemented methodofmay be performed in any suitable order, e.g., consecutively, simultaneously, or a combination thereof, subject to, where applicable, a particular order being necessitated, e.g., by input/output relations.

6 FIG. 6 FIG. 600 610 600 The method(s) may be implemented on a computer as a computer implemented method, as dedicated hardware, or as a combination of both. As also illustrated in, instructions for the computer, e.g., executable code, may be stored on a computer readable medium, e.g., in the form of a seriesof machine-readable physical marks and/or as a series of elements having different electrical, e.g., magnetic, or optical properties or values. The executable code may be stored in a transitory or non-transitory manner. Examples of computer readable mediums include memory devices, optical storage devices, integrated circuits, servers, online software, etc.shows an optical disc.

Examples, embodiments or optional features, whether indicated as non-limiting or not, are not to be understood as limiting the invention as claimed.

It should be noted that the above-mentioned embodiments illustrate rather than limit the invention, and that those skilled in the art will be able to design many alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses shall not be construed as limiting the claim. Use of the verb “comprise” and its conjugations does not exclude the presence of elements or stages other than those stated in a claim. The article “a” or “an” preceding an element does not exclude the presence of a plurality of such elements. Expressions such as “at least one of” when preceding a list or group of elements represent a selection of all or of any subset of elements from the list or group. For example, the expression, “at least one of A, B, and C” should be understood as including only A, only B, only C, both A and B, both A and C, both B and C, or all of A, B, and C. The invention may be implemented by means of hardware comprising several distinct elements, and by means of a suitably programmed computer. In the device claim enumerating several means, several of these means may be embodied by one and the same item of hardware. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage.