Explainable neural network for anomaly detection

This application is related to the following commonly owned applications:

The following materials are incorporated by reference for all purposes as if fully set forth herein:

The technical field relates to computer security and explaining artificial intelligence decisions and more specifically to explaining neural network detection of an anomaly in an authentication request.

The subject matter discussed in this section should not be assumed to be prior art merely as a result of its mention in this section. Similarly, a problem mentioned in this section or associated with the subject matter provided as background should not be assumed to have been previously recognized in the prior art. The subject matter in this section merely represents different approaches, which in and of themselves can also correspond to implementations of the claimed technology.

An unfortunate characteristic of neural network classifiers is that they are considered a “black box” when it comes to their predictions. In other words, although a neural network can provide highly accurate classification of input, it is often difficult to know why the neural network decided that classification. When the neural network provides a result that is different than a security administrator's expectation, the black box nature of neural networks makes it hard for the administrator, who needs to take some action, to determine whether to trust the neural network's result over the administrator's own judgement.

The lack of neural network transparency can be a challenge when deciding how to act on detected anomalies in computer network authentication requests. It is desirable to use neural networks to detect anomalies in authentication requests due to their high accuracy. However, the lack of transparency prevents the administrator from making good decisions whether to override the neural network's findings.

Explanation tools exist that can explain supervised models to clients as part of the field of Explainable AI (XAI). For example, Google has a project called Vertex AI that purports to provide explainability. However, results often sacrifice accuracy in favor of explainability.

An opportunity arises for explaining an anomaly detected by an unsupervised encoder-decoder model while retaining accuracy of neural-network generated results. Improved results may ensue.

The technology disclosed relates to explaining an anomaly detected in an authentication request by an unsupervised anomaly detection model, such as an encoder-decoder model. The technology involves obtaining the unsupervised encoder-decoder model, the encoder-decoder model having been trained to detect anomalous authentication requests. The technology also involves obtaining an explanation model, such as a multi-layer perceptron (MLP) model, trained to provide MLP results similar to encoder-decoder results when the same features are provided to both the unsupervised encoder-decoder model and the MLP model. The encoder-decoder results serve as ground truth during the MLP model training in the disclosed technology, which also includes embedding authentication request features from a live authentication request into an embedding space to generate embedding features, and providing the embedding features to the encoder-decoder model. The technology further involves detecting, by the encoder-decoder model, that the live authentication request is anomalous, and based on the detecting of anomaly, providing the embedding features to an explanation tool using the MLP. The disclosed technology further includes obtaining, from the explanation tool, an explanation comprising at least an identification of the authentication request feature that contributed most to the detecting of anomaly.

Particular aspects of the technology disclosed are described in the claims, specification and drawings.

The following detailed description is made with reference to the figures. Example implementations are described to illustrate the technology disclosed, not to limit its scope, which is defined by the claims. Those of ordinary skill in the art will recognize a variety of equivalent variations on the description that follows.

One way to safeguard protected endpoints on a communication network is determining if an authentication request, which meets the base authentication requirements (e.g., a correct userID/password pair, or a valid bearer token), is made in an anomalous context. In one example of anomalous context, a user that usually authenticates from Alexandria, VA requests authentication from a different city. When an access request is made from an anomalous context, it is often prudent to require step-up authentication (e.g., asking security questions or asking the user to input a security code sent to her phone.)

Step-up authentication can be annoying to users, especially when computer network administrators know, or should know, that the user is expected to authenticate in that anomalous context (e.g., user works from home from Chantilly, VA one day a week). It is desirable to set network rules and policies to account for expected anomalies to maintain a high level of network security through step-up while avoiding unnecessary requiring step-up for predictable anomalous authentication requests. For example, if a network security administrator understands that the user is expected to generate authentication requests in a different city, the security administrator can set rules and policies accordingly.

Neural networks, a broad class of machine learning models that are amenable to training by either unsupervised learning or supervised learning, are well-suited for anomaly detection due to their high accuracy. Neural networks “train” a series of layers of weighted nodes and weighted edges to model optimal solutions in multi-dimensional solution spaces. However, it can be difficult to obtain accurate feature attribution from neural networks, so security administrators might not be able to determine whether the detected anomaly was based on an expected anomaly (different city), or an unexpected anomaly (e.g., different country). Without accurate feature attribution, it is difficult to craft responsive rules and policies about step-up authentication, thus foiling the goal of not frustrating users.

The disclosed technology involves an explanation model, such as Multi-layer perceptron (MLP) or 1D Convolutional Neural Network (1D CNN), trained to parallel the loss function (e.g., a Kullback-Leibler divergence) of an anomaly detection neural network that is trained to detect anomalies in use cases such as authentication requests. This technology converts an unsupervised training problem into a supervised training problem since the explanation model undergoes supervised learning based on the anomaly detection neural network. The technology permits highly accurate feature attribution while retaining high accuracy of anomaly detection during production.

Acronyms used in this disclosure are identified the first time that they are used. These acronyms are terms of art, often used in standards documents. Except where the terms are used in a clear and distinctly different sense than they are used in the art, we adopt the meanings found in security system environments. For the reader's convenience, many of them are listed here:

Next, we describe an architectural diagram for explaining an anomaly detected by an unsupervised encoder-decoder model, using a threat protection solution to supplement an authentication journey

Architecture

illustrates an architecture for explaining an anomaly detected by an unsupervised encoder-decoder model. Becauseis an architectural diagram, certain details are intentionally omitted to improve clarity of the description. The discussion ofwill be organized as follows. First, the elements of the figure will be described, followed by their interconnections. Then, the use of the elements in the system will be described in greater detail.

Systemincludes users, authentic customer network, threat protection solutionand internet. Usersinclude authentic usersand inauthentic users. Customer networkincludes access manager, authentication journey, protected endpointsand authentication history. Threat protection solutionincludes feedback engine, anomaly detection engineand explanation engine.

Usersare attempting to access protected endpointshosted on customer network. Usersinclude authentic user(who are supposed to have access to protected endpoints) and inauthentic users(who are not supposed to have access to protected endpoints).

Customer networkhas policies designed to admit only authentic usersand deny inauthentic users. Access managerenforces those policies by filtering between the authentication requests from authentic usersand inauthentic users. Access managerperforms the filtering by sending usersthrough authentication journeys. Although the example of systemillustrates access manageras part of customer network, in some implementations, access managercan also be hosted on the cloud. Examples of products suitable to serve as access manageris ForgeRock Access Management and ForgeRock Identity Platform. Other vendors can also have products providing user authentication and/or enforce policies related to user authentication, and those products fall within the ambit of an access manager.

Authentication journeyprovides a series of interactions with usersand based on those interactions, determines whether to facilitate access to protected endpoints. Authentication journeygoverns interactions such as username/password, bearer-token based SSO, new user registration and step-up authentication. Authentication journeycan also employ data of prior authentication attempts from authentication history storeto determine how authentication requests from usersshould be routed. Authentication journeycan also use feedback from threat protection solutionto help determine authentication request routing.

Authentication history storecan store information from one or more tenants into tables of a common database image to form an on-demand database service (ODDS), which can be implemented in many ways such as a multi-tenant database system (MTDS). A database image can include one or more database objects. In other implementations, the databases can be relational database management systems (RDBMSs), object-oriented database management systems (OODBMSs), distributed file systems (DFS), no-schema database, or any other data storing systems or computing devices. In some implementations, the gathered metadata is processed and/or normalized.

Protected endpointsare communication endpoints that the customer owning customer networkwishes to protect. Examples of protected endpointsinclude applications, services and resources.

Threat protection solutioncan host several services, amongst which is anomaly detection engine, that provide authentication-request related analysis to predict the risk that an authentication request originates from inauthentic userrather than authentic user. One example of a product suitable to serve as threat protection solutionis ForgeRock Autonomous Access. Other products, from other network security vendors, that can detect anomalous authentication requests fall within the ambit of this disclosure.

Feedback enginecan assemble and report feedback related to whether an anomaly was detected. Feedback provided by feedback engine includes at least a risk score generated by anomaly detection engineand can also include an explanation of the risk score generated by explanation engine.

Anomaly detection engineprovides a risk score that the access request, ostensibly from authentic user, originated from an anomalous context. Anomaly detection enginecan use a variety of approaches, that include one or more neural networks such as encoder-decoder neural networks, to determine the risk score. Anomaly detection enginecan use the neural networks as part of an ensemble. Implementations of anomaly detection engineusing an ensemble can include, but are not limited to, heuristic rules, other types of machine learning models such as cluster models (e.g., centroid clustering, density clustering, distribution clustering, hierarchy clustering) and regression models.

Explanation enginecan explain the result of anomaly detection engineby providing the features that contributed most to the result. The explanation can include a quantity of features. The quantity can be preset, or can be set on demand. The quantity can be set to a single feature, or set to a quantity greater than 1 (e.g., top 3 features).

Internet(aka “cloud”) is a network connecting computers that are remote from one another. Internetcan span the Earth or a single country.

While systemis described herein with reference to particular blocks, it is to be understood that the blocks are defined for convenience of description and are not intended to require a particular physical arrangement of component parts. Further, the blocks need not correspond to physically distinct components. To the extent that physically distinct components are used, connections between components can be wired and/or wireless as desired. The different elements or components can be combined into single software modules and multiple software modules can run on the same processors.

Moreover, this technology can be implemented using two or more separate and distinct computer-implemented systems that cooperate and communicate with one another. This technology can be implemented in numerous ways, including as a process, a method, an apparatus, a system, a device, a computer-readable medium such as a computer-readable storage medium that stores computer-readable instructions or computer program code, or as a computer program product comprising a computer usable medium having a computer readable program code embodied therein. The technology disclosed can be implemented in the context of any computer-implemented system, including a database system or a relational database implementation like an Oracle™ compatible database implementation, an IBM DB2 Enterprise Server™ compatible relational database implementation, a MySQL™ or PostgreSQL™ compatible relational database implementation, or a Microsoft SQL Server™ compatible relational database implementation or a NoSQL non-relational database implementation such as an Apache Cassandra™ compatible non-relational database implementation, a BigTable™ compatible non-relational database implementation or an HBase™ or DynamoDB™ compatible non-relational database implementation. Next, we discuss embedding input into an embedding space.

Embedding is the technique of representing data to points in a continuous space (“embedding space”), such that points in embedding space that are close together are semantically related. Embedding can encode categorical values into the embedding space such that algorithms designed to work on continuous variables (e.g., most machine learning models) can be applied to data sets where some or all of the features are categorical.

One example of an embedding algorithm is fastText. FastText was designed to create embedding spaces from literal words by using subword structures. Detailed discussion of fastText can be found in “Enriching Word Vectors with Subword Information,” herein “Bojanowski,” which is incorporated by reference in its entirety herein. The following discussion is an overview of fastText based on Bojanowski.

FastText builds a subword model for each word. To do so, fastText first pads each word within a corpus of word sequences with beginning and ending characters (in Bojanowski, these are represented by the characters ‘<’ and ‘>’. See Bojanowski at page 137). Then, fastText extracts n-grams within a set range of n from the padded words. Each n-gram of each word has an associated vector representation. Each word also has a vector representation. Using the example of the fastText process on the word “there,” and for n-grams of size G=3, the word is first padded in special characters, resulting in <there>. The n-grams of the word “there” are <th, the, her, ere and re>, and also includes the padded word itself <there>. To save memory, the n-grams are hashed into a limited set of buckets.

FastText scores pairs of words (target word and context word) by taking the summation of the products of each n-gram vector for a target word and a context word vector. More formally:

where s is the scoring function, w is a target word at position t, c is a context word that surrounds w, g is an n-gram of w, Gis the set of n-grams associated with w,

is the vector representation of g at position t and vis the vector representation of the context word. Scoring between words not only occurs between words and positive context words, but also uses samples of negative context words for each positive context word.

fastText trains the model using stochastic gradient descent on the negative log likelihood function:

where m is a negative example and Nis a set of sampled negative context words for w.

Although the process of implementing fastText is presented using the example of literal words, extending the example to categorical data would be understood by those skilled in the art. Referring to the ongoing example of anomaly detection in authentication requests, individual authentication requests can be characterized by features, examples of which include geographical country (e.g., “India”), geographical city (e.g., “Hyderabad), device (e.g., “Google”), model (e.g., “Pixel 3”) day of week (e.g., “Thursday”), part of day (e.g., “morning”), OS (e.g., “Windows”), OS version (e.g., “Windows 7”) and browser (e.g., “Firefox”). All of the example features are categorical data. A history of authentication requests can be treated as a text corpus, each request can treated as a sentence, and each feature value can be treated as a word.

Besides fastText, other examples of embedding algorithms include one-hot encoding, Principal Component Analysis, Singular Value Decomposition, Latent Semantic Indexing, Word2Vec, GloVe, Embeddings from Language and Model, Bidirectional Encoder Representations from Transformers and many more.

Once threat protection solutionhas performed embedding, the dimensions of the embedding space (“embedding features”) are used as features, instead of features from authentication request (“authentication request feature”) to train ML models during a training phase and as input to ML models during a production phase.

Next, we discuss an example encoder-decoder model, Variational Autoencoder, that can be used as part of anomaly detection engine.

illustrates a Variational Autoencoder usable as an anomaly detection model. VAEis example of an autoencoder encoder-decoder model which optimizes an approximation of the posterior (which can be intractable) using a Variational Bayes approach and is part of an ongoing example of an anomaly detection model usable by anomaly detection engineto classify authentication requests as anomalous or non-anomalous. VAEincludes embedded authentication request features X, encoder q(z|x), latent space Z, mean μ, variance σ, decoder p(x|z), reconstruction loss layer.

Bayesian inference is a class of techniques that update probability of a hypothesis as more evidence becomes available. A marginal likelihood is a distribution of observed data, represented as a likelihood function that is integrated over the space of the possible parameter values that could define a particular mathematical model. Sometimes, the marginal likelihood function includes an integral that is intractable. Variational Bayesian methods are a family of Bayesian techniques for approximating intractable integrals, such as Expectation Maximation (EM), or mean-field approximation. Sometimes, EM and mean-field algorithms also cannot be used.

VAEcan be used when the marginal likelihood function is intractable and known workarounds to the marginal likelihood function are also intractable. Detailed discussion of VAE can be found in “Auto-Encoding Variational Bayes,” (herein “Kingma”) which is incorporated by reference in its entirety herein. The following discussion is overview of VAE based on Kingma and the figure includes many reference characters similar to symbols used in Kingma. Authentication request features are embedded into n dimensions using fastText embedding to obtained embedded authentication request features x. In one example, n is one hundred. In other examples, n can be set to other values and this adjustment can affect training speed and/or model accuracy. Encoder q(z|x) uses embedded authentication request features x as input and predicts mean p and variance a of the logarithmic distribution in latent space for each feature. Since the marginal likelihood function can be intractable, VAEapproaches this problem by recognition model q(z|x) where z is an unknown distribution and x are discrete variables in that distribution. The recognition model, encoder q(z|x), has the role of the encoder model of in an AE. Decoder p(x|z) samples from the latent space distributions, based on mean p and variance a and generates a vector representing latent space Z. In the instant example, VAEperforms sampling with Gaussian sampling because authentication request features are continuous data. In some implementations, VAEcan also be used with the Bernoulli distribution when features are binary. Decoder p(x|z) attempts to recreate the input based on these samples.

VAEcan be used when the marginal likelihood function can be intractable and known workarounds to the marginal likelihood function are also intractable. Detailed discussion can be found in Kingma.

Continuing the description of VAE, the difference between the input and output is the reconstruction loss that is determined by reconstruction loss layer. During training, the reconstruction loss is used to adjust the parameters using the “reparameterization trick” during back propagation. During production, the reconstruction loss is used to estimate anomaly and contributes to the risk score. Libraries used to implement VAEare “TensorFlow” Python package and “Gensim.” An implementation of this technology used TensorFlow version 2.7.0 and Gensim 3.8.3. In the implementation, authentication request features from the VAE input layer were embedded into one hundred dimensions with fastText. In the implementation, he encoder and decoder each had two layers of one hundred nodes, and the latent space had fifty nodes. Next, is an example of a multi-layer perceptron neural network that can mirror the input and reconstruction loss of the VAE.