NGUYEN QUOC PHONG ET AL: "GEE: A Gradient-based Explainable Variational Autoencoder for Network Anomaly Detection", 2019 IEEE CONFERENCE ON COMMUNICATIONS AND NETWORK SECURITY (CNS), IEEE, 10 June 2019 (2019-06-10), pages 91 - 99, XP033599405, DOI: 10.1109/CNS.2019.8802833
LUCA GILLI ET AL.: "Uncertainty quantification for criticality problems using non-intrusive and adaptive polynomial chaos techniques", ANNALS OF NUCLEAR ENERGY, vol. 56, 2013
| CLAIMS 1. A computer-implemented method for generating local explanations and error estimates of a predetermined machine learning module configured for executing a predetermined supervised machine learning model, which is adapted to receive input query data and to output corresponding prediction data, comprising the steps of: equipping said predetermined machine learning module with a variational autoencoder, VAE, including an encoding module and a decoding module; inputting to the encoder module of the VAE the input query data and the corresponding output prediction data, the encoder module being configured to generate encoding data in a latent space that depend on both the input query data and the output prediction data; generating a plurality of synthetic perturbations to said encoding data by adopting a predetermined sampling strategy around each encoding data; inputting said plurality of synthetic perturbations to the decoder module together with said output prediction data, the decoder module being configured to generate synthetic input query data; providing said synthetic input query data to the machine learning model to obtain corresponding synthetic output prediction data; and generating output explanation and error data based on the encoding data in the latent space and synthetic input query data and output prediction data. 2. The method according to claim 1, wherein said plurality of synthetic perturbations to said encoding data are selected within a predetermined distance of the encoding data corresponding to input query data based on the probability distribution in the latent space. 3. The method according to claim 1 or 2, wherein said predetermined sampling strategy includes generating perturbations that are normally distributed around each encoding data element, each element of the encoding data being represented as where zo and zi are the first two coefficients of a Polynomial Chaos Expansion of the element of the encoding data, is a random variable defined by a standard normal distribution and Ti is a orthonormal polynomial basis belonging to the Wiener-Askey scheme, wherein the output prediction data are expressed as where is a vector containing the random variables used to represent the encoding data, N is the number of polynomial basis, and a are the solution of a n-dimensional integral 4. The method according to claim 1 or 2, wherein said predetermined sampling strategy is a deterministic sampling that includes the following steps: encoding an input query data; generating a perturbation of the encoded data leading to a synthetic input query data so that the norm between the input query data and the synthetic input query data is minimized; expressing the encoding data of said synthetic input query data using a first order Polynomial Chaos Expansion, PCE, and the output prediction data as n-th order truncated PCE; calculating the truncated PCE coefficients of the output prediction data by using the standard deviation of the output prediction data as a convergence criterion. 5. The method according to any one of the preceding claims, wherein generating output explanation data includes: providing representative input query data to the machine learning model; calculating representative encoding data of said representative input query data; encoding training input query data of the machine learning model to generate corresponding training encoding data in the latent space; calculating the distance in the latent space between said representative encoding data and said training encoding data; and comparing said distance with a specified distance threshold; and highlighting, for each input query data, the most similar training input query data when said distance is less than a predetermined distance threshold. 6. The method according to any one of claims 1 to 4, wherein generating output explanation data includes: providing a surrogate interpretable model of the machine learning model; training said surrogate interpretable model with said synthetic input query data and output prediction data; and if said surrogate interpretable model is a decision tree, said output explanation data is the decision path associated with said input query data within the decision tree. 7. The method according to any one of the preceding claims, wherein generating error data includes: storing input query data and corresponding encoding data over time, associated with a related time stamp; acquiring and storing a related feedback of True Positive Rate of output prediction data issued by the machine learning model based on said stored input query data and corresponding encoding data; receiving a current input query data and calculating the current encoding data corresponding thereto; retrieving a predetermined number of encoding data among the stored encoding data having the smallest Euclidean norms with respect to the current encoding data; and generating error data if the average Euclidean norm of the retrieved encoding data with respect to the current encoding data is greater than a specified norm threshold. 8. A decision support system for generating local explanations and error estimates of a predetermined machine learning model executed in a predetermined supervised machine learning processing module which is adapted to receive input query data and to output corresponding prediction data, the decision supporting system including a variational autoencoder associated to the machine learning module, said variational autoencoder including an encoder module and a decoder module, the encoder module being coupled to the machine learning module to receive the 20 input query data and the corresponding output prediction data and configured to generate latent space data, the decoder module being coupled to the encoder module to receive said latent space data and to the machine learning module to receive the corresponding output prediction data, the decoder module being configured to generate synthetic input query data, the decision support system being configured for performing a method for generating local explanations and error estimates of a predetermined machine learning module configured for executing a predetermined supervised machine learning model, according to any one of claims 1 to 9. 9. A computer program or group of computer programs that may be executed by a processing system, comprising one or more code modules for implementing a method for generating local explanations and error estimates of a predetermined machine learning module configured for executing a predetermined supervised machine learning model, according to any one of claims 1 to 9. |
Creating the mapping between input and output is usually a challenging task as the relationship between the former and the latter is usually highly complex and contains various forms of noise. The job of the data scientist is to find the best algorithm to perform the mapping task (for example a neural network), to train it and to validate it against a number of metrics (e.g. Accuracy, Area Under ROC).
Having found an optimal function y = f(x) is however only the first step towards the creation of a useful automation tool.
First, the algorithm needs to be "packaged" into a production ready system for example by creating a RESTful API to allow the user to query the model. As an alternative, the algorithm is "packaged" into a system for controlling an industrial plant or machine whereby input queries are automatically generated by the system when investigating the response of the industrial plant or machine to possible operating conditions. Second, a production model tends to be affected by the following issues:
Lack of transparency in the relationship between output and input. If the model is used within a Decision Support System the end user needs to understand why the algorithm came up with a certain recommendation.
The functional mapping performed by the model is not going to perfectly represent the data used in the training process. The model is not infallible and the user should be able to understand when the model is highly likely to fail and when it needs to be bypassed.
The model might become outdated with time as the historical examples used to train it might not accurately represent reality anymore. An automated decision system should be able to understand when its underlying model is becoming inaccurate and when it needs to be retrained against new data.
These are some of the most important aspects that need to be considered when developing a machine learning based tool. Tackling them means increasing trust in the tool itself and can determine its effective usefulness.
The aim of the present invention is to provide a solution that avoids the drawbacks of the prior art. Particularly, an aim of the present invention is to improve the deployment of machine learning models by generating efficient local explanations and robust error estimates for a given, unknown machine learning model in an Al-based predictive processing and control system, e.g. a decision support system employed in the control of industrial processes. More specifically, the aim of the invention is to enrich the information generated by each query by allowing a local explanation around the query point and to perform robust uncertainty quantification about the prediction reliability.
According to the invention, this aim is achieved by a computer-implemented method for generating local explanations and error estimates in machine learning models, having the features claimed in claim 1.
Preferred embodiments are defined in the dependent claims, whose content is also to be considered an integral part of the present description.
Further subjects of the invention are a system for generating local explanations and error estimates in machine learning models, as well as a computer program implementing said method, as claimed.
In summary, the solution proposed by the invention combines techniques from the fields of probabilistic programming and interpretable machine learning.
The invention is based on the use of a variational autoencoder, composed of both an encoder and a decoder, that is trained to minimize the reconstruction error between the encoded- decoded data and the initial data, to project the input data to a given machine learning model into a latent space with reduced dimensionality to discriminate between different queries in an unambiguous manner. The variational autoencoder is chosen because of its probabilistic generative property, i.e. the variational autoencoder is adapted to learn the probability distribution over the input space and create plausible perturbations within a predetermined distance of the original input data in the latent space based on said probability distribution with the aim of getting a local explanation around the query point and to perform robust uncertainty quantification about the prediction reliability.
Particularly, according to the invention, also the output of the machine learning model is used as a conditional variable to train the variational autoencoder at the encoder side, which is now called a conditional variational autoencoder, and to condition the generative process thereof in the latent space at the decoder side.
The invention allows to put an existing machine learning model into production following the paradigms of trustworthy Al which have been recently summarized in the EU report presented in the paper "On Artificial Intelligence - A European approach to excellence and trust", White paper, Brussels, 19.2.2020 COM(2020) 65 final.
The invention may be implemented in a computer environment (packaged for example in a Docker container) that allows to import an existing machine learning model and to generate a querying mechanism for such a model in the form of a RESTful API. The existing model can be any supervised machine learning architecture as represented in Figure 1.
Advantageously, the method and system according to the invention are designed to produce for each new query Xi the following information: an anomaly score to measure the likelihood of Xi belonging to the same distribution of the training data Xt: a high anomaly score will warn the user that the model was not trained to map Xi, yi; an explanation about the black box response yi, by building a local surrogate; a robust estimate about the reliability of the answer, by calculating an error probability for the specific query. Moreover the API is also designed to: allow the user to give real time feedback over the correctness of single or groups of predictions; and perform a real time monitoring of the model performance.
Further functional and structural characteristics and advantages of the present invention are set out in the detailed description below, provided purely as a non-limiting example, with reference to the attached drawings, in which: figure 1 shows a prior art exemplary schematic diagram of a supervised machine learning model; figure 2 is a schematic diagram of the application of a variational autoencoder to a machine learning model, according to the invention; figures 3a and 3b show a number of exemplary diagrams of input data distributions; figure 4 shows an example of a neural network architecture of the encoder of figure 3; figure 5 is a schematic diagram of the runtime architecture of a supervised machine learning model equipped with a variational autoencoder, in a system for generating local explanations and error estimates in the machine learning model according to the invention; and figure 6 is a schematic diagram of a system for generating local explanations and error estimates in a machine learning model according to the invention.
Having regard to the known architecture of the supervised machine learning model of figure 1, a system for generating local explanations and error estimates in said machine learning model according to the invention is based on equipping said machine learning model with a variational autoencoder, as depicted in figure 2.
An exemplary decision support system generally indicated 10 includes processing means 12 configured for executing a predetermined supervised machine learning model based on input query data X 14 which results in an output prediction data Y 16. Said processing means 12 may be considered as a machine learning module. The decision support system may be used by an automated control system or by a human operator, e.g. for controlling a machine, an industrial plant, an industrial process or a business management system. A variational autoencoder is associated to the machine learning model and includes an encoder module 20 coupled to the processing means 12 to receive in input both the input query data 14 and the corresponding output prediction data 16. The encoder module 20 is configured to generate latent space data z 22 that depends on both the input query data and the output prediction data. The latent space data z are provided as input to a decoder module 24, a further input of the decoder module being coupled to the output prediction data 16. The decoder module 24 is configured to generate synthetic input query data X’ 26.
In the invention, the first key component that is introduced in order to define the overall system is a generative model used to model the probability distribution of the problem data. In general, the data used to train a machine learning algorithm is distributed according to a certain probability distribution P(x).
For example, an automated loan approval system is trained by using historical examples coming from the outcome of loans given to past customers. This customer data is distributed according to certain probability distribution, i.e. one could calculate the average age or the probability of having a higher annual salary given the customer age and profession. Figure 3 a shows a practical example of probability distributions for a dataset describing individuals according to a number of categorical and numerical features.
Another example is given in figure 3b, where a system for controlling an industrial process is trained by using historical examples coming from the outcome of sensors, respectively indicated sensor 1, sensor 2 and sensor 3 and machine settings. The sensor data and machine settings are distributed according to certain probability distributions.
Defining P(x) quantitatively is extremely challenging, and therefore a generative approach is considered. The data x are considered to be generated by a hidden variable z, or probabilistically the conditional probability P(x|z) may be calculated. Generative models are usually used to sample plausible realizations of x starting from a simplified random variable z. Ideally this random variable should represent hidden generative concepts, for example the social status of an individual or a particular operational regime of a machinery. The objective of a generative model is to represent the original data described by potentially complex probability distributions P(x) (as the ones shown in Figure 3) using a more compact representation, for example a normal distribution. One of the most effective ways to obtain this kind of generative distributions is by using a Variational AutoEncoder (VAE). A VAE is based on an encoding network configured to compress an input data space into a latent space P(z|x), and on a decoding network to model the desired generative distribution P(x|z).
According to this kind of architecture, the invention is based on associating with the machine learning model a VAE customized by conditioning the encoding and decoding processes to the outcome of the machine learning model. This means that the latent representation z is not only conditioned to the input x but also to the outcome of the machine learning model y=f(x), or probabilistically P(z|x; y). The same is done for the decoding network as well, P(x|z; y).
The encoder module 20 and the decoder module 24 are preferably implemented using neural networks. Figure 4 shows a practical example of an encoding architecture. The output of the black box (e.g. the softmax layer for classification problems or a normalized prediction value for regression ones) is used to condition the encoding process by stacking it to the input x.
Introducing the conditional dependency with respect to the machine learning model output allows to obtain a probabilistic characterization of its behaviour since the generative model is trained to generate plausible points x’ (plausible input data) given the output of the machine learning module itself. The machine learning model in the machine learning module 12 is not known and it may be referred to in the following as a black-box model.
The training of the VAE may be done in the usual manner, i.e. by minimizing the reconstruction error Xt - X' - where Xt is the training input query data vector and X’ is the synthetic (reconstructed) input query data - and by using a constraint for the random variable z in the form of a Kullback-Leibler divergence with respect to a normal distribution.
The variable y is presented as a categorical distribution for classification problems and as a gaussian variable for regression ones. Advantageously, the architecture described so far allows to: obtain a robust encoding of the training data. Robust means that the encoded euclidean distance in the latent space | |zi - Z2112 between two latent space data vectors is small for input query data vectors xi, X2 which are discriminated as similar by the black box model; generate plausible perturbations. The generative model can be used to create synthetic perturbations around a prediction point (input query data x) and obtain a set X’ of synthetic input query data x’. These perturbations are considered plausible because they represent points close to the prediction one that are distributed according to the probability P(x|y);
Detect anomalies. Given a prediction point (input query data) xi the reconstruction error | |xi - X’I | (2 can be used to determine whether the point (a new input query data) belongs to the same distribution used to train the VAE model.
The trained VAE model described above is used during runtime as described in Figure 5.
In Figure 5, live data represents a single or a batch of queries performed through the RESTful API or automatically generated by a system for controlling an industrial plant or machine. For each query data x the following actions are performed: a. The input query data x are passed to the machine learning module 12 (the black box model) to get prediction data y. b. These prediction data y are passed to the encoder module 20 of the VAE together with the originating input query data x to get encoding data z in the latent space. c. Starting from the encoding data z, a number of neighboring points (synthetic perturbations) Z’ = z+dZ are generated by adopting a predetermined sampling strategy. d. The set of points (latent space synthetic perturbation data) Z’ is used to generate synthetic input query data X' through the decoder module 24, which are passed to the same machine learning model (black box model) in the same machine learning module 12 or in an identical machine learning module 12' to obtain the corresponding output prediction data Y’.
The VAE module therefore generates, for each input query data x, a corresponding encoding z in the latent space and a number of synthetic pairs of input query data and output prediction data X', ¥'. This information may be used by an explanation and uncertainty analysis module in a system for generating local explanations and error estimates of the machine learning model, to generate additional information for the user or the control system.
A system for generating local explanations and error estimates of the machine learning model is schematically depicted in Figure 6.
The decision support system according to the invention 10 is depicted to receive in input live data as input query data x and to generate encoded data z in the latent space, and synthetic pairs of input query data and output prediction data X', Y'. The encoded data z in the latent space, and synthetic pairs of input query data and output prediction data X', ¥' are provided in input to an explanation and uncertainty analysis module 30, which is also configured to receive input encoded training data Zt generated by the VAE based on the training data of the machine learning model. The explanation and uncertainty analysis module 30 is configured to output explanation and error data E as will be explained below, and is coupled to a monitoring and feedback database 32.
Examples of synthetic queries are included in the following tables. The first row (grey background) represents an input data vector xi while the rest of the rows are the values generated by the VAE. These rows represent, as mentioned above, synthetic points x' realistically distributed. sex age education occupation
Female 24 Bachelors White-Collar
Female 29 Bachelors Sales
Male 22 Bachelors Admin
Female 34 Dropout Sales
Female 27 Dropout Sales sensor 1 sensor 2 setting 1 sensor 3
1.3 20 off 0.0
SUBSTITUTE SHEET (RULE 26) 1.1 22 off 0.0
1.5 18 off 0.0
1.4 19 on 0.5
1.4 19 on 0.5
In the following, a sampling strategy for generating synthetic perturbations Z’ = z+dZ from encoding data z, and corresponding synthetic input query data X’, is disclosed.
The generation of synthetic points or synthetic input query data X’ requires a sampling strategy around the encoded variable z. This strategy can be based on a purely random approach (by assuming that dZ is for example normally distributed) or on a deterministic one. If one considers dZ to be normally distributed one can think of the output of the blackbox model y’ as a random variable as well, distributed according to a certain probability density function.
Given this premise, the behaviour of y’ with respect to the stochastic variable z can be reconstructed by using a Polynomial Chaos Expansion (PCE) as described in Luca Gilli et al. "Uncertainty quantification for criticality problems using non-intrusive and adaptive polynomial chaos techniques", Annals of Nuclear Energy, Volume 56, 2013. Such an expansion can be used to reconstruct any random variable using an orthonormal polynomial basis 'Pi belonging to the Wiener-Askey scheme. This allows for example to represent each one of the elements of z with the first and second coefficient of the PCE as
In this case <^i is a random variable defined by a standard normal distribution.
Having expressed z using such an expansion leads to the following expression for y: where is a vector containing the same random variables used to represent z and N is the number of polynomial basis included in the expansion, which is a user's choice. The orthonormal properties of the polynomial family chosen to represent z and y allows to express the coefficients c, as the solution of a n-dimensional integral
This kind of integral can be numerically solved using an adaptive cubature rule. The solution process involves the collection of a number of points X’ together with the corresponding outputs Y’ which can be eventually stored to generate surrogate models and counterfactual explanations.
A preferred algorithm to perform the deterministic sampling includes the following steps: a. Given a point to explain X, find the corresponding encoding z. b. If ||X’ - X| is larger than a specified threshold, find Znew that minimizes this norm otherwise z new = z. This can be done using a number of gradient descent iterations: c. Express Znew using a first order PCE and y as n-th order (specified by user) truncated PCE. d. Calculate the expansion coefficients of y by using the adaptive cubature rules. In this step the standard deviation of y is used as a convergence criterion.
Generating the sampling points as cubature rules allow to reduce the number of queries needed to build meaningful information if compared to purely random sampling strategies. Furthermore, the points associated with the largest |y’ - y| can be used as counterfactual examples: searching in the latent spaces makes the generated points X’ close to the point to explain and more importantly more likely to belong to the training data distribution.
In the following, explanation techniques as adopted by the explanation and uncertainty analysis module 30 are disclosed.
The synthetic points generated by the VAE for each query can be used to augment the prediction output with additional local explanations and with real time uncertainty analysis. According to the invention, two types of local explanations may be provided: the first type is done by using representative examples while the second is generated using local surrogates.
The explanation via representative examples requires the encoding, performed at a setup level, of the training data, as depicted in Figure 6. Once this information is available, it is possible to calculate the distance ||z - ztrain||2 between each new encoded query z and the encoded training set ztrain. This represents an unambiguous metric (compared for example to calculating the distance when in presence of categorical data) that highlights, for each query, the most similar points inside the training dataset. These points can be presented to the user or the control system as pertinent examples or as criticism to the current prediction.
This kind of explanation is also enriched highlighting which parts of the input query data X make the model consider the prediction more or less similar to an instance from the training set. This is done by using the notion of Adjoint Perturbation Theory.
First, the difference between the latent representation of an input query data z and the representation of a training point considered to be similar ztrainj is expressed as d | |z - Ztrain_i||2
The objective is to understand how a small change in the input query data (prediction point) 5X affects the distance d.
According to the Adjoint Perturbation Theory a first order change in d can be expressed as the inner product between the perturbation 5X and the solution of an adjoint problem φ 0
If one considers an encoder architecture defined by a neural network with one hidden layer, the adjoint problem can be expressed as where W T ’ 2 is the transpose of the weight matrix defining the connections between hidden and output layers of the encoder, a 1 is the hidden layer activation for input X, and f is the first order derivative of the activation function.
With this notation one can consider the vector <p° to represent an "importance map": if the absolute value of one of its elements is high, a small change in the corresponding element of 5X will correspond to a large change in the distance between two latent representations. One can therefore use <p° to highlight important columns when defining similarities of differences between points.
Alternatively, the explanation via local surrogate is obtained by training an interpretable model Y = g(X) using the synthetic points (X', Y’) generated by the VAE module. As discussed in the previous section, these points can be sampled either randomly or deterministically. The interpretable model can be, for example, a decision tree or a lasso regression and can be used to generate explanations. A decision tree can be for example used to provide the user or the control system with local anchoring rules or counterfactual reasoning. The advantage of using the synthetic points (X*, Y’) lies in the fact that these points only contain plausible values for the current prediction, thus reducing the cardinality of the surrogate problem considerably.
In order to keep track of the performance of the machine learning model during time in order to monitor phenomena such as concept drift and covariate shift, the monitoring and feedback database 32 is provided.
Each single input query x is stored in this database, together with its encoding z. In order to be able to monitor performances, a user or a control system should also provide feedback regarding the accuracy, i.e. the True Positive Rate, of a number of queries over time.
A query mechanism may be designed to allow the user or control system to:
Give a feedback over a single query, based on a query ID number.
Generate a table of past queries that need to be labeled in order to have reliable monitoring metrics. Every time feedback is provided to the database, the decision support system 10 associates the corresponding query ID with a timestamp and stores the information in a separate database entity Zfeedback. When the generation of a subset of past queries that need to be labeled is required, the following two distances are calculated for each entity zi contained in Z.
The first distance is used to determine what is the closest point for which a feedback is available in the database 32, while the second one calculates a weighted feedback age (Ati is the time since a feedback was provided for the i-th entry in Zfeedback). In analogy to the classical exploration-exploitation tradeoff, a user will be able to decide whether to focus on giving feedback for "unseen" queries (presenting high di values) or to monitor the timedependent behaviour of "previously seen" ones (represented by the average feedback age d 2 ).
Once the number of feedback points is determined for each type, a subsample of points with the highest metrics is chosen by adopting a Latin Hypercube sampling strategy.
Once the database Zfeedback is populated the user will be able to obtain a real time error estimate for a new query xt This is done by using the following approach:
1. Given an input query Xi, calculate the corresponding encoding zi
2. Find the N points in Zfeedback with the lowest Euclidean (L2) norm with respect to zi, where N is a predetermined number
3. If the average L2 norm for this N points is higher than a specified threshold, raise a warning 4. Calculate the "success rate" on this set of N points.
The success rate can be calculated, quite straightforwardly, as the number of correct queries over the number N. Alternatively, the user will be able to obtain this information using a Bayesian approach: the N points obtained in step 2 can be used as observations inside the Bayes formula where the likelihood is a Binomial distribution describing how many correct predictions are expected in N trials given the prior belief in the model confidence.
If the prior is set as a Beta distribution modeling the confidence of the black box classifier for the N points, the posterior can then be analytically expressed as another Beta distribution. This posterior can be used to update the model confidence thus obtaining a more robust estimate of the expected precision associated with each live query.
Advantageously, the computer-implemented method and system architecture of the invention allows to put any existing machine learning model into production following the paradigms of “trustworthy Al”, by enriching each model queries with local explanations and error estimates and by defining an efficient way to provide model feedback during runtime.
This is done by using meaningful perturbations which reduce the search space that needs to be explored in order to find a local explanation and counterfactual examples. The search space reduction is very much problem dependent, however it can be as big as one order of magnitude. Moreover, using the deterministic sparse grid to calculate the coefficients of the spectral expansion reconstructing the variable z reduces the number of samples needed to build a meaningful local surrogate and to find counterfactuals by 60/70% (for a typical VAE architecture).
Similarities and differences between points are efficiently highlighted by means of the solution of the adjoint problem.
Moreover, using the encoded representation or latent space z allows for an efficient feedback mechanism as it represents an unique way to bin and cluster real time queries.
Finally, the uncertainty analysis approach that is implemented in the invention is a more refined alternative to standard calibration procedures as it allows for more refined results given different input types.
Naturally, the principle of the invention remaining unchanged, the embodiments and the constructional details may vary widely from those described and illustrated purely by way of non-limiting example, without thereby departing from the scope of the invention as defined in the appended claims.