Expressing uncertainty in neural networks for production systems

Static analysis

For tasks such as condition-monitoring or anomaly detection, only the signal values x∈Rn at some point in time t are used, i. e., the analysis uses a static feature vector only [3]. Such anomalies are also called point anomalies. Here, the assumption is that no information is coded in the sequence of values and all necessary information is contained in the current signal values.

In this paper we will use neural network-based autoencoders (AE) for this task. Figure 1 shows such a network structure.

Figure 1

The general solution architecture for the static anomaly detection.

First, the static input feature vector x is transferred into a latent representation, normally corresponding to a low dimensional space—this is called the encoding phase. From this reduced representation, the original signals are reconstructed as xˆ—this is called the decoding phase. During the learning phase, the neural network is trained in a way that minimizes a so-called loss function, a function which quantifies how close the true value x and the estimate xˆ are. During operation, this can also be a measure for the normality of the new data point.

Dynamic analysis

A totally different situation arises when important information is coded in the sequence of signal values over time. Such anomalies are also called contextual anomalies [4]. Given historical data X={x(t−i),…x(t)} and a new data vector x(t+1), the probability is computed that x(t+1) is the logical continuation of the series.

Figure 2 shows the usual solution approach for such a dynamic analysis.

A neural network f is trained during a training phase on a time series {x1,…xp} so that xˆt=f(xt−i,…xt),i∈N is a good prediction for x(t+1), i. e., the distance ||xt+1,xˆt+1|| is minimized. During operation, this is a measure for the normality of the new data point xt+1.

Figure 2

The general solution architecture for the dynamic detection.

In both cases, the learned models must estimate the uncertainty of the prediction. Figure 3 visualized this: In several ML-approaches (Figure 3, left), a Euclidean distance ||xˆ,x||2 is used to identify unusual data points. Please note that this distance incorporates a measure of similarity, not a measure of uncertainty about the expected similarity.

Figure 3

The role of uncertainty in anomaly detection.

On the right hand side of Figure 3, the estimate xˆ comes with a measure of uncertainty, so the degree of similarity between xˆ and x can be quantified. In this work, we model the uncertainty as a variance of a normally distributed random variable xˆ.

This paper deals with two main research questions (RQ):

Research Question 1. How can we, for a dynamic anomaly detection, encode the uncertainty of the prediction xˆt+1 into the metric ||xt+1,xˆt+1||?

Research Question 2. How can we, for a static anomaly detection, encode the uncertainty of the prediction xˆt into the metric ||xt,xˆt||?

3.1 Dynamic analysis

Time-series data extracted from CPPSs in most cases contain a certain amount of noise, either caused by the measurement itself, or variances in underlying processes. In cases where this kind of uncertainty is not constant, i. e., heteroscedastic, a prediction is difficult, as it will likely show similar noise. It is hardly possible to distinguish between certain and uncertain predictions xˆi by monitoring a fixed distance to the sensor value xi, as this distance can be shifting with x. Referring to Figure 3 in the introduction of this paper, a prediction with a large Euclidean distance to the ground truth, can be certain in case the normal behavior of the CPPS shows a high variance in this specific operational mode.

Figure 6

Example for homo- and heteroscedastic uncertainty.

Figure 6 shows the same signal added with homoscedastic and heteroscedastic uncertainty. The heteroscedastic case shows Gaussian noise where σ2 is a linear function of x. This dependency can be seen in similarity to the prediction of xˆ: It is another function that has to be learned from the dataset and thus be predicted.

In case of comparing new samples to the predictions made, the learned confidence interval helps to classify whether the new samples were expected as shown in Figure 7. The shaded area shows a 2σ confidence interval. This will later be used for anomaly detection.

Figure 7

Example for prediction in context of uncertainty. For a signal containing a normal distributed heteroscedastic noise, the mean xˆ1 as well as the standard deviation σ is predicted.

To predict data points from a time-series of multiple sensors of a CPPS, a recurrent neural net (RNN) architecture can be applied. An RNN uses internal states to encode temporal information between the data points. Similarly to an autoencoder (Figure 1), these internal, i. e., hidden, states of the RNN have to be decoded into a prediction of xˆt+1. For this task, a feed-forward architecture containing one or more layers is used. This architecture implies that the sensor signals are generated from a latent space, i. e., hidden states, inside the CPPS.

Figure 8

Neural network architecture for dynamic data using an LSTM layer as an RNN.

For our approach, we use a Long Short-Term Memory network (LSTM), which is a common type of RNN [16]. As depicted in Figure 8, the LSTM layer takes the time-series {xt,⋯,xt−n} as input and encodes a hidden representation ht of it. To incorporate a measure of uncertainty, the fully connected layer decodes the hidden states into a prediction for the sensor values xˆt+1 and their standard deviation σt+1. The following part will first explain how the LSTM decodes a time series into a hidden representation followed by a derivation of a loss function that incorporates the σt+1 output to enable the presented neural network to be trained.

The zoomed representation in Figure 8 (dashed lines) shows an LSTM cell architecture as used in [20]. The central element that characterizes the family of RNNs is the use of a memory cell [21]. In case of an LSTM it is realized as a cell state c which models internal states of the CPPS. The interconnection of the cells through c and h enables information to flow forward from past time steps to the current time step t. The LSTM cell at time step t takes the input samples xt, the previous hidden representation ht−1, and the previous cell state ct−1 into account to build the cell state ct.

Gates control the information flow inside the LSTM cell and enable the network to take important information from a certain point in time into account, e. g., the beginning of a transient phase, and neglect other points in time, e. g., stationary phases. The gates are realized as neural network layers according to Equation (6).

The matrices W,U,V of every gate are weights of the NN layer and thus learned during the training phase of the network. The sigmoid function limits the gates values to an interval of [0,1]. Depending on their particular value, information is neglected or accepted. In case of the forget gate ft=1 and the input gate it=0, the cell state ct is calculated only from the previous cell state ct−1. In the opposite case where ft=0, it=1, only the hidden representation ht−1 and the input samples xt are taken into account. Before they pass the input gate it, they are weighted by u, w which are learned during the training process.

For cases where the gates take values 0<α<1, ct is calculated from new inputs as well as the cell state of the previous cell. The tanh multiplication acts as a regularization as it restricts the cell states within an interval of [−1,1]. Which features of ct contribute to the hidden representation vector ht is controlled through the output gate ot.

The dimensions of ht and ct determine the number of internal states. If it is chosen less than the number of sensors (dim(xt)), the LSTM is forced to learn a reduced latent representation, similar to the decoder part of the autoencoder in Figure 1. To train the shown network by reducing the prediction loss, the loss function has to incorporate σt+1. This can be achieved by expressing the loss function as a maximum likelihood approach considering three underling assumptions:

1. All relevant hidden states ht can be learned from the data.

2. Gaussian distribution for covariance of the error xˆt+1−xt+1 (e. g., white noise).

3. Knowing the hidden states ht, the remaining noise on each sensor xti is independent (e. g., white noise).

We enforce the NN to compute xt+1i and σt+1i, with regard to the maximum likelihood, to describe the data by expressing the loss function as the negative log maximum likelihood of Equation (7). This results in the integration of the standard deviation into the mean square error loss function (MSE) as proposed by [4]. By dividing through σt+1i, and the log term as a penalty term in Equation (8), the maximum-likelihood-error loss (MLE) is formed.

While using the standard MSE loss function for the combination of an LSTM network followed by a fully connected layer cannot properly model noise in a signal, a leftover loss at the end of the learning phase will be present. The MLE loss forces the NN to reduce this leftover loss by learning σt+1i that fits best to the noisy sensor data. Mapping the hidden representation of the LSTM into σt+1i and xˆt+1i by a fully connected layer can be interpreted as mapping to a specific internal state of the CPPS into prediction and standard deviation. In case of heteroscedastic uncertainty as shown in Figure 6, the linear function changing the variance, as well as x2=f(x1) itself is learned.

As the introduced NN architecture is able to predict the standard deviation σt+1 together with the sensor values xˆt+1, a confidence interval as shown in Figure 7 can be applied to the prediction. It acts as a measure of uncertainty for predictions made, and can be used to make further decisions, e. g., classifying measured data points to detect anomalies. Furthermore, in control loops a high standard deviation in certain operational modes can be taken as mistrust in the NN model, leading to using a backup model with lower performance [22].

In [23] a NN similar to Figure 8 is used to detect anomalies on two artificial and one real world dataset. With reference to Figure 7, a fixed decision border together with an MSE loss function is compared to the MLE loss function in Equation (8) using an automatically generated uncertainty interval |xt+1i−xˆt+1i|>k·σt+1i.

Table 1 shows the F1 scores together with the metric used. Artificial dataset 1 contains a sine and a saw-tooth signal with Gaussian noise of different intensity levels added to each signal. Six anomalies are added to the sine signal as shown in Figure 9.

Figure 9

Anomalies in sine signal of DS1: (1) Higher Noise, (2) Higher Amplitude, (3) Higher Frequency, (4) Lower Amplitude, (5) Offset, (6) No Signal.

Artificial dataset 2 contains ten signals that are generated from a latent random-walk on surface in a three dimensional room which is mapped into 10-D space through a sinusoidal function. After a phase of normal behavior, an error of two intensities is added to the latent space. The results shown refer to the high intensity error, as this case can be properly identified as an anomaly. For low intensity errors both metrics show poor results. The real world dataset consists of 231 days of operation of an Ion Mill Etching System. As several recipes for products are operated, the most frequent one (recipe No. 67 with 51 % of the time) is used. The degeneration processes of a pressure signal is identified and marked as anomalous before this leads to non-acceptable product quality.

For all datasets it is shown that the learned uncertainty interval of the MLE Approach is at least equal or better than a fixed border for every sensor. In DS 1 the anomalies, added to one of two signals, are detected very well. DS 2 shows similar results for MSE and MLE. The manipulation of the latent space for the anomalous case results in a well detectable anomalous system behavior as all sensor signals show anomalous behavior. Similar to DS 1 the IMES DS shows good results for the MLE metric. This can be explained as many sensors behave normal and only one sensor shows anomalous behavior. The choice of metric also shows that 2σ is a reliable hyperparameter for anomaly detection. This reduces the amount of setting manual thresholds, and makes the method suitable for practical CPPS integration.

3.2 Static analysis

For the static analysis case, we use a Denoising Autoencoder [8]. We compare this result with a standard Autoencoder, and a Restricted Boltzmann Machine to include an energy-based approach.

As already shown in Fig. 1, the autoencoder is a stack of layers of neurons, with at least one hidden layer, consisting of an encoder and a decoder architecture. We used an altered approach, i. e., a Denoising Autoencoder (DAE). The difference to the vanilla autoencoder depicted in Section 1 is limited to the input. We use x˜t∈Rn which represent damaged data points. The objective of the DAE is to reconstruct the unaltered data distribution x for the training and validation phase. Through a stochastic function q, i. e., Gaussian noise function, the initial data input x is transformed to a manipulated version x˜ which serves as input to the encoder-decoder architecture. With the encoder network f, the altered input x˜ gets encoded into a latent representation z, before decoded to the output y with the decoder network gθ′.

Figure 10

Representation of the basic principle of a Denoising Autoencoder.

The data used are generated through a random walk algorithm, in reference to Section 3.1, and normalized in accordance to its mean and variance. In total 50 sensors were simulated, with 50,000 signal values each. The architecture consisted of 3 encoder layers reducing in terms of number of neurons, the latent representation layer, consisting of 10 neurons, and 3 decoder layers with increasing number of neurons back to the same number of outputs as the initial number of input neurons. Before the data go into the encoder, Gaussian noise is being applied. We applied noise only in the training and validation phase. We used the Adam optimizer with a learning rate of 10−3 and a weight decay of 10−4, and trained for 100 epochs with a batch size of 50. We minimize the reconstruction error L(x,y) as a mean squared error (MSE) through training of the network. Additionally, an equal size of validation dataset, and test dataset, consisting of normal and anomalous data points, respectively, are used for the evaluation.

Categorizing data points as normal or anomalous is estimated through the MSE, serving as the distance between ground truth and predicted values, and a threshold that is set at the standard deviation of 1σ above the mean of the validation error. Predictions going beyond the threshold are viewed as anomalies, with the Euclidean distance serving as an estimate for uncertainty. One of the sensor’s distance error is depicted exemplarily in Figure 11.

Figure 11

Visual representation of one exemplary sensor with the distances of prediction and ground truth of the normal (orange; validation dataset) and anomalous (blue; test dataset) data points. Prediction after the training of the DAE.

We found that adding Gaussian noise leads to higher F1 scores. By using the noise-induced input x˜, we achieve robustness to a partial manipulation of the input data. The manipulation also serves generalization purposes through the augmentation of training data due to the stochastic transformation of the input [8]. We used different levels of gaussian noise, and found 20 % to be a good estimate. The results are based on the top three averaged F1 scores. For the confusion matrices in Figure 12, we used the average of the respective results. Additionally, we show in Table 2, that the RBM on the same data performs better than both autoencoder. This goes back to the inherent ability of energy-based methods to learn probability distributions from an unknown data distribution, referring to Section 2.

Figure 12

(a) Evaluating the confusion matrix of the DAE shows a robust detection of anomalies. 93.81 % of all anomalies were detected, whereas only 83.01 % of non-anomalous data points were classified as such. This means around 16.99 % detections of anomalies are in fact wrong and only 6.19 % of detected anomalies are in fact non-anomalous. (b) It is noticeable that the vanilla autoencoder, i. e., without denoising ability, performs significantly worse in detecting normal data, i. e., the false positive rate is significantly higher. The true positive rate shows no significant difference to the DAE.