Automatic Error Classification and Root Cause

Determination while Replaying Recorded Workload

Data at SAP HANA
Neetha Jambigi Thomas Bach Felix Schabernack Michael Felderer
University of Innsbruck SAP SAP University of Innsbruck
Innsbruck, Austria Walldorf, Germany Walldorf, Germany Innsbruck, Austria
[email protected] 0000-0002-9993-2814 [email protected] 0000-0003-3818-4442
arXiv:2205.08029v1 [cs.SE] 16 May 2022

Abstract—Capturing customer workloads of database systems at any time for any new version. On a high level, the implicit
to replay these workloads during internal testing can be beneficial
test oracle, i.e. the verification whether the test has passed
for software quality assurance. However, we experienced that or not, is whether the replay executes successfully or if there
such replays can produce a large amount of false positive
alerts that make the results unreliable or time consuming to are any errors encountered during the execution. However,
analyze. Therefore, we design a machine learning based approach the recording also allows a fine-grained analysis of individual
that attributes root causes to the alerts. This provides several queries and performance. Capturing database workload for
benefits for quality assurance and allows for example to classifytesting purposes is also investigated by previous work [6]–
whether an alert is true positive or false positive. Our approach[9]. Other related fields are capture and replay approaches for
considerably reduces manual effort and improves the overall
quality assurance for the database system SAP HANA. We discuss GUI [10] and selective replays for program executions [11],
the problem, the design and result of our approach, and we but having the direct connection to the database provides
present practical limitations that may require further research. more information compared to a scripted testing type that is
Index Terms—DBMS, record and replay, error classification often applied for GUI testing. In addition, capturing customer
workloads has also benefits compared to artificial tests.
I. I NTRODUCTION While the idea of replaying recorded database queries sounds
As SAP develops software to manage business operations, simple in theory, it poses severe challenges in practice. Besides
the quality of the software is an important aspect. Customers the size of such replays, the costs to record and re-run them
demand that the software works as expected. Even more, in the or legal issues with respect to customer data that must be
context of business operations, customers expect that newer clarified, we also found that this approach shows false positives,
versions of a software provide at least the same functionality i.e. errors that in fact do not indicate regressions. Such false
as the version before, i.e. they expect that new versions do not positives are caused by non-deterministic behavior such as
contain regressions. This requirement is rather challenging for multi-threading and concurrency, sources of randomness, date
very large software projects as each modification to the source and time, missing functionality of the record tool, hardware
code can potentially lead to unexpected side effects that may specific code paths, influence of the record tool on the observed
not be predictable by humans due to the project size. behavior, missing data, different configuration settings. Another
In the case of SAP HANA, a database management system category of sources for false positives is external factors such
for enterprise applications developed by SAP [1], [2], the as infrastructure issues with the network, hardware, or the
source code consists of about 10 million lines of code and storage application. Altogether, we found in practice a wide
is changed about 200 times a day, making it impossible for range of reasons why false positives appear.
a single person to understand all parts and all modifications Given that false positives appear, we need a strategy to
at any point of time. Therefore, SAP maintains an extensive incorporate them in the test assessment. Assuming a replay
set of over 100 000 regression tests, i.e. tests that assure a shows a set of Pa errors, then we can classify all shown errors
new software version has no regressions [3], [4]. However, in two distinct subsets Pa = Pt ∪Pf where Pt are true positive
even with such an extensive test suite, customers may use the errors representing regressions and Pf are false positive errors,
software in a way that is not explicitly tested. For example, i.e. no regressions. It is desirable to keep Pf empty, but it
customers may use different table and data distribution, work- is in practice not possible to achieve this with a reasonable
load management, system utilization, system size or system effort. Analyzing each item in Pf by developers can easily
configuration. It is therefore important to test a new software create huge efforts if |Pf | ≥ 100 and may not be feasible if
version with customer workloads. One approach at SAP to |Pf | ≥ 105 . In practice, we observed |Pf | ≥ 106 for large and
achieve this is recording all workload from a customer system complex replays due to inherent characteristics and functional
over a certain time [5]. This workload can then be replayed limitations of the capture and replay.
 Handling a large set of Pf therefore requires an automated provided by the test results before releasing a new database
approach. As the items in Pf may have a common structure, a version to customers, it is essential to distinguish between the
machine learning approach seems to be well suited to classify true and false positives automatically.
the issues automatically. For this purpose, we identify a set During the replay of a captured workload, we collect the
of information about each error such as the related SQL result of the execution and a set of attributes for each query
queries, error codes, error messages, stack traces of errors (typically a SQL statement). Such a collection represents an
or information from bug tracker and project management tools. event. Operators are the quality assurance experts responsible
In the machine learning domain, we call such characterizing for the replay result analysis. Operators assess the collected
information features. Given a set of pre-defined error classifi- events to identify the root causes of the failures. In this section,
cations, the machine learning approach derives a mathematical we briefly describe how a rule-based approach supports the
model of how to conclude the error classification based on the assessment of such events and why such an approach has severe
features, i.e. the algorithm learns to classify. limitations. We then introduce our machine learning based tool
Furthermore, our approach does not only do a binary to automatically support the assessment, which we developed
classification between true positive and false positive, it also under the internal name MIRA - Machine Intelligence for
learns to associate the root causes for each item. With that, Replay Analysis.
we can group the true and false positives results into common
sets to further support the analysis of the test results. For this A. Limitations of the Rule-Based Approach
purpose, we also maintain a database of previously encountered Currently, at SAP, a solution for automating the replay error
and manually verified testing results. reasoning involves utilizing a combination of Request Id and
Other works also investigate the classification of errors Error Code. Request Id and Error Code are a part of the set
into true positives and false positives in the domain of of attributes collected for an event. There are Request Ids to
software defect [12]–[14], the analysis of logs [15], [16] or, identify failed replay events. There is an associated Error Code
more general, in outlier analysis [17]. In contrast, we utilize for every failed event in the replay. When this combination
additional information we gain from the database itself to for a specific capture appears for the first time, it gets tracked
improve the accuracy of our classification. Other work also as a failure. An operator assigns a tag value to the failures
utilizes the information of defect tracking tools and machine depending on the root cause, that indicates whether the failure
learning for automatically prioritizing bugs [18], detecting bug is a real issue or not. If applicable, a bug id is assigned as well.
duplication [19], [20] and classification of root causes for In a subsequent replay, if the tool detects a known tuple for
bugs [21]. In our case, we combine bug categorization with the (Request Id, Error Code) combination, the failed event is
classification supported by the additional information we gain labeled based on the tag of a matched entry. The tag assignment
from the database during the execution and from previous for failed events is done manually once for every capture to
classification that were verified by human operators. create a baseline to compare in successive replays.
This paper is structured as follows. Section II presents the This approach has the following limitations: i) Different
tool design and the industrial context of our approach. In captures could be workloads captured from the same system
Section III we describe our machine learning approach to error at different periods or may belong to different systems al-
classification and present a detailed account of different stages together. Request Ids are automatically generated for every
and results of evaluations in Section IV. In Section V we SQL statement when a workload is captured and is a unique
discuss the application in practice and feedback from the end identifier for an SQL query. It is used to identify a particular
users. Section VI contains our experiences, limitations, lessons query in replays of the same capture. For example, Query Q:
learned and open questions. We conclude in Section VII. SELECT col1, col2 FROM Table_A; has a Request Id 101 in
capture 1. In every replay of the workload from capture 1,
II. I NDUSTRIAL C ONTEXT AND T OOL D ESIGN the Query Q will have the same Request Id 101. However,
As described in section I, the development of SAP HANA if the same query is executed in different captures, it will be
utilizes a large set of unit and integration tests, but testing assigned a different Request Id which is valid only within the
with customer workloads is still a beneficial task for quality replays of the respective capture. When Query Q fails across
assurance. Replaying captured customer workload can help to two different captures it is tracked as Event A and Event B
identify software defects early that are not detected by other as shown in Table I. Even though failed events A and B have
tests types. In comparison to other scripted tests, there is no the same error codes, the tag of event A cannot be used to
artificial data or underlying assumptions made by test engineers assign a tag for event B due to different Request Ids. Therefore,
that can prevent the software system from malfunctions. even though similar errors occur, we cannot compare failures
Additionally, we can selectively record and replay critical across different captures. ii) There can be discrepancies in label
periods in the database system driven by specific application assignment between different operators for errors arising due to
processes and user behavior such as year-end closing, financial similar reasons. iii) In HANA, there are cases where the same
planning, or weekly reports. However, compared to artificial error code encompasses multiple types of errors. This problem
tests, the replay tests may be less stable. To ensure that test and becomes clear when the relevant error messages are analyzed.
quality owners can trust the overall product quality assessment Therefore, comparing merely error codes is not adequate.
 Table I: Rule-based approach However, this is not possible in our data as the attributes have
Capture Event Id (Request Id, Error Code) Tag Bug Id
no discernible ordinality within them.
The error messages in our data are unstructured and have
1 A (101, 202) Bug 1234
2 B (102, 202) Bug 1234 different lengths. We treat every error message as a document
and employ techniques like TFIDF [22] and Doc2Vec [23]
to create a vectorized representation of the error message of
B. The MIRA Tool an event. Concatenating the one hot encoded event attributes
with the vectorized error message further exacerbates the high
MIRA is developed and integrated as a microservice into
dimensionality problems.
the pipeline associated with automated capture and replay as
We apply the K-Nearest Neighbors (KNN) algorithm as
a part of the release and test process. Whenever a captured
the classification model in MIRA. KNN relies on a distance
workload is replayed, results are automatically forwarded to
metric to identify neighbors or the most similar instances of an
MIRA for the classification of failed replay events. The data,
instance to be classified. The curse of dimensionality is a well-
we train MIRA with, will change due to new software defects
known problem in machine learning and is known to adversely
being constantly introduced in the development process or
affect classifiers like the KNN [24], [25]. To address the
due to changes in the capture and replay tool itself. The
problem of encoding our data while avoiding high dimensions,
machine learning model will need to adapt to the changes of
we introduce a distance function detailed in Algorithm 1. We
the data. The architecture and interface of the microservice are
calculate a total distance between two events by adding up the
built to easily accommodate such changes. The service offers
distances between the individual attributes of the events. We
a machine-learning-process-centric API that allows control
refer to this distance as ’custom distance’ (CD) in the paper.
of an end-to-end machine learning process from creating a
With custom distance, we can assign different weights to the
prediction to correcting predicted classifications and re-training
attributes. The weight of an attribute is multiplied with the
these corrections back into the model. In Fig. 1 we present
distance of the attributes resulting in increased or decreased
an overview of MIRA’s position in the capture and replay
influence of the attribute towards the distance calculation. For
test process, where a capture C1 is replayed in multiple
instance, in the production model of MIRA, Error Codes and
HANA versions and the failed events are redirected to MIRA
Error Messages have higher weights in comparison the other
for classification. We have implemented this API in close
attributes as they are the most effective at identifying the
cooperation with the operators to incorporate their feedback
reason for the failure. We set the weights of the attributes
and improve MIRA’s usability as a framework.
in the production model based on cross-validation results and
III. M ACHINE LEARNING BASED E RROR C LASSIFICATION domain knowledge of our experts.
FOR R EPLAY A NALYSIS IN MIRA
Algorithm 1 CustomDistance
As described in the previous section, the core part of MIRA
is a machine learning based solution to classify the failed replay 1: Input: x, y ← Event
events into the root causes of their failures. There are two main 2: distance ← 0
aspects we address as a part of our solution: i) Classification 3: for i ← 1 to len(x) do
Model: Learning a model to classify failed events into their 4: if type(x[i]) == 0 ErrorM essage0 then
root causes. ii) Uncertainty Measures: Defining measures to 5: x[i] ← T extV ectorizer(x[i])
reflect the uncertainties of the model. 6: y[i] ← T extV ectorizer(y[i])
7: distance ← distance + cosine_distance(x[i], y[i])
A. Classification Model 8: else
In MIRA, we only use failed events to train a classification 9: binary_comparison ← 0
model. Amongst all the collected attributes for the failed replay 10: if x[i]! = y[i] then
events, we use a subset that includes HANA Server Error Code, 11: binary_comparison ← 1
HANA Server Error Message, Request type, SQL Type and 12: end if
SQL Sub Type to train a classifier. Based on our domain 13: distance ← distance + binary_comparison
knowledge, these attributes are the most predictive of the 14: end if
reason for the failure. In our data, all the selected attributes are 15: end for
categorical, apart from the error message. All categorical and 16: return distance/len(x)
unstructured data need to be converted into numerical data for
utilizing them as an input to the machine learning algorithms.
There are very few ways to encode categorical attributes with a B. Uncertainty Measures
categorical target. Furthermore, each of the attributes contains To make MIRA reliable, we identify mechanisms that reflect
several unique values in them and one hot encoding will result uncertainties of the model. For this purpose, we augment every
in very high dimensional data. Discretization or grouping the classification in MIRA with two measures that reflect the
values of an attribute can help reduce the number of unique classifier’s confidence in classifications. We refer to them as
values and thus reduce the number of resulting dimensions. Probability and Confidence. Both the measures range between
 Figure 1: Overview of MIRA within the Capture and Replay test pipeline of SAP HANA
Workload captured
from System 1
C1

Operator Analyses
Results
Failed Events
HANA build 1
Replay C1
MIRA Operator
Sign Off
Failed Events
HANA build 2
Replay C1
Retrain with training data
augmented with corrected
classifications and new items
HANA build n Failed Events
Replay C1

0 to 1. We consider a classification unreliable if the calculated IV. E VALUATION

probability or confidence has a low value. The threshold for In this section we present an overview of our data, data
low values is a pre-defined constant based on our experiments processing, evaluation procedure, and the results of our ap-
with the training data. Operators manually inspect and correct proach. We perform a k-fold stratified cross-validation [27] for
such unreliable classifications. The corrected classifications are hyperparameter optimization and classifier evaluation. Since
used to train MIRA further. we attempt to handle the issue of high dimensionality with
Probability: We calculate the probability for KNN on voting the Custom Distance (CD), we compare our approach to KNN
weighted by the distance of the neighbors. This reflects with Euclidean Distance (ED) [28], a distance measure that
uncertainty when there are members from different classes suffers in high dimensions. Additionally, we choose XGBoost
contributing equivalently towards the classification. However, [29] as it has been shown to be a very effective classification
the probability alone does not indicate uncertainty in cases algorithm across several domains of machine learning [30]. We
where a new event without resemblance to the events in the one hot encode all the categorical attributes, and concatenate
training data is classified with a probability of 1.0. This occurs them with vectorized error message to prepare the data for
when the neighbors contributing towards a classification all the baseline models. We present results for both TFIDF and
belong to the same class but are almost entirely different from Doc2Vec vectorization techniques. Table III shows the final
the event to be classified. hyperparameter settings. We present both F1-scores weighted
Confidence: Confidence is an additional measure to identify by class size (F1 Weighted) and a simple average across all
uncertainties not reflected by the probability. By design, custom classes (F1) in the results. We consider this representation
distance allows us to calculate the maximum possible distance suitable for data with a class imbalance as in our dataset.
between two vectors based on weights assigned to the features.
To calculate the confidence of a classification, we adopt A. Data
a modified version of KNN confidence criteria as defined in Prior to MIRA, operators investigated each failed event
previous work [26] (Eq. (5)). The modification is with respect manually. If the failure was caused by a known issue, operators
to the boundary consideration for the classification. In our assign a reason for it as part of the replay analysis. Otherwise,
case, we use the maximum distance between two event vectors, a new bug is created, linked to the failure reason, and assigned
calculated using custom distance described in Algorithm 1, as to the event. The reason describes the fundamental cause of
a boundary. The confidence for a predicted class C for an event a failure, including reported HANA bugs, JIRA issues, and
x is calculated as shown in Eq. (1): capture and replay problems. These failed events are used as
MIRA training data. Discrepancies such as failed events with
 
min{d(x, yj )| 1 < j < k and yj C}
f (x, C) = 1 − the same root cause but different labels occur as a result of
maximum distance
(1) different operators analyzing replays or mistakes made during
where d(x, y) is the custom distance between events x and y. replay analysis. Our domain experts reviewed it for labeling
The numerator is the distance from event x to the nearest errors to ensure that the training data was of high quality.
neighbor from class C belonging to the K neighbors con- Table II presents some examples of the data where each row
tributing towards the classification. The denominator is the represents one failed event. We categorize failures based on
maximum possible distance between two event vectors with their root cause. Some tables, for example, are unavailable
custom distance. The distance between two events is at its during the replay due to failure to install specific HANA
maximum when there are no matching values in any of the plugins during the replay. Attempts to access such tables are
selected attributes. Therefore, we consider two events to be likely to fail with errors such as “Could not find table/view
considerably different from one another when the distance X”. These failures are regarded as false positives because the
between them exceeds a pre-defined threshold for the distance. plugins are not available for the specific HANA binary. They
 Figure 2: Overview of evaluation process in MIRA

Restructure
Phase 1
Classes

Full Data Training

(~65000 samples) Cross validation
Downsample Data
Update Model Parameters

Analyse Results Re-Process/Update

Trial Model
(Accuracies & Data & Retrain
Deployment
Uncertainties) Model

Phase 2
Deploy New model

are grouped under a single root cause that represents the failure vectorization models. The vector size in TFIDF reduced to
to install HANA plugins. As a result, several thousands of about 6 000 from about 14 000 and the number of unique terms
failures can be aggregated under only a few hundred root in the Doc2Vec model also reduced. Downsampling improved
causes. Furthermore, the set of root causes grows as HANA the performance of the classifier across all the models, the
evolves and new issues are identified. We currently have a training cross validation results before and after the final
total of 93 classes of which we categorize 73 as false positives iteration of downsampling can be seen in Table V. All of
and remaining 20 classes as real database issues. Each class the results presented in this work are based on experiments
represents one root cause or reason of failure. The attribute conducted using the downsampled dataset. We have stated
Type of SQL indicates whether the statement is a DDL, DML, explicitly in case the entire dataset is used.
etc., SQL Sub Type indicates finer granularity of the SQL
type. The Error Code and Error Message are collected for Figure 3: Example of Error Message preprocessing
every failed event from the HANA SQL interface exposed
to HANA clients interacting with the database. There are 63 "invalid name of function or procedure: no procedure with name
unique Error Codes, 5 SQL types, 16 SQL sub types, and 9 YXR_TAB_NAME_1jk89{[ found: line 123 col 56 (at pos 89)"
types of Request Names in the data.
1) Class Imbalance: Considering that some failures occur [ 'invalid', 'name, 'function', 'procedure', 'procedure', 'with', 'name',
more frequently than others, there are more samples for some 'yxr', 'tab', 'name', '1jk89', 'found', 'line', '123', 'col', '56', 'pos', '89' ]
types of failures compared to others creating a class imbalance
in the data. We handle the class imbalance in the dataset to
some extent by downsampling the larger classes. We cannot 2) Preprocessing and Vectorization of Error Messages: We
upsample the minority classes through duplication as it could present the average lengths of the error messages in each class
allow unimportant terms like temporary names to influence in Fig. 4b. Some of the error messages are very long but 90% of
the vectorization of Error Message. Creating artificial samples our classes have error messages containing less than 100 terms.
from our data is quite challenging, as it contains just categorical We utilize the domain knowledge of our experts to ascertain the
and textual attributes. contributions of error message patterns towards identification
Random downsampling is not feasible for most of the classes of the root causes. Error messages generally contain valuable
in our data because a single root cause might consist of multiple information like location of the error which includes line num-
errors that vary slightly in pattern and it is necessary to retain bers and positions. There can be errors coming from the same
a representation of all the patterns in the data. To this end, file but with different root causes. Retaining numbers present
we refine the dataset one class at a time. We collected all in the error messages helps us to exploit this information.
known error patterns within each class and selected enough However, not all numbers are equally useful, for instance,
samples from each pattern to have an effective representation of the error messages also contain temporary table names with
the error within the data. We performed cross-validation after numbers, alphabets and nonalphanumeric characters. Such
each class is processed, to assess impacts of downsampling temporary names are unique in each error message and are
on the whole classification results. We started with a dataset not useful for identifying root causes. Therefore, we attempt
containing about 65 000 training samples, which was reduced to discard such strings as a part of preprocessing the error
to about 25 600 samples after the downsampling process. messages. Our final preprocessing steps involve lower casing,
Fig. 4a shows the final class distribution. The downsampling removal of nonalphanumeric characters, removal of stopwords.
also eliminates a substantial amount of unimportant terms Fig. 3 shows an example of a preprocessed error message.
from the vocabulary of the resulting Doc2Vec and TFIDF Five different classes share this error message pattern. Among
 Table II: Data example with artificial content for failed events and labels
Event Id Error Code Error Message SQL Type Subtype Request Type Reason
10000021 250986 ’invalid argument: cannot determine volume id’ 1 1 Type1 Reason1
10000022 7000012 ’Internal error: error=invalid table name: Could not find table’ 7 1 Type2 Reason2

these classes, differences are only with respect to the name of Table IV: Average F1-Scores 5-fold cross-validation on training
the table the error occurs on and the location of the error. The data using only Error Messages
numbers appearing after terms like ’line’, ’pos’ also indicate a Classifiers
root cause in our data. Retaining the context of terms through
F1 Weighted F1
word order is beneficial in such cases. A Bag of Words [31]
model like TFIDF ignores the word order and therefore will fail Vectorization XGBoost KNN+ED XGBoost KNN+ED
to capture such dependencies. Since our Error Messages are TFIDF 94.78 93.66 91.03 93.09
Doc2Vec 94.52 95.12 92.40 93.40
short, creating n-Grams can help us to retain the word order
but this will result in high dimensional data. Due to these
reasons, we utilize the Doc2Vec to create embeddings for the
B. Analysis of the Classification Results
error messages. We additionally tag each error message with
the class label while training the embedding model. This helps We analyse the performance of the machine learning compo-
us to establish an additional context of patterns belonging to nent in classifying failed replay events. We discuss our findings
the same root cause. Terms such as ’1jk89’ may repeat only in from the analysis of cross-validation results of training data
very rare cases and are mostly eliminated by setting a suitable and also the model performance in our production environment.
minimum term frequency during vectorization. Based on the Fig. 2 presents an overview of the process.
hyperparameter optimization, 3 is a suitable minimum term 1) Cross-Validation Results of Training Data: Despite our
frequency for both TFIDF and Doc2Vec models. We are able to experts’ label verification efforts, our full training dataset with
retain all of the important terms from the minority classes and 65 000 samples had some incorrectly labeled events. Since
discard most of the unimportant terms from the vocabulary of some of the misclassifications revealed the wrongly labeled
the models. Doc2Vec results in better performance across most events, we identified and corrected the remaining labeling
of our classification experiments. Therefore, we productively discrepancies during our initial cross-validation with training
use Doc2Vec as a vectorizer in MIRA. data. This process also compelled a more thorough analysis
of the root cause categorization in our data and helped us
Table III: Final hyperparameters from 5-fold cross-validation restructure the root causes into more meaningful clusters than
before. Most of the remaining misclassifications are due to
Model Library Final hyperparameters events belonging to classes that share all values except the
TFIDF Sklearn [32] stopwords=’english’, max_df=1.0, min_df=3 error message. The error messages from these classes also
epochs=50, vector size=50, min_count=3, share a lot of vocabulary. E.g., similar errors with different
Doc2Vec Gensim [33] table names: ’Cannot find the table name <TableName>’. We
window=5, epochs for infer_vector=200
KNN+CD Sklearn K=11, weighted by distance, method=’brute’ could resolve many of these issues by continuously adapting
KNN+ED Sklearn K=11, weighted by distance, method=’auto’
the text preprocessing for error messages and selecting an
appropriate vectorization technique, in our case Doc2Vec.
max_depth=20, learning_rate=0.1
XGBoost XGBoost [29] objective=’multi:softprob’, booster=’gbtree’,
As seen in our experiments, KNN with CD has a better
eval_metric=’logloss’, n_estimators=300 F1-score in most cases compared to our baselines. XGBoost
performs on par with our approach and is also a comparatively
faster classifier than KNN for predictions. However, for em-
3) Feature Selection: We conducted experiments to compare ploying the classification model in production, we analyze the
the performance of classification models trained only using results with a focus on two aspects. i) the model should be
vectorized error messages and models trained with all the able to reflect uncertainties when the failed events cannot be
attributes. We do not present results for KNN CD here as classified with high certainty. ii) interpretability of the results.
without additional attributes it is merely a KNN with cosine We set the thresholds for the probability based on the analysis
distance. Even though the F1-score from 5-fold cross-validation of correctly classified events in the cross-validation results.
is presented in Table IV indicate a reasonable performance There are on average only 11 events per fold that are correctly
using only Error Messages, models built with all the selected classified but have probabilities below the threshold of 0.9
attributes have better F1-scores across all the classifiers as in case of KNN with CD. In case of XGBoost, there are
presented in Table V. With these experiments our final selection on average 110 correctly classified events whose probabilities
of attributes includes Error Codes, SQL types, SQL sub types, ranged from 0.5 to 0.8 and are below the threshold setting
Request Names, and Error Messages of which Error Message of 0.8. The XGBoost model will result in a relatively larger
is the most important attribute. workload for operators in comparison to the KNN with custom
 Figure 4: Data overview
6000
500

Average Length in #Words

5000
400

#Training Items
4000
300
3000
2000 200

1000 100

0 0
0 20 40 60 80 0 20 40 60 80
Class ID Class ID
(a) Class distribution (b) Average length of error messages in each class
Table V: Average F1-Scores of 5-fold cross-validation on training data
Classifiers
F1 Weighted F1
Data Vectorization KNN+CD KNN+ED XGBoost KNN+CD KNN+ED XGBoost
Downsampled TFIDF 99.13 95.67 98.78 98.69 98.20 98.31
Doc2Vec 99.70 98.98 99.41 98.80 97.63 97.96
Full data TFIDF 98.12 94.32 96.67 97.83 95.01 97.01
Doc2Vec 98.67 96.50 97.70 97.01 95.32 97.21

distance model. Furthermore, there are more possibilities of and colors of the clusters indicate different classes. Some of
uncertain classifications surpassing a lower threshold setting the classes, like 58 and 36, have dense and well separated
for a certainty measure, which we attempt to avoid in MIRA. clusters from other classes. Such classes have less variance in
It is currently harder to set such a threshold in case of their error patterns and are generally not misclassified. Clusters
XGBoost. The prediction probabilities of KNN with CD have that are larger and more dispersed within a single class suggest
less variance in comparison to the probabilities of XGBoost. the presence of multiple error patterns. Classes such as 27 and
Furthermore, the classifications of KNN models are easier to 50 share vocabularies and are prone to misclassifications.
interpret than the classifications of XGBoost. This is especially
true when Doc2Vec is used for vectorization. MIRA logs the Figure 5: Visualizations of Doc2Vec embeddings using t-SNE
neighbors and their distances during a classification, making it
easy to analyze the results. For these reasons, we are currently
17 7 72
using the KNN with CD to classify failed events in MIRA. 15 78 48
58 60
However, in our case, XGBoost is a candidate for creating an 55 44 82 13 37 0
81 14 4 6538 35
22 53
ensemble setup in MIRA. 10 9 86
34 70 24
21 50
2) Retraining: We currently monitor the classification re- 36 79 84
57 61
sults very closely in MIRA. The operators comprehensively 40 88 27 8
1
31 30 71 67
analyze all uncertain classifications to identify misclassifica- 43 3 66
52 29 26
77
59 90
54 33 49 46
73
tions and correct them and add them to a collection. In this
39 51 18 16 45
23
28 64 42
collection we also include the new failure types identified 85 32 768741
20
89 11 74
25 80 12
through an ongoing manual replay analyses. We augment the 19
2 47 83 65 62
training data with this collection of samples and retrain the 63 69
75
6856
classifier once a week. Based on our observations, retraining
has improved the performance in the subsequent classification
results in terms of accuracy and confidence of the predictions.
3) Analysis with Visualizations: Considering that the Error
Messages play the most important role in our classification
process, obtaining an approximation of how the error messages In practice, the visual guide is helpful to the operators as
of different classes are embedded in the vector space with well to obtain an overview of the proportion of the classes
respect to each other is helpful. We visualize the Doc2Vec in the data, identify the categories of root causes that are
embeddings using the t-SNE [34]. In Fig. 5 we present a brief placed very close to each other and prone to misclassification.
overview of the embeddings of a subset of our data. The label Currently the MIRA administrators make a final decisions
on many aspects such as adding new training items and Table VII: Average accuracy and false uncertainty from evalu-
accepting corrected classifications. As the operators gain more ation of MIRA in production
experience with MIRA, they can rely on such visualizations Phase #Replays Accuracy False Uncertainty Data KNN
to make the aforementioned decisions, making this process
1 18 98.61 0.5% to 15% 65 000 K=3
more democratic. Based on our discussions with the operators, 2 10 97.83 <0.5% 25 600 K=11
an interactive visualization for the Error Message embeddings
along with a metric like F1-Score to indicate the changes in
performance from one training iteration to the next, can be events which are then reclassified and added to the training
beneficial to understand the classification results of MIRA. data to retrain the classifier. With the retrained classifier, clas-
4) Alternative approaches: We evaluated an alternative sification accuracy for Replay 2 was 99.40 %. Since both the
approach by concatenating all the attribute values for an event replays belong to the same capture, retraining the model with
into one string and vectorizing it to use as a input to the corrected classifications improved the classification accuracy
classifiers. The average F1 Score over 5 fold Cross-Validation for Replay 2. There were less than 0.5% false uncertainties
is shown in Table VI. This approach impacts the classification across both replays. However, subsequent replay classifications
of several minority classes. However for our data, this approach deteriorated in terms of accuracy and certainty, resulting in over
is a promising alternative. 15% false uncertainties, increasing the analysis workload. We
discovered that the majority classes hampered the predictions
Table VI: Average F1-Scores 5-fold cross-validation on con- of several minority classes, and the low K value exacerbated
catenated and vectorized features the problem. As a result, we downsampled the majority classes,
Classifiers as indicated in Section IV-A, and further optimized the model.
F1 Weighted F1
In Phase 2, the model was trained on downsampled data with
25 600 samples, restructured root causes and K=11. Fig. 4
Vectorization XGBoost KNN+ED XGBoost KNN+ED
shows an overview of the final dataset. The classifications
TFIDF 99.11 98.32 95.13 95.32 are comparatively more robust, with a higher probability and
Doc2Vec 98.78 99.20 94.61 93.12
confidence for correct classifications, based on a detailed
analysis of 10 replays. A smaller number of training events
5) Performance of MIRA in Production Environment: We results in fewer distance calculations and a faster model. The
discuss the performance of MIRA in production in two phases average prediction accuracy is about 97%, with less than 0.5%
i) Phase 1: Performance of model trained on the full dataset false uncertainties across all 10 replays. Most of the uncertain
ii) Phase 2: Performance of the optimized model based on the classifications in MIRA are misclassified events that pertain to
feedback from Phase 1. Fig. 2 presents an overview of the two an untrained new root cause category.
phases. Using MIRA, we have classified failed events from 51 Across both the phases, despite the need for analysis of
replays from 7 different captured workloads. The number of results, frequent changes and updates to the model, we have
failed events per replay ranged from 3 000 to 161 000, with 4 verified that MIRA results in a significantly smaller workload
to 35 classes per replay. We found 75 unique classes out of 93 for the operators in comparison to the previous approach of
classes present in the training data. Common settings across replay analysis for SAP HANA.
both the phases include Doc2Vec model for the Error Messages,
a trained KNN with CD, threshold of 0.9 for probability V. P RACTICAL A PPLICATION AND D ISCUSSION
and 0.7 for confidence to filter uncertain classifications. We In this section, we describe the contributions of MIRA
closely examined MIRA’s classifications of failed events from towards replay analysis at SAP, expand on some of the
at least 28 replays where the reasons for failures were known. challenges we encountered in the process of applying MIRA
Whenever the predicted reason is correct but one or both of in practice and propose possible solutions. We briefly discuss
the certainty measures fall below the defined thresholds, we the feedback from the end-users of MIRA.
consider this a false uncertainty. Due to the fact that operators
investigate all uncertain classifications, false uncertainties are A. Application of MIRA
regarded as additional workload caused by MIRA. As a result, MIRA does not depend on a particular capture and therefore
we use false uncertainty as one of the metrics to evaluate allows a high degree of automation for result analysis in
MIRA. Table VII summarizes our production evaluation. comparison to the existing rule-based approach. It improves
In Phase 1, we utilized the model trained with the full dataset the user experience for the operators because it reduces manual
with about 65 000 training samples with K=3 for KNN. We work and the time between test result and test assessment. Due
set a low K value to accommodate the minority classes. We to categorization of the failed events into known issues, the
classified failed events from 24 replays, of which we describe operators have a much smaller workload of only analyzing
the analysis of 2 replays belonging to the same capture. Each uncertain results from MIRA. MIRA serves as a central
replay has over 13,000 failed events and four root cause classes. repository for all known issues and eliminates having to
Classification accuracy for Replay 1 was 97.83 %. In Replay 1, maintain error metadata per capture. Ultimately, MIRA offers
uncertainty measurements correctly identified the misclassified a much more scalable analysis and allows us to scale the entire
capture and replay process to many concurrent systems as part
[40155]{311057}[ 2020-12-21 19:25:23 e MNO module_name.cc(XXX) :
of the HANA test and release pipeline. assertion failed: (some_constraint == false); mhr: 1; nhr: 0; message:
Employing MIRA gives us a holistic overview of all product
issues in replays, their frequency, and distribution across Figure 6: Example of an artificial trace file entry of Failed
customer systems. This permits tracking of bugs over time and Assertion: Error messages are augmented with italicized parts
code line. This leads to better product release and upgrade
decisions, as well as customer-specific optimizations like
configuration and data distribution changes. are partially unifying and standardizing the way they convey
system failures to end users. In practice, components now offer
B. Feedback from End-Users less error context and more guidance for customers or end
The productive usage of MIRA provides an immediate users on how to deal with an issue. This category of failures
benefit to operators, as database issue are pre-analyzed by has fixed error messages. For instance, “further analysis is
the machine learning model to classify the symptoms to the required to identify the root cause of the failure”. Since these
correct root cause. This greatly helps to synchronize analysis failures produce identical events, MIRA cannot classify them
efforts as symptoms are not analyzed multiple times if they using the current set of features. Uncertainty measures will
appeared on multiple replays. With our experiments, MIRA has fail to identify them as well.
shown reliable performance across several replays. Therefore In some of these cases, additional error context exists that
the operators are not required to remember a variety of different helps the operators classify the errors correctly. Stack traces
symptoms and their actual root cause as they can trust the initial are one of the event attributes available for specific HANA
assessments and classifications of the model. failures. In these cases, the operators manually analyze the
However, the introduction of a machine learning tool, which stack traces collected for the failures to distinguish between
relies on a constant feedback and training loop from operators, such failures. With an initial analysis, we were able to identify
has shown that this also creates an overhead for operators, helpful patterns within the trace files across all such failures to
as they have to analyze the machine learning predictions and help the model distinguish between such events. For instance,
correct them if needed. These additional efforts have to be we present a sample entry of an assertion failure in Fig. 6.
considered, especially in phases where the model is still in its We preprocess the entries in the trace files to extract an entry
early stages of training and configuration. Ultimately, as data carrying such an error message and append this extracted string
from more replays is gathered and trained into the model, we to the original error message. These augmented error messages
assume that this operational overhead will gradually decrease are then used as input to create the Doc2Vec embeddings for
over time. To ensure that the amount of manual checks and the error messages. Our data contains several categories of
interventions is limited to a minimum, multiple conditions failure, and trace files are not available for all of the failure
have to be provided by MIRA: i) Ensure a stable classification classes. Whenever a trace file is available for the failure, we
result over multiple predictions to avoid operator confusion extend the preprocessing steps for the error messages.
and distrust in the model’s judgement. ii) Provide a confidence We performed a 5-fold cross-validation on the training data
metric that reliably indicates uncertainties to an operator to to test this approach. We were able to successfully classify
trigger a manual intervention. iii) Ensure that the model is this category of failed events into their correct root causes. We
re-trained continuously to represent the new HANA errors as present the change in Doc2Vec embeddings using t-SNE plot
quickly as possible in the subsequent predictions. in Fig. 7 where the classes appear to be well separated after
using the trace file entries. We currently only have 5 classes
VI. C HALLENGES , L IMITATIONS AND I MPROVEMENTS with such a scenario. The frequency of such failures is rare.
We identify new failure types frequently in an ongoing replay We need to gather more data and conduct more experiments
analysis and therefore the data for MIRA also evolves. As to apply this solution productively. However, the stack traces
the data for learning in MIRA grows and diversifies, we will prove to be a valuable feature for classifying replay failures.
encounter problems that will need our current setup and the ii) Change in Structure of Table Names: The table name
models to be constantly adapted. In this section we briefly is the most important part of the error message for iden-
discuss some of the challenges we faced in MIRA and report tifying several root causes in MIRA. In our data, the ta-
limitations and potential improvements of our solution. ble names with dynamically generated parts mostly have
a structure of ’TABLENAME_ijh78fk’ where the dynamic
A. Challenges of Evolving Data part is concatenated to the table names with an ’_’. This is
The evolving data leads to two main challenges: i) Identical easily separated and eliminated during the text preprocessing
Failed Events, and ii) Change in Structure of Table Names. steps for error messages. However, in some of the recent
i) Identical Failed Events: Historically, system faults in errors, we have discovered instances with table names like
HANA have been propagated transparently to administrators ’TABLENAMEijh78fk’. This presents new challenges to our
and customers. Although this gives a lot of context for failures existing set up, like the minimum frequency setting for text
for expert operators, it also creates ambiguity for end users vectorization and the vectorization technique itself. Due to the
as the errors are hard to understand. Recently, components unique temporary string, these terms will always be treated
 (a) Without using trace files how the correctness of machine learning approaches should be
tested. This results in several practical questions such as: What
is the meaning of a test oracle in the machine learning domain?
What exactly is a failure for a machine learning result? What
metrics can we use for test adequacy? We welcome any ideas
5 and suggestion towards these questions.
0 Currently, MIRA retrains the classification model with cor-
5
38
68
75
rected classifications, creating a semi-supervised learning setup.
86 56
10
This setup is still prone to issues of inconsistent correction of
uncertain classifications by operators and misclassifications not
0 captured by both the uncertainty measures. Such events getting
5
0 10 trained into the model will lead to deterioration of the model
5
10 15
15 20 quality over time. MIRA administrators are responsible for
20
observing the model quality and taking necessary action.

VII. C ONCLUSION
5 In this work, we discussed MIRA, an approach to automat-
0
86

ically classify failed replay events into their root causes using
75
5 machine learning. As a part of the solution, we propose a
10
68

38
56
customized distance function to mitigate issues of encoding
categorical attributes containing several values resulting in
5 high dimensional data. We have shown our approach to be
0 5 10 highly accurate and reliable with our evaluation for our data.
10 15 15 MIRA makes the process of replay failure analysis more
20 25 20
consistent, saves manual effort and allows to focus on solving
issues concerning the quality of the underlying software. The
(b) Using trace files
involved effort incurs considerably lesser cost than the error-
Figure 7: Comparison of changes in Doc2Vec embeddings prone manual inspection. We also highlighted several practical
before and after using trace files limitations and open questions in the areas of supporting
changes over time, testing machine learning based approaches,
and best practices for human-in-the loop architectures.
as out-of-Vocabulary (OOV) terms and therefore will have
We intend to automate model monitoring and retraining
no influence on the classification. The model will be severely
within MIRA in the future. Considering that we find stack
handicapped by such OOV issues. To alleviate these issues, we
traces useful for categorization, we’re working on adding an
have identified fasttext [35] as an alternative for vectorizing the
exception stack trace comparison as an additional feature and
error messages. fasttext creates embeddings for the subword
expanding MIRA’s custom distance computation.
vectors and therefore can generate vectors for such OOV words
by using a subword like ’TABLENAME’.
R EFERENCES
B. Limitations and Improvements [1] F. Färber, N. May, W. Lehner, P. Große, I. Müller, H. Rauhe, and J. Dees,
“The SAP HANA database – an architecture overview,” Bulletin of the
The data will change over time when new software defects Technical Committee on Data Engineering / IEEE Computer Society,
are introduced. It is a challenge to recognize changes and to vol. 35, no. 1, pp. 28–33, 2012. (Cited on page: 1)
maintain an up to date training data, retrain the classifier and [2] F. Färber, S. K. Cha, J. Primsch, C. Bornhövd, S. Sigg, and W. Lehner,
reassess the threshold settings for the model certainty. Both “SAP HANA database: Data management for modern business applica-
tions,” SIGMOD Record, vol. 40, no. 4, Jan. 2012. (Cited on page: 1)
the text vectorization techniques, TFIDF and Doc2Vec, require [3] T. Bach, A. Andrzejak, and R. Pannemans, “Coverage-based reduction
retraining to accommodate the vocabulary of new failures. of test execution time: Lessons from a very large industrial project,” in
Additionally, document embeddings may need retraining when 2017 IEEE International Conference on Software Testing, Verification
and Validation Workshops, ser. ICSTW 2017. Washington, DC, USA:
there is a remarkable change in context of terms in the new data IEEE Computer Society, March 2017, pp. 3–12. (Cited on page: 1)
in comparison to the data used for learning the embeddings. [4] T. Bach, R. Pannemans, and S. Schwedes, “Effects of an economic
The aspect of detecting changes can also be generalized to approach for test case selection and reduction for a large industrial
project,” in 2018 IEEE International Conference on Software Testing,
the general question of how machine learning approaches are Verification and Validation Workshops (ICSTW). Washington, DC, USA:
tested from the software engineering perspective. As machine IEEE Computer Society, April 2018, pp. 374–379. (Cited on page: 1)
learning applies statistics, it inherently has some uncertainty [5] S. Baek, J. Song, and C. Seo, “RSX: Reproduction scenario extraction
technique for business application workloads in DBMS,” in 2020 IEEE
about the results. It remains unclear, and to our knowledge of International Symposium on Software Reliability Engineering Workshops
the current state of research, it is also an unsolved problem, (ISSREW). IEEE, 2020, pp. 91–96. (Cited on page: 1)
 [6] J. Yan, Q. Jin, S. Jain, S. D. Viglas, and A. Lee, “Snowtrail: Testing with [26] C. Dalitz, “Reject options and confidence measures for KNN classi-
production queries on a cloud database,” in Proceedings of the Workshop fiers,” Schriftenreihe des Fachbereichs Elektrotechnik und Informatik
on Testing Database Systems, ser. DBTest’18. New York, NY, USA: Hochschule Niederrhein, vol. 8, pp. 16–38, 2009. (Cited on page: 4)
Association for Computing Machinery, 2018. (Cited on page: 1) [27] M. Stone, “Cross-validatory choice and assessment of statistical predic-
[7] F. Arruda, A. Sampaio, and F. A. Barros, “Capture & replay with text- tions,” Journal of the Royal Statistical Society: Series B (Methodological),
based reuse and framework agnosticism.” in International Conference vol. 36, no. 2, pp. 111–133, 1974. (Cited on page: 4)
on Software Engineering and Knowledge Engineering (SEKE), 2016, pp. [28] B. O’neill, Elementary differential geometry. Elsevier, 2006. (Cited on
420–425. (Cited on page: 1) page: 4)
[8] L. Galanis, S. Buranawatanachoke, R. Colle, B. Dageville, K. Dias, [29] T. Chen and C. Guestrin, “XGBoost: A scalable tree boosting system,”
J. Klein, S. Papadomanolakis, L. L. Tan, V. Venkataramani, Y. Wang, in Proceedings of the 22nd ACM SIGKDD International Conference
and G. Wood, “Oracle database replay,” in Proceedings of the 2008 on Knowledge Discovery and Data Mining, ser. KDD ’16. New
ACM SIGMOD International Conference on Management of Data, ser. York, NY, USA: ACM, 2016, pp. 785–794. [Online]. Available:
SIGMOD ’08. New York, NY, USA: Association for Computing http://doi.acm.org/10.1145/2939672.2939785 (Cited on pages: 4 and 6)
Machinery, 2008, p. 1159–1170. (Cited on page: 1) [30] C. Bentéjac, A. Csörgő, and G. Martínez-Muñoz, “A comparative analysis
[9] Y. Wang, S. Buranawatanachoke, R. Colle, K. Dias, L. Galanis, S. Pa- of gradient boosting algorithms,” Artificial Intelligence Review, vol. 54,
padomanolakis, and U. Shaft, “Real application testing with database no. 3, pp. 1937–1967, 2021. (Cited on page: 4)
replay,” in Proceedings of the Second International Workshop on Testing [31] Y. Zhang, R. Jin, and Z.-H. Zhou, “Understanding bag-of-words model:
Database Systems, 2009, pp. 1–6. (Cited on page: 1) a statistical framework,” International Journal of Machine Learning and
[10] S. Sprenkle, E. Gibson, S. Sampath, and L. Pollock, “Automated replay Cybernetics, vol. 1, no. 1-4, pp. 43–52, 2010. (Cited on page: 6)
and failure detection for web applications,” in Proceedings of the 20th [32] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion,
IEEE/ACM international conference on automated software engineering, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vander-
2005, pp. 253–262. (Cited on page: 1) plas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duch-
[11] A. Orso and B. Kennedy, “Selective capture and replay of program esnay, “Scikit-learn: Machine learning in Python,” Journal of Machine
executions,” ACM SIGSOFT Software Engineering Notes, vol. 30, no. 4, Learning Research, vol. 12, pp. 2825–2830, 2011. (Cited on page: 6)
pp. 1–7, 2005. (Cited on page: 1) [33] R. Řehůřek and P. Sojka, “Software Framework for Topic Modelling with
[12] A. Podgurski, D. Leon, P. Francis, W. Masri, M. Minch, J. Sun, Large Corpora,” in Proceedings of the LREC 2010 Workshop on New
and B. Wang, “Automated support for classifying software failure Challenges for NLP Frameworks. Valletta, Malta: ELRA, May 2010,
reports,” in 25th International Conference on Software Engineering, 2003. pp. 45–50, http://is.muni.cz/publication/884893/en. (Cited on page: 6)
Proceedings. IEEE, 2003, pp. 465–475. (Cited on page: 2) [34] L. Van der Maaten and G. Hinton, “Visualizing data using t-sne.” Journal
of machine learning research, vol. 9, no. 11, 2008. (Cited on page: 7)
[13] J. Kahles, J. Törrönen, T. Huuhtanen, and A. Jung, “Automating
[35] P. Bojanowski, E. Grave, A. Joulin, and T. Mikolov, “Enriching word
root cause analysis via machine learning in agile software testing
vectors with subword information,” Transactions of the Association for
environments,” in IEEE Conference on Software Testing, Validation and
Computational Linguistics, vol. 5, pp. 135–146, 2017. (Cited on page: 10)
Verification (ICST). IEEE, 2019, pp. 379–390. (Cited on page: 2)
[14] Y. Feng, J. Jones, Z. Chen, and C. Fang, “An empirical study on soft-
ware failure classification with multi-label and problem-transformation
techniques,” in 2018 IEEE 11th International Conference on Software
Testing, Verification and Validation (ICST). IEEE, 2018, pp. 320–330.
(Cited on page: 2)
[15] M. Du, F. Li, G. Zheng, and V. Srikumar, “Deeplog: Anomaly detection
and diagnosis from system logs through deep learning,” in Proceedings
of the 2017 ACM SIGSAC Conference on Computer and Communications
Security, 2017, pp. 1285–1298. (Cited on page: 2)
[16] C. Bertero, M. Roy, C. Sauvanaud, and G. Trédan, “Experience report:
Log mining using natural language processing and application to anomaly
detection,” in IEEE International Symposium on Software Reliability
Engineering (ISSRE). IEEE, 2017, pp. 351–360. (Cited on page: 2)
[17] V. Hodge and J. Austin, “A survey of outlier detection methodologies,”
Artificial intelligence review, vol. 22, no. 2, 2004. (Cited on page: 2)
[18] J. Uddin, R. Ghazali, M. M. Deris, R. Naseem, and H. Shah, “A survey
on bug prioritization,” Artificial Intelligence Review, vol. 47, no. 2, pp.
145–180, 2017. (Cited on page: 2)
[19] J. Zou, L. Xu, M. Yang, M. Yan, D. Yang, and X. Zhang, “Duplication
detection for software bug reports based on topic model,” in 2016 9th
International Conference on Service Science (ICSS). IEEE, 2016, pp.
60–65. (Cited on page: 2)
[20] B. S. Neysiani and S. M. Babamir, “Automatic duplicate bug report
detection using information retrieval-based versus machine learning-
based approaches,” in 2020 6th International Conference on Web
Research (ICWR). IEEE, 2020, pp. 288–293. (Cited on page: 2)
[21] G. Catolino, F. Palomba, A. Zaidman, and F. Ferrucci, “Not all bugs
are the same: Understanding, characterizing, and classifying bug types,”
Journal of Systems and Software, vol. 152, 2019. (Cited on page: 2)
[22] G. Salton and M. J. McGill, Introduction to modern information retrieval.
USA: McGraw-Hill, Inc., 1986. (Cited on page: 3)
[23] Q. Le and T. Mikolov, “Distributed representations of sentences and
documents,” in International conference on machine learning. PMLR,
2014, pp. 1188–1196. (Cited on page: 3)
[24] A. Mucherino, P. J. Papajorgji, and P. M. Pardalos, “K-nearest neighbor
classification,” in Data mining in agriculture. Springer, 2009, pp. 83–
106. (Cited on page: 3)
[25] V. Pestov, “Is the k-NN classifier in high dimensions affected by the
curse of dimensionality?” Computers & Mathematics with Applications,
vol. 65, no. 10, pp. 1427–1437, 2013. (Cited on page: 3)