Sieve: Actionable Insights from

Monitored Metrics in Microservices

Jörg Thalheim1 , Antonio Rodrigues2 , Istemi Ekin Akkus3 , Pramod Bhatotia1 ,
Ruichuan Chen3 , Bimal Viswanath4 , Lei Jiao5 , Christof Fetzer6
1 University of Edinburgh, 2 Carnegie Mellon Univ., 3 NOKIA Bell Labs, 4 University of Chicago, 5 University of Oregon, 6 TU Dresden

Abstract this transformation becomes difficult with the increasing size and
Major cloud computing operators provide powerful monitoring complexity of the application.
tools to understand the current (and prior) state of the distributed In this paper, we share our experience on: How can we derive
systems deployed in their infrastructure. While such tools provide a actionable insights from the monitored metrics in distributed systems?
detailed monitoring mechanism at scale, they also pose a significant In particular, given a large number of monitored metrics across
challenge for the application developers/operators to transform the different components (or processes) in a distributed system, we
huge space of monitored metrics into useful insights. These insights want to design a platform that can derive actionable insights from
are essential to build effective management tools for improving the the monitored metrics. This platform could be used to support a
efficiency, resiliency, and dependability of distributed systems. wide-range of use cases to improve the efficiency, resiliency, and
This paper reports on our experience with building and deploy- reliability of distributed systems.
ing Sieve—a platform to derive actionable insights from monitored In this work, we focus on microservices-based distributed sys-
metrics in distributed systems. Sieve builds on two core compo- tems because they have become the de-facto way to design and
nents: a metrics reduction framework, and a metrics dependency deploy modern day large-scale web applications [51]. The microser-
extractor. More specifically, Sieve first reduces the dimensionality vices architecture is an ideal candidate for our study for two reasons:
of metrics by automatically filtering out unimportant metrics by First, microservices-based applications have a large number of dis-
observing their signal over time. Afterwards, Sieve infers metrics tributed components (hundreds to thousands [48, 59]) with complex
dependencies between distributed components of the system using communication patterns, each component usually exporting several
a predictive-causality model by testing for Granger Causality. metrics for the purposes of debugging, performance diagnosis, and
We implemented Sieve as a generic platform and deployed it for application management. Second, microservices-based applications
two microservices-based distributed systems: OpenStack and Share- are developed at a rapid pace: new features are being continuously
Latex. Our experience shows that (1) Sieve can reduce the number integrated and deployed. Every new update may fix some existing
of metrics by at least an order of magnitude (10 − 100×), while pre- issues, introduce new features, but can also introduce a new bug.
serving the statistical equivalence to the total number of monitored With this rapid update schedule, keeping track of the changes in
metrics; (2) Sieve can dramatically improve existing monitoring the application as a whole with effects propagating to other compo-
infrastructures by reducing the associated overheads over the en- nents becomes critical for reliability, efficiency, and management
tire system stack (CPU—80%, storage—90%, and network—50%); purposes.
(3) Lastly, Sieve can be effective to support a wide-range of work- The state-of-the-art management infrastructures either rely on
flows in distributed systems—we showcase two such workflows: ad hoc techniques or custom application-specific tools. For instance,
Orchestration of autoscaling, and Root Cause Analysis (RCA). This prior work in this space has mostly focused on analyzing message-
technical report is an extended version of our conference publica- level traces (instead of monitored metrics) to generate a causal
tion [87]. model of the application to debug performance issues [33, 45]. Al-
ternatively, developers usually create and use custom tools to ad-
dress the complexity of understanding the application as a whole.
1 Introduction For example, Netflix developed several application-specific tools
Most distributed systems are constantly monitored to understand for such purposes [8, 48] by instrumenting the entire application.
their current (and prior) states. The main purpose of monitoring is These approaches require either complicated instrumentation or
to gain actionable insights that would enable a developer/operator sophisticated techniques to infer happens-before relationships (for
to take appropriate actions to better manage the deployed system. the causal model) by analyzing message trace timestamps, making
Such insights are commonly used to manage the health and resource them inapplicable for broader use.
requirements as well as to investigate and recover from failures This paper presents our experience with designing and building
(root cause identification). For these reasons, monitoring is a crucial Sieve, a system that can utilize an existing monitoring infrastruc-
part of any distributed system deployment. ture (i.e., without changing the monitored information) to infer
All major cloud computing operators provide a monitoring infras- actionable insights for application management. Sieve takes a data-
tructure for application developers (e.g., Amazon CloudWatch [2], driven approach to enable better management of microservices-
Azure Monitor [13], Google StackDriver [5]). These platforms pro- based applications. At its core, Sieve is composed of two key mod-
vide infrastructure to monitor a large number (hundreds or thou- ules: (1) a metric reduction engine that reduces the dimensionality
sands) of various application-specific and system-level metrics as- of the metric space by filtering out metrics that carry redundant
sociated with a cloud application. Although such systems feature information, (2) a metric dependency extractor that builds a causal
scalable measurement and storage frameworks to conduct moni- model of the application by inferring causal relationships between
toring at scale, they leave the task of transforming the monitored metrics associated with different components.
metrics into usable knowledge to the developers. Unfortunately,
 Module (1) enables Sieve to identify “relevant” metrics for a given Table 1. Number of metrics exposed by microservices.
application management task. For instance, it might be sufficient
to monitor only a few metrics associated with error states of the Microservices Number of metrics
application instead of the entire set when monitoring the health Netflix [61] ∼ 2,000,000
of the application. It is important to also note that reducing the Quantcast [14] ∼ 2,000,000
metric space has implications for deployment costs: frameworks Uber [16] ∼ 500,000,000
like Amazon CloudWatch use a per-metric charging model, and ShareLatex [23] 889
OpenStack [18, 20] 17,608
not identifying relevant metrics can significantly drive up the cost
related to monitoring the application.
Module (2) is crucial for inferring actionable insights because it
is able to automatically infer complex application dependencies. In dependencies between the components and utilizing these depen-
a rapidly updating application, the ability to observe such complex dencies with the exported metrics becomes a challenging task. As
dependencies and how they may change is important for keeping a result, understanding how the application performs as a whole
one’s understanding of the application as a whole up-to-date. Such becomes increasingly difficult.
up-to-date information can be helpful for developers to quickly Typical microservices-based applications are composed of hun-
react to any problem that may arise during deployment. dreds of components [48, 59]. Table 1 shows real-world microservices-
We implemented Sieve as a generic platform, and deployed it based applications that have tens of thousands of metrics and hun-
with two microservices-based distributed systems: ShareLatex [23] dreds of components. We experimented with two such applications,
and OpenStack [28]. Our experience shows that (1) Sieve can re- ShareLatex [23] and OpenStack [19], each having several thousands
duce the number of monitored metrics by an order of magnitude of metrics and order of tens of components. The metrics in these
(10 − 100×), while preserving the statistical equivalence to the total applications come from all layers of the application like hardware
number of monitored metrics. In this way, the developers/operators counters, resource usage, business metrics or application-specific
can focus on the important metrics that actually matter. (2) Sieve metrics.
can dramatically improve the efficiency of existing metrics monitor- To address this data overload issue, developers of microservices-
ing infrastructures by reducing the associated overheads over the based applications usually create ad hoc tools. For example, applica-
entire system stack (CPU—80%, storage—90%, and network—50%). tion programmers at Netflix developed several application-specific
This is especially important for systems deployed in a cloud infras- tools for such purposes [8, 48]. These tools, however, require the
tructure, where the monitoring infrastructures (e.g. AWS Cloud- application under investigation to be instrumented, so that the
Watch) charge customers for monitoring resources. And finally, (3) communication pattern between components can be established
Sieve can be employed for supporting a wide-range of workflows. by following requests coming into the application. This kind of in-
We showcase two such case-studies: In the first case study, we use strumentation requires coordination among developers of different
ShareLatex [23] and show how Sieve can help developers orches- components, which can be a limiting factor for modularity.
trate autoscaling of microservices-based applications. In the second Major cloud computing operators also provide monitoring tools
case study, we use OpenStack [28] and show how developers can for recording all metric data from all components. For example,
take advantage of Sieve’s ability to infer complex dependencies Amazon CloudWatch [2], Azure Monitor [13], and Google Stack-
across various components in microservices for Root Cause Analy- Driver [5]. These monitoring tools aid in visualizing and processing
sis (RCA). Sieve’s source code with the full experimentation setup metric data in real-time (i.e., for performance observation) or af-
is publicly available: https://sieve-microservices.github.io/. ter an issue with the application (i.e., for debugging). These tools,
however, either use a few system metrics that are hand-picked by
developers based on experience, or simply record all metric data
2 Overview for all the components.
In this section, we first present some background on microservices- Relying on past experience may not always be effective due to
based applications and our motivation to focus on them. Afterwards, the increasing complexity of a microservices-based application. On
we present our goals, design overview, and its possible use cases. the other hand, recording all metric data can create significant
monitoring overhead in the network and storage, or in the case
2.1 Background and Motivation of running the application in a cloud infrastructure (e.g., AWS), it
Microservices-based applications consist of loosely-coupled dis- can incur costs due to the provider charging the customers (e.g.,
tributed components (or processes) that communicate via well- CloudWatch). For these reasons, it is important to understand the
defined interfaces. Designing and building applications in this way dependencies between the components of a microservice-based
increases modularity, so that developers can work on different application. Ideally, this process should not be intrusive to the
components and maintain them independently. These advantages application. Finally, it should help the developers to identify and
make the microservices architecture the de facto design choice for minimize the critical components and metrics to monitor.
large-scale web applications [51].
While increasing modularity, such an approach to developing 2.2 Design Goals
software can also increase the application complexity: As the num- While designing Sieve, we set the following goals.
ber of components increases, the interconnections between compo- • Generic: Many tools for distributed systems have specific
nents also increases. Furthermore, each component usually exports goals, including performance debugging, root cause analysis
several metrics for the purposes of debugging, performance diag- and orchestration. Most of the time, these tools are custom-
nosis, and application management. Therefore, understanding the built for the application in consideration and target a certain
2
 Step#1: Load the application Step#2: Reduce metrics
Step#3: Identify dependencies

Load Application Component 1 met

ric
k clusters rela
tion
N metrics
k << N

...
Call graph Metrics time series Component 2 Component 3

Figure 1. High level steps of Sieve.

goal. Our goal is to design a generic platform that can be any modifications to the application code. In addition, Sieve records
used for a wide-range of workflows. all exposed metrics by all components. Note that this recording
• Automatic: The sheer number of metrics prohibits manual only happens during the creation of the call graph and not during
inspection. On the other hand, designing a generic system runtime.
to help developers in many use cases might require man-
Step #2: Reduce metrics. After collecting the metrics, Sieve ana-
ually adjusting some parameters for each use case. Our
lyzes each component and organizes its metrics into fewer groups
tool should be as automated as possible while reducing the
via clustering, so that similar-behaving metrics are clustered to-
number of metrics and extracting their relationships. How-
gether. After clustering, Sieve picks a representative metric from
ever, we leave the utilization of our platform’s output to the
each cluster. These representative metrics as well as their clusters
developers, who may have different goals.
in a sense characterize each component.
• Efficient: Our platform’s operation should be as efficient
as possible. Minimizing analysis time becomes important Step #3: Identify dependencies. In this step, Sieve explores the
when considering distributed systems, such as microservices- possibilities of one component’s representative metrics affecting
based applications. another component’s metrics using a pairwise comparison method:
each representative metric of one component is compared with each
Assumptions. While developing Sieve, we made two assumptions.
representative metric of another component. Sieve uses the call
• We assume that the developers can supply a workload gen- graph obtained in Step 1 to choose the components to be compared
erator for the application under investigation. This assump- (i.e., components directly communicating) and the representative
tion should be reasonable and less demanding for developers metrics determined in Step 2. As a result, the search space is signif-
than fully instrumenting each component and/or figuring icantly reduced compared to the naïve approach of comparing all
out relationships across all metrics. components with every other component using all metrics.
• It is possible for specific use cases to have additional assump- If Sieve determines that there is a relationship between a metric
tions. For example, a root cause analysis scenario would of one component and another metric of another component, a
require knowing a faulty and non-faulty (correct) version dependency edge between these components is created using the
of the application. corresponding metrics. The direction of the edge depends on which
component is affecting the other.
2.3 Sieve Overview
The underlying intuition behind Sieve is two-fold: Firstly, in the 2.4 Potential Use Cases
metric dimension, some metrics of a component may behave with
We envision Sieve can be useful to the developers or operators
similar patterns as other metrics of that component. Secondly, in
of distributed systems to build a wide-range of management tools
the component dimension, there are dependencies between com-
to improve the efficiency, reliability, and resiliency of distributed
ponents. As a result, monitoring all metrics of all components at
systems. Each of these cases might require some tweaks and specific
runtime may be unnecessary and inefficient (as components are
knowledge about the application. Nevertheless, we think that the
not independent).
output of Sieve can be a starting point. In Section 4, we showcase
In this paper, we present Sieve to reduce this complexity by sys-
two of these use cases with two different applications.
tematically analyzing the application to filter collected metrics and
to build a dependency graph across components. To showcase the Orchestration and autoscaling of components. The pairwise
generality of this dependency graph and its benefits, we then utilize investigation of representative metrics of components produces the
Sieve to orchestrate autoscaling of the ShareLatex [23] application— dependencies across components. By definition, the dependency
an online collaboration tool, and to perform Root Cause Analysis graph shows the order of bottlenecked components. As a result of
(RCA) in OpenStack [28]—a cloud management software (§4). this graph, developers can have a better understanding of which
At a high level, Sieve’s design follows three steps as shown in components need to be scaled out first, meaning that the number
Figure 1. of monitored components can be reduced. Furthermore, the depen-
Step #1: Load the application. Sieve uses an application-specific dencies show the metrics that are affected, meaning that one only
load generator to stress the application under investigation. This needs to monitor a limited set of metrics rather than every metric
load generator can be provided by the application developers. For exported by these components. In combination, these reductions
example, OpenStack already uses a load generator named Rally are reflected in the monitoring overhead. Also, scaling decisions
[21]. During the load, Sieve records the communications among are based on fewer components and metrics, so that potentially
components to obtain a call graph. This recording does not require conflicting scaling decisions can be avoided.
3
Root cause analysis. It is possible that updates can introduce bugs interfaces similar to remote procedure calls. We model these com-
and problems into the application. Changing dependency graphs munications between the components as a directed graph, where
(i.e., after updates) may indicate that the problem got introduced the vertices represent the microservice components and the edges
during a particular update that caused the dependency graph to point from the caller to the callee providing the service.
change. Identifying such changes will be useful in debugging the By knowing which components communicate directly, we can
application by pointing to the root cause of the problem. reduce the number of component pairs we need to check to see
Performance diagnosis. Similar to the root cause analysis sce- whether they have a relation (see Section 3.3). Although it is pos-
nario, the resource-usage profile of a component may change after sible to manually track this information for smaller-sized applica-
an update. Even if the interfaces between the components may tions, this process becomes quickly difficult and error-prone with
stay the same, the update may address a previously unknown bug, increasing number of components.
introduce a new feature and/or implement the same functionality There are several ways to understand which microservice com-
in a more efficient and/or a different way. Such changes can have ponents are communicating with each other. One can instrument
effects on other parts of the application, and the complexity of the the application, so that each request can be traced from the point it
application may render these effects not easily foreseeable. The enters the application to the point where the response is returned
dependency graph can be utilized to understand the overall effect to the user. Dapper [84] from Google and Atlas [48, 61] from Netflix
on the application the update can have. rely on instrumenting their RPC middleware to trace requests.
Another method to obtain communicating components is to mon-
Placement decisions. When scaling decisions are made, the resource- itor network traffic between hosts running those components using
usage profiles of components become important, because compo- a tool like tcpdump. After obtaining the traffic, one can map the
nents with similar resource-usage profiles may create contention exchanged packets to the components via their source/destination
(e.g., two components may be CPU-bound). As a result, placing a addresses. This method can produce communicating component
new instance of a component on a computer where other instances pairs by parsing all network packets, adding significant computa-
of the same component or instances of another component with tional overhead and increasing the analysis time. Furthermore, it
a similar resource-usage profile run may not yield the additional is possible that many microservice components are deployed onto
benefit of the extra instance. The dependency graph lists which the same host (e.g., using containers), making the packet parsing
metrics are affected in a component under load. This information difficult due to network address translation on the host machine.
can be used to extract resource-usage profiles of components to One can also observe system calls related to network opera-
avoid such contention during scale-up events. tions via APIs such as ptrace() [31]. However, this approach adds a
lot of context switches between the tracer and component under
3 Design observation.
In this section, we detail the three steps of Sieve. Sieve employs sysdig to obtain the communicating pairs. sys-
dig[24] is a recent project providing a new method to observe
3.1 Load the Application system calls in a more efficient way. Utilizing a kernel module, sys-
For our systematic analysis, we first run the application under dig provides system calls as an event stream to a user application.
various load conditions. This loading serves two purposes: First, The event stream also contains information about the monitored
the load exposes a number of metrics from the application as well processes, so that network calls can be mapped to microservice
as the infrastructure it runs on. These metrics are then used to components, even if they are running in containers. Furthermore,
identify potential relationships across components. Second, the it enables extraction of the communication peer via user-defined
load also enables us to obtain a call graph, so that we can identify filters. Employing sysdig, we avoid the shortcomings of the above
the components that communicate with each other. The call graph is approaches: 1) We do not need to instrument the application, which
later used to reduce the amount of computation required to identify makes our system more generally applicable, 2) We add little over-
the inter-component relationships (§3.3). The load test is intended head to obtain the call graph of an application for our analysis (see
to be run in an offline step and not in production. Section 6.1.3).

Obtaining metrics. During the load of the application, we record 3.2 Reduce Metrics
metrics as time series. There are two types of metrics that we can
The primary goal of exporting metrics is to understand the per-
leverage for our analysis: First, there are system metrics that are
formance of applications, orchestrating them and debugging them.
obtained from the underlying operating system. These metrics
While the metrics exported by the application developers or commonly-
report the resource usage of a microservice component, and are
used microservice components may be useful for these purposes,
usually related to the hardware resources on a host. Examples
it is often the case that the developers have little idea regarding
include usages in CPU, memory, network and disk I/O.
which ones are going to be most useful. Developers from different
Second, there are application-level metrics. Application devel-
backgrounds may have different opinions: a developer specializing
opers often add application-specific metrics (e.g., number of active
in network communications may deem network I/O as the most im-
users, response time of a request in a component). Commonly-used
portant metric to consider, whereas a developer with a background
components (e.g., databases, load balancers) and certain language
on algorithms may find CPU usage more valuable. As a result of
runtimes (e.g., Java) may provide statistics about specific operations
these varying opinions, often times many metrics are exported.
(e.g., query times, request counts, duration of garbage collection).
While it may look like there is no harm in exporting as much in-
Obtaining the call graph. Generally speaking, applications us- formation as possible about the application, it can create problems.
ing a microservices architecture communicate via well-defined Manually investigating the obtained metrics from a large number
4
of components becomes increasingly difficult with the increasing in one component may not reflect on another component’s met-
number of metrics and components [38]. This complexity reflects rics immediately (e.g., due to the network delay of calls between
on the decisions that are needed to control and maintain the ap- components).
plication. In addition, the overhead associated with the collection Additionally, k-Shape is robust against distortion in amplitude
x −µ
and storage of these metrics can quickly create problems. In fact, because data is normalized via z-normalization (z = σ ) before
Amazon CloudWatch [2] charges its customers for the reporting of being processed. This feature is especially important because differ-
the metrics they export. As a result, the more metrics an application ent metrics may have different units and thus, may not be directly
has to export, the bigger the cost the developers would have to comparable.
bear. k-Shape works by initially assigning time series to clusters ran-
One observation we make is that some metrics strongly correlate domly. In every iteration, it computes new cluster centroids ac-
with each other and it might not be necessary to consider all of cording to SBD with the assigned time series. These centroids are
them when making decisions about the control of the application. then used to update the assignment for the next iteration until the
For example, some application metrics might be strongly correlated clusters converge (i.e., the assignments do not change).
with each other due to the redundancy in choosing which metrics to We make three adjustments to employ k-Shape in Sieve. First, we
export by the developers. It is also possible that different subsystems preprocess the collected time series to be compatible with k-Shape.
in the same component report similar information (e.g., overall k-Shape expects the observations to be equidistantly distributed
memory vs. heap usage of a process). In addition, some system in the time domain. However, during the load of the application,
metrics may offer clues regarding the application’s state: increased timeouts or lost packets can cause gaps between the measurements.
network I/O may indicate an increase in the number of requests. To reconstruct missing data, we use spline interpolation of the
The direct outcome of this observation is that it should be possi- third order (cubic). A spline is defined piecewise by polynomial
ble to reduce the dimensionality of the metrics the developers have functions. Compared to other methods such as averages of previous
to consider. As such, the procedure to enable this reduction should values or linear interpolation, spline interpolation provides a higher
happen with minimal user effort and scale with increased numbers degree of smoothness. It therefore introduces less distortion to
of metrics. the characteristics of a time-series [69]. Additionally, monitoring
To achieve these requirements, Sieve uses a clustering approach systems retrieve metrics at different points in time and need to be
named k-Shape [76] with a pre-filtering step. While other approaches discretized to match each other. In order to increase the matching
such as principal component analysis (PCA) [52] and random pro- accuracy, we discretize using 500ms instead of the original 2s used
jections [75] can also be used for dimensionality reduction, these in the original k-Shape paper [76].
approaches either produce results that are not easily interpreted Our second adjustment is to change the initial assignments
by developers (i.e., PCA) or sacrifice accuracy to achieve perfor- of metric time series to clusters. To increase clustering perfor-
mance and have stability issues producing different results across mance and reduce the convergence overhead, we pre-cluster met-
runs (i.e., random projections). On the other hand, clustering re- rics according to their name similarity (e.g., Jaro distance [63])
sults can be visually inspected by developers, who can also use any and use these clusters as the initial assignment instead of the de-
application-level knowledge to validate their correctness. Addition- fault random assignment. This adjustment is reasonable given that
ally, clustering can also uncover hidden relationships which might many developers use naming conventions when exporting met-
not have been obvious. rics relating to the same component or resource in question (e.g.,
Filtering unvarying metrics. Before we use k-Shape, we first “cpu_usage", “cpu_usage_percentile"). The number of iterations to
filter metrics with constant trend or low variance (var ≤ 0.002). converge should decrease compared to the random assignment,
These metrics cannot provide any new information regarding the because similar names indicate similar metrics. Note that this ad-
relationships across components, because they are not changing justment is only for performance reasons; the convergence of the
according to the load applied to the application. Removing these k-Shape clustering does not require any knowledge of the vari-
metrics also enables us to improve the clustering results. able names and would not be affected even with a random initial
assignment.
k-Shape clustering. k-Shape is a recent clustering algorithm that During the clustering process, k-Shape requires the number of
scales linearly with the number of metrics. It uses a novel distance clusters to be previously determined. In an application with sev-
metric called shape-based distance (SBD). SBD is based on a nor- eral components, each of which having various number of metrics,
malized form of cross correlation (NCC) [76]. Cross correlation is pre-determining the ideal number of clusters may not be straight-
calculated using Fast Fourier Transformation and normalized using forward. Our final adjustment is to overcome this limitation: we
the geometric mean of the autocorrelation of each individual met- iteratively vary the number of clusters used by k-Shape and pick
ric’s time series. Given two time series vectors, x⃗ and y⃗, SBD will the number that gives the best silhouette value [81], which is a
take the position w, when sliding x⃗ over y⃗, where the normalized technique to determine the quality of the clusters. The silhouette
cross correlation maximizes. value is −1 when the assignment is wrong and 1 when it is a per-
SBD(⃗
x , y⃗ ) = 1 − maxw (NCCw (⃗
x , y⃗ )) (1) fect assignment [32]. We use the SBD as a distance measure in the
silhouette computation.
Because k-Shape uses a distance metric based on the shape of In practice, experimenting with a small number of clusters is
the investigated time series, it can detect similarities in two time sufficient. For our applications, seven clusters per component was
series, even if one lags the other in the time dimension. This fea- sufficient, where each component had up to 300 metrics.
ture is important to determine relationships across components in
microservices-based applications because a change in one metric
5
Representative metrics. After the clustering, each microservice typically run in the same data center and their components com-
component will have one or more clusters of metrics. The number of municate over a LAN, where typical round-trip times are in the
clusters will most likely be much smaller than the number of metrics order of milliseconds. Sieve uses a conservative delay of 500ms for
belonging to that component. Once these clusters are obtained, unforeseen delays.
Sieve picks one representative metric from each cluster. To pick the To apply the Granger Causality tests and check whether the past
representative metric from each cluster, Sieve determines the SBD values of metric X can predict the future values of metric Y , both
between each metric and the corresponding centroid of the cluster. models are compared via the F-test [70]. The null hypothesis (i.e.,
The metric with the lowest metric is chosen as the representative X does not granger-cause Y ) is rejected if the p-value is below a
metric for this cluster. critical value.
The high-level idea is that the behavior of the cluster will match However, one has to consider various properties of the time se-
this representative metric; otherwise, the rest of the metrics in the ries. For example, the F-test requires the time series to be normally
cluster would not have been in the same cluster as this metric. The distributed. The load generation used in Section 3.1 can be adjusted
set of representative metrics of a component can then be used to to accommodate this requirement. Also, the F-test might find spuri-
describe a microservice component’s behavior. These representative ous regressions when non-stationary time series are included [56].
metrics are then used in conjunction with the call graph obtained Non-stationary time series (e.g., monotonically increasing counters
in Section 3.1 to identify and understand the relationships across for CPU and network interfaces) can be found using the Augmented
components. Dickey-Fuller test [58]. For these time series, the first difference
is taken and then used in the Granger Causality tests. Although
3.3 Identify Dependencies longer trends may be lost due to the first difference, accumulating
To better understand an application, we need to find dependencies metrics such as counters do not present interesting relationships
across its components. A naïve way of accomplishing this goal for our purposes.
would be to compare all components with each other using all After applying the Granger Causality test to each component’s
possible metrics. One can clearly see that with the increasing num- representative metrics with its neighbouring component’s repre-
ber of components and metrics, this would not yield an effective sentative metrics, we obtain a graph. In this graph, we draw an edge
solution. between microservice components, if one metric in one component
In the previous section, we described how one can reduce the Granger-causes another metric in a neighbouring component. This
number of metrics one has to consider in this pairwise comparison edge represents the dependency between these two components
by clustering and obtaining the representative metrics of each com- and its direction is determined by Granger causality.
ponent. Still, comparing all pairs of components using this reduced While Granger Causality tests are useful in determining predic-
set of metrics may be inefficient and redundant considering the tive causality across microservice components, it has some limi-
number of components in a typical microservices-based application tations that we need to consider. For example, it does not cover
(e.g., tens or hundreds). instantaneous relationships between two variables. More impor-
Sieve uses the call graph obtained in Section 3.1 to reduce the tantly, it might reveal spurious relationships, if important variables
number of components that need to be investigated in a pairwise are missing in the system: if both X and Y depend on a third variable
fashion. For each component, we do pairwise comparisons using Z that is not considered, any relationship found between X and Y
each representative metric of its clusters with each of its neighbour- may not be useful. Fortunately, an indicator of such a situation is
ing components (i.e., callees) and their representative metrics. that both metrics will Granger-cause each other (i.e., a bidirectional
Sieve utilizes Granger Causality tests [57] in this pairwise com- edge in the graph). Sieve filters these edges out.
parison. Granger Causality tests are useful in determining whether
a time series can be useful in predicting another time series: In a 4 Applications
microservices-based application, the component interactions closely In this section, we describe two use cases to demonstrate Sieve’s
follow the path a request takes inside the application. As a result, ability to handle different workflows. In particular, using Sieve’s
these interactions can be predictive of the changes in the metrics base design, we implemented 1) an orchestration engine for au-
of the components in the path. Granger Causality tests offer a sta- toscaling and applied it to ShareLatex [23], and 2) a root cause
tistical approach in understanding the relationships across these analysis (RCA) engine and applied it to OpenStack [19].
components. Informally, Granger Causality is defined as follows.
If a metric X is Granger-causing another metric Y, then we can 4.1 Orchestration of Autoscaling
predict Y better by using the history of both X and Y compared to For the autoscaling case study, we used ShareLatex [23]—a popular
only using the history of Y [54]. collaborative LaTeX editor. ShareLatex is structured as a microservices-
To utilize Granger Causality tests in Sieve, we built two linear based application, delegating tasks to multiple well-defined compo-
models using the ordinary least-square method [35]. First, we com- nents that include a KV-store, load balancer, two databases and 11
pare each metric X t with another metric Yt . Second, we compare node.js based components.
each metric X t with the time-lagged version of the other metric Yt : Sieve’s pairwise investigation of representative metrics of com-
Yt −Laд . Covering the cases with a time lag is important because the ponents produces the dependencies across components. By leverag-
load in one component may not be reflected on another component ing this dependency graph, our autoscaling engine helps developers
until the second component receives API calls and starts processing to make more informed decisions regarding which components and
them. metrics are more critical to monitor. As a result, developers can
Sieve utilizes short delays to build the time-lagged versions generate scaling rules with the goal of adjusting the number of
of metrics. The reason is that microservices-based applications active component instances, depending on real-time workload.
6
 Get dep. graphs Extract new and Calculate cluster Filter edges by
0 1 3 4 5 Final rankings
& cluster metadata discarded metrics novelty and similarity novelty and similarity
C
Correct Faulty Rank components Novelty Similarity A C New edge
2 A
B B by novelty 2 2 0 0 0 ½ 1 1 0 0 0 B C Discarded
B
C x
A C A C A B A B Unchanged
metric
2 1 3 Legend: component cluster

Figure 2. Sieve’s root cause analysis methodology.

More specifically, we use Sieve’s dependency graph and ex- Table 2. Description of dependency graph differences considered
tract (1) guiding metrics (i.e., metrics to use in a scaling rule), (2) by the root cause analysis engine.
scaling actions (i.e., actions associated with reacting to varying
Scoping level Differences of interest
loads by increasing/decreasing the number of instances subject
to minimum/maximum thresholds), and (3) scaling conditions (i.e., Present in F version, not in C (new)
Component metrics
Present in C version, not in F (discarded)
conditions based on a guiding metric triggering the corresponding
scaling action). Below, we explain how we use Sieve to generate a Clusters Cluster includes new/discarded metrics
scaling rule: New/discarded edge between similar clusters
Dep. graph edges Different time-lag between similar clusters
#1: Metric. We pick a metric m that appears the most in Granger Includes clusters w/ new/discarded metrics
Causality relations between components.
#2: Scaling actions. In our case study, we restrict scaling actions
to scale in/out actions, with increments/decrements of a single metric list shows the metrics in that component potentially related
component instance (+/−1). to the issue, providing a more fine-grained view. With the help of
this list, developers can reduce the complexity of their search for
#3: Conditions. The scale in/out thresholds are defined from the
the root cause.
values of m according to a Service Level Agreement (SLA) condition.
Figure 2 shows the five steps involved in the comparison. At
For ShareLatex, such an SLA condition can be to keep 90% of all
each step, we extract and analyze Sieve’s outputs at three different
request latencies below 1000ms. The thresholds for m are iteratively
granularity levels: metrics, clusters, and dependency graph edges.
refined during the application loading phase.
The levels and corresponding differences of interest are described
4.2 Root Cause Analysis in Table 2. We describe the steps in more detail below.
#1: Metric analysis. This step analyzes the presence or absence
For the root cause analysis (RCA) case study, we used OpenStack [19,
of metrics between C and F versions. If a metric m is present in
28], a popular open-source cloud management software. OpenStack
both C and F, it intuitively represents the maintenance of healthy
is structured as a microservices-based application with a typical
behavior associated with m. As such, these metrics are filtered out
deployment of ∼10 (or more) individual components, each often
of this step. Conversely, the appearance of a new metric (or the
divided into multiple sub-components [83]. Due to its scale and
disappearance of a previously existing metric) between versions is
complexity, OpenStack is susceptible to faults and performance
likely to be related with the anomaly.
issues, often introduced by updates to its codebase.
In microservices-based applications such as Openstack, com- #2: Component rankings. In this step, we use the results of step
ponents can be updated quite often [62], and such updates can 1 to rank components according to their novelty score (i.e., total
affect other application components. If relationships between com- number of new or discarded metrics), producing an initial group of
ponents are complex, such effects may not be easily foreseeable, interesting components for RCA.
even when inter-component interfaces are unchanged (e.g., if the #3: Cluster analysis: novelty & similarity. Clusters aggregate
density of inter-component relationships is high or if the activation component metrics which exhibit similar behavior over time. The
of relationships is selective depending on the component’s state clusters with new or discarded metrics should be more interesting
and inputs). Sieve’s dependency graph can be used to understand for RCA compared to the unchanged clusters of that component
the update’s overall effect on the application: changing dependency (with some exceptions, explained below). For a given component,
graphs can indicate potential problems introduced by an update. we compute the novelty scores of its clusters as the sum of the num-
By identifying such changes, Sieve can help developers identify ber of new and discarded metrics, and produce a list of {component,
the root cause of the problem. metric list} pairs, where the metric list considers metrics from the
Our RCA engine leverages Sieve to generate a list of possible clusters with higher novelty scores.
root causes of an anomaly in the monitored application. More In addition, we track the similarity of a component’s clusters
specifically, the RCA engine compares the dependency graphs of between C and F versions (or vice-versa). This is done to identify
two different versions of an application: (1) a correct version; and two events: (1) appearance (or disappearance) of edges between
(2) a faulty version. Similarly to [64, 66], we assume that the system versions; and (2) attribute changes in relationships maintained
anomaly (but not its cause) has been observed and the correct and between C and F versions (e.g., a change in Granger causality
faulty versions have been identified. The result of this comparison time lag). An edge between clusters x and y (belonging to com-
is a list of {component, metric list} pairs: the component item points ponents A and B, respectively) is said to be ‘maintained between
to a component as a possible source for the issue, whereas the versions’ if their respective metric compositions do not change
7
 AMI(1, 2) AMI(1, 3) AMI(2, 3)
0.8 1.0
0.8

0.8
0.6
0.6
AMI 0.6

AMI

AMI
0.4 0.4
0.4

0.2 0.2
0.2

0.0 0.0 0.0

t
con clsi
doc ts
doc pd.
hap ore

spe dis

t
con clsi

t
con clsi
pos oxy
rea gres
e

g
trac tags

web
h.

doc ts
doc pd.
hap ore

spe dis
pos oxy
rea gres
e

g
trac tags

web
h.

doc ts
doc pd.
hap ore
pos oxy
rea gres
e
spe dis
g
trac tags

web
h.
cha

cha

cha
l-tim

l-tim

l-tim
llin

llin

llin
tac

tac

tac
k-c

k-c

k-c
re

re

re
st

st

st
-u

-u

-u
r

r
t

t
Figure 3. Pairwise adjusted mutual information (AMI) scores between 3 measurements.
A ) ≈ S (M A )
significantly between C and F versions, i.e. if S (Mx,C generator using Locust [11], a Python-based distributed load gen-
x ′,F
and S (My,CB ) ≈ S (M B ). M A and M A are the metric compo- eration tool to simulate virtual users in the application (1, 041 LoC).
y ′,F x,C x ′,F
For OpenStack, we used Rally [21], the official benchmark suite
sitions of clusters x and x ′ of component A, in the C and F versions,
from OpenStack.
respectively. S is some measure of cluster similarity (defined below).
For metric collection, Sieve uses Telegraf [25] to collect appli-
Both events – (1) and (2) – can be an indication of an anomaly, be-
cation/system metrics and stores them in InfluxDB [7]. Telegraf
cause one would expect edges between clusters with high similarity
seamlessly integrates with InfluxDB, supports metrics of commonly-
to be maintained between versions.
used components (e.g., Docker, RabbitMQ, memcached) and can
We compute the cluster similarity score, S, according to a modified
run custom scripts for collection of additional metrics exposed by
form of the Jaccard similarity coefficient
application APIs (e.g., [20]). With this setup, Sieve can store any
A ∩ MA |
|Mi,C time-series metrics exposed by microservice components.
j,F
S= A |
(2) For the call graph extraction, Sieve leverages sysdig call tracer [24]
|Mi,C
to obtain which microservice components communicate with each
To eliminate the penalty imposed by new metrics added to the other. We wrote custom scripts to record network system calls
faulty cluster, we only consider the contents of the correct cluster with source and destination IP addresses on every machine hosting
in the denominator (instead of the union of Mi,CA and M A ). the components (457 LoC). These IP addresses are then mapped
j,F
to the components using the cluster manager’s service discovery
#4: Edge filtering. To further reduce the list of {component, metric
mechanism.
list} pairs, we examine the relationships between components and
We implemented Sieve’s data analytics techniques in Python
clusters identified in steps 2 and 3. We identify three events:
(2243 LoC) including metric filtering, clustering based on k-Shape,
1. Edges involving (at least) one cluster with a high novelty and Granger Causality. The analysis can also be distributed across
score multiple machines for scalability.
2. Appearance or disappearance of edges between clusters Lastly, we also implemented two case studies based on the Sieve
with high similarity infrastructure: autoscaling in ShareLatex (720 LoC) and RCA in
3. Changes in time lag in edges between clusters with high OpenStack (507 LoC). For our autoscaling engine, we employed
similarity Kapacitor [9] to stream metrics from InfluxDB in real-time and to
Event 1 isolates metrics related to edges which include at least install our scaling rules using its user-defined functions. For the
one ‘novel’ cluster. Events 2 and 3 isolate clusters which are main- RCA engine, we implemented two modules in Python: one module
tained between C and F versions, but become interesting for RCA extracts metric clustering data (125 LoC) and the other module (382
due to a change in their relationship. Novelty and similarity scores LoC) compares clustering data and dependency graphs.
are computed as in step 3. We define thresholds for ‘high’ novelty
and similarity scores.
6 Evaluation
#5: Final rankings. We present a final list of {component, metric
Our evaluation answers the following questions:
list} pairs. The list is ordered by component, following the rank
given in step 2. The metric list items include the metrics identified 1. How effective is the general Sieve framework? (§6.1)
at steps 3 and 4. 2. How effective is Sieve for autoscaling? (§6.2)
3. How effective is Sieve for root cause analysis? (§6.3)
5 Implementation
We next describe the implementation details of Sieve. Our system
6.1 Sieve Evaluation
implementation, including used software versions, is published at
https://sieve-microservices.github.io. For load generation, Sieve Before we evaluate Sieve with the case studies, we evaluate Sieve’s
requires an application-specific load generator. We experimented general properties: (a) the robustness of clustering; (b) the effective-
with two microservices-based applications: ShareLatex [23] and ness of metric reduction; and (c) the monitoring overhead incurred
OpenStack [19, 28]. For ShareLatex, we developed our own load by Sieve’s infrastructure.
8
 Before clustering After clustering
Number of metrics
102

Time to complete
0.4

10k requests [s]

101
0.2

100
0.0
native sysdig tcpdump
t
con lsi

s
doc cts
doc d.
files re
hap re
y
p os o db

spe s
rea sql
e

lling
cha

tag
l-tim
rox

red
sto
to
-up
c

tgre
ta

ng
mo

Figure 5. Completion time for HTTP requests when using tcpdump,

Figure 4. Average no. of metrics after Sieve’s reduction. sysdig or native (i.e., no monitoring).

Table 3. InfluxDB overhead before Sieve’s reduction of metrics.

Experimental setup. We ran our measurements on a 10 node
cluster, every node with a 4-core Xeon E5405 processor, 8 GB DDR2- Metric Before After Reduction
RAM and a 500GB HDD. For the general experiments, we loaded CPU time [s] 0.45G 0.085G 81.2 %
ShareLatex using Sieve five times with random workloads. The DB size [KB] 588.8 36.0 93.8 %
random workloads also help to validate whether the model stays Network in [MB] 11.1 2.3 79.3 %
consistent, if no assumption about the workload is made. Network out [KB] 15.1 7.4 50.7 %

6.1.1 Robustness reducing the number of metrics, we compare the results of the
We focus on two aspects to evaluate Sieve’s robustness. First, we in- clustering with the actual number of metrics in the application. We
vestigate the consistency of clustering across different runs. Second, identified 889 unique metrics within ShareLatex, meaning that an
we try to validate whether the metrics in a cluster indeed belong operator would have to understand and filter these metrics. Sieve’s
together. clustering reduces this number to 65 (averaged across five runs).
Consistency. To evaluate consistency, we compare cluster assign- Figure 4 shows the reduction in the number of metrics for the indi-
ments produced in different measurements. A common metric to vidual components in ShareLatex. Note that this measurement is
compare cluster assignments is Adjusted Mutual Information (AMI) with high Silhouette scores for the clusters, which implies that the
score [88]. AMI is normalized against a random assignment and metrics reduction does not affect the quality of the clusters.
ranges from zero to one: If AMI is equal to one, both clusters match
6.1.3 Monitoring Overhead
perfectly. Random assignments will be close to zero.
Figure 3 shows the AMI of cluster assignments for individual We evaluate Sieve’s overhead based on two aspects. First, we com-
components for three independent measurements. To reduce the pare different techniques for obtaining the call graph and show how
selection bias we apply randomized workload in a controlled envi- our approach fairs. Second, we investigate the overhead incurred
ronment. As a result, they should constitute a worst-case perfor- by the application by comparing the monitoring overhead with and
mance for the clustering. Our measurements show that the average without using Sieve.
AMI is 0.597, which is better than random assignments. Based on Overheads. To measure the monitoring overhead during the load-
these measurements, we conclude the clusterings are consistent. ing stage, we run an experiment with 10K HTTP requests for a
Validity. To evaluate the validity of the clusters, we choose three small static file using Apache Benchmark [12] on an Nginx web
criteria: (1) Is there a common visible pattern between metrics in server [15]. Because the computational overhead for serving such
one cluster? (2) Do metrics in a cluster belong together assuming a file is low, this experiment shows the worst-case performance
application knowledge? (3) Are the shape-based distances between for sysdig and tcpdump. Figure 5 shows the time it takes to com-
metrics and their cluster centroid below a threshold (i.e., 0.3)? plete the experiment. While tcpdump incurs a lower overhead than
We choose three clusters with different Silhouette scores (high, sysdig (i.e., 7% vs. 22%), it provides less context regarding the com-
medium, low). According to the above criteria, we conclude the ponent generating the request and requires more knowledge about
clustering algorithm can determine similar metrics. For example, the network topology to obtain the call graph. sysdig provides all
application metrics such as HTTP request times and correspond- this information without much additional overhead.
ing database queries are clustered together. Similar to consistency, Gains. To show the gains during the runtime of the application after
higher Silhouette scores indicate that the clusters are more mean- using Sieve, we compare the computation, storage and network
ingful and potentially more useful for the developers. We omit the usage for the metrics collected during the five measurements. We
details for brevity. store all collected metrics in InfluxDB and measure the respective
resource usage. We then repeat the same process using the metrics
6.1.2 Effectiveness found by Sieve; thus, simulating a run with the reduced metrics.
The purpose of clustering is to reduce the number of metrics ex- Table 3 shows the relative usage of the respective resources with
posed by the system without losing much information about the Sieve. Sieve reduces the monitoring overhead for computation,
system behavior. To evaluate how effective our clustering is in storage and network by 80%, 90% and 50%, respectively.
9
Table 4. Comparison between a traditional metric (CPU usage) and postgresql
Sieve’s selection when used as autoscaling triggers.
tags
Metric CPU usage Sieve Difference [%]
contacts
Mean CPU usage per com- 5.98 9.26 +54.82 clsi spelling
filestore
ponent
SLA violations (out of 1400 188 70 -62.77
samples)
Number of scaling actions 32 21 -34.38 web
haproxy
docstore
mongodb
chat

6.2 Case-study #1: Autoscaling

We next evaluate the effectiveness of Sieve for the orchestration
of autoscaling in microservices. doc-updater real-time
track-changes
Experimental setup. For the autoscaling case study, we used
ShareLatex [23] (as described in §4.1). We used 12 t2.large VM- redis

Instances on Amazon EC2 with 2 vCPUs, 8GB RAM and 20 GB

Amazon EBS storage. This number of instances were sufficient to Figure 6. Relations between components based on Granger Causal-
stress-test all components of the application. The VM instances ity in ShareLatex. The dashed lines denote relationships with metric
were allocated statically during experiments as Docker containers. http-requests_Project_id_GET_mean.
We created a Docker image for each ShareLatex component and
used Rancher [22] as the cluster manager to deploy our containers
across different hosts. ‘faulty’ code versions. Moreover, these showcase Sieve’s effective-
ness while analyzing two different types of bugs: (1) a crash in the
Dataset. We used a HTTP trace sample from soccer world cup
case of #1533942; and (2) performance regression (e.g. increase in
1998 [6] for an hour long trace. Note that the access pattern and re-
latency) in the case of #1590179. We compare the documented root
quested resources in the world cup trace differs from the ShareLatex
causes to the lists of root causes produced by our RCA engine.
application. However, we used the trace to map traffic patterns for
our application to generate a realistic spike workload. In particular, 6.3.1 Bug 1533942 : Failure to launch a VM
sessions in the HTTP trace were identified by using the client IP.
Afterwards, we enqueued the sessions based on their timestamp, Bug #1533942 manifests itself as follows: when launching a new
where a virtual user was spawned for the duration of each session VM instance using the command line interface, one gets the error
and then stopped. message ‘No valid host was found. There are not enough hosts avail-
able.’ despite the availability of compute nodes. Without any other
Results. We chose an SLA condition, such that 90th percentile of directly observable output, the instance falls into ‘ERROR’ state and
all request latencies should be below 1000ms. Traditional tools, such fails. A similar failure is used as a representative case in [55, 83].
as Amazon AWS Auto Scaling [1], often use the CPU usage as the
Root cause. The failure is caused by the crash of an agent in the
default metric to trigger autoscaling. Sieve identified an application
Neutron component, namely the Open vSwitch agent. The Open
metric named http-requests_Project_id_GET_mean (Figure 6) as a
vSwitch agent is responsible for setting up and managing virtual
better metric for autoscaling than CPU usage.
networking for VM instances. The ultimate cause is traced to a
To calculate the threshold values to trigger autoscaling, we used
configuration error in OpenStack Kolla’s deployment scripts [29].
a 5-minute sample from the peak load of our HTTP trace and
iteratively refined the values to stay within the SLA condition. As Experimental setup. We deployed OpenStack components as con-
a result, we found that the trigger thresholds for scaling up and tainerized microservices using Kolla [28]. We configured Kolla to
down while using the CPU usage metric should be 21% and 1%, deploy 7 main OpenStack components along with 12 auxiliary
respectively. Similarly, for http-requests_Project_id_GET_mean, the components. Some components can be sub-divided in several mi-
thresholds for scaling up and down should be 1400ms and 1120ms, croservices and replicated among deployment hosts, for a total of 47
respectively. microservices. This information is summarized in Table 5. We use
After installing the scaling actions, we ran our one-hour trace. OpenStack’s telemetry component (Ceilometer) to expose relevant
Table 4 shows the comparison when using the CPU usage and OpenStack-related metrics and extract them via Telegraf.
http-requests_Project_id_GET_mean for the scaling triggers. When The Openstack versions used for the correct (C) and faulty (F)
Sieve’s selection of metric was used for autoscaling triggers, the versions are listed in Table 8. The configurations for the Kolla
average CPU usage of each component was increased. There were deployments of each version are publicly available 1 , as well as the
also fewer SLA violations and scaling actions. monitoring infrastructure and evaluation scripts 2 .
The infrastructure consists of two m4.xlarge Amazon EC2 VM
6.3 Case-study #2: Root Cause Analysis instances to run OpenStack components (16 vCPUs, 64 GB RAM
and 20 GB Amazon EBS storage) and three t2.medium VM instances
To evaluate the applicability of Sieve to root cause analysis, we re-
(2 vCPUs, 4GB RAM and 20 GB EBS storage) for the supporting
produce two representative OpenStack anomalies, Launchpad bugs
components (measurement, database and deployment).
#1533942 [29] and #1590179 [30]. We selected these issues because
they have well-documented root causes, providing an appropriate 1 https://github.com/sieve-microservices/kolla
ground truth, and allowing for the identification of ‘correct’ and 2 https://github.com/sieve-microservices/rca-evaluation

10
 # components, clusters
80
a) Cluster novelty 60
b) Edge novelty c) # of components, clusters and metrics
67 400
50 New Components
# of clusters

and metrics
60

# of edges
40 Discarded 300 Clusters
40 30 27 Lag change 221 Metrics
25
200 163
19 20 Unchanged 121
20 10 13 11
10 5 4 2 6 100 68
5 4 2 1 2 2
1 0 0 1 13 29 10 16 7 10 3 5
0 0 0
New Discarded New Changed Total 0.0 0.5 0.6 0.7 0.0 0.5 0.6 0.7
and Similarity threshold Similarity threshold
Discarded
Figure 7. (a) Cluster novelty score. (b) Edge novelty score. (c) No. of components & clusters after edge filtering w/ varying thresholds.
New
Nova 2nd to the root cause. Utilizing different cluster similarity scores en-
Unchanged 4th libvirt ables developers to filter out some of the edges that may not be
Deleted 5 metrics
RabbitMQ
1st Lag ch. active_anon
relevant. Figures 7(b & c) show the effect of different cluster simi-
Nova 63 metrics
Neutron 3
rd
larity thresholds for all components in Table 7 when filtering edges.
20 metrics API messages
Server 22 metrics Without any similarity thresholds, there are 41 edges of interest,
messages_ack-diff
nova_instances_in_state_ERROR corresponding to a set of 13 components, 29 clusters and 221 met-
nova_instances_in_state_ACTIVE usage
neutron_ports_in_status_DOWN rics that might be relevant to the root cause (Figure 7(c)). A higher
neutron_ports_in_status_ACTIVE
threshold reduces the number of the {component, metrics list} pairs:
filtering out clusters with inter-version similarity scores below 0.50,
Figure 8. Final edge differences for RCA evaluation between top 5 there are 24 edges of interest, corresponding to 10 components, 16
components of Table 7 with similarity threshold of 0.50. clusters and 163 metrics.
Figure 8 shows the edges between the components at the top-
5 rows of Table 7, with a similarity threshold of 0.50. Note that
Results. We expect the RCA engine’s outcome to include Neutron one component (i.e., Nova scheduler) was removed by the similar-
component, along with metrics relating VM launches and network- ity filter. Another interesting observation is that one of the new
ing. The {component, metrics list} pairs with Neutron should be edges includes a Nova API component cluster, in which the nova-
ranked higher than others. instances-in-state-ACTIVE metric is replaced with nova-instances-in-
To generate load on OpenStack, we run the ‘boot_and_delete’ state-ERROR. This change relates directly to the observed anomaly
(B&D) task 100 times with the Rally benchmark suite [21], which (i.e., error in VM launch). The other end of this edge is a cluster
launches 5 VMs concurrently and deletes them after 15-25 seconds in the Neutron component, which aggregates metrics related to
(details about Rally tasks in Table 6). We apply this process to the VM networking, including a metric named neutron-ports-in-status-
correct (C) and faulty (F) versions of OpenStack. For the faulty ver- DOWN. This observation indicates a causal relationship between
sion, the task fails as described above. We then apply the remaining the VM failure and a VM networking issue, which is the true root
stages of Sieve and feed the output to the RCA engine. For both cause of the anomaly.
versions, the dependency graphs are composed by 16 components, We also note that similarity a high threshold may filter out use-
with 647 edges in the NF version, and 343 edges in the F version. ful information. For example, the Neutron component cluster with
Below, we summarize the findings of RCA steps. the neutron-ports-in-status-DOWN metric is removed with similar-
Steps #1 & #2: Metric analysis and component rankings. The ity thresholds above 0.60. We leave the study of this parameter’s
total number of unchanged metrics exceeds that of ‘novel’ metrics sensitivity to future work.
(i.e., new and/or discarded) by an order of magnitude. Furthermore,
Step #5: Final rankings. The rightmost column on Table 7 shows
the initial component novelty ranking puts the Nova and Neutron
the final rankings, considering edge filtering step with a 0.50 sim-
components (known to be directly related with the anomaly) within
ilarity threshold. Figure 8 shows a significant reduction in terms
the top 4 positions out of 16 (Table 7). This confirms the intuition
of state to analyze (from a total of 16 components and 508 metrics
behind our approach: novel metrics are more likely to be related to
to 10 and 163, respectively) because of the exclusion of non-novel
a failure.
clusters. For example, for Nova API, the number of metrics reduces
Step #3: Cluster novelty & similarity. Computing the cluster from 59 to 20, for Neutron server from 42 to 22. Furthermore, our
novelty scores shows that the novel metrics from step 1 are dis- method includes the Neutron component as one of the top 5 com-
tributed over only 27 of the 67 clusters (Figure 7(a)), even conserva- ponents, and isolates an edge which is directly related with the true
tively considering a cluster to be novel if it contains at least one new root cause of the anomaly.
or discarded metric. Considering only novel clusters reduces the
number of metrics and the number of edges for the developers to 6.3.2 Bug 1590179 : Fernet token performance regression
analyze for the root cause in step 4. We also compute the similarity
The main symptom of bug #1590179 is a general decrease in the rate
scores for these novel clusters and use the similarity in the next
at which Openstack processes user requests, in-between Openstack
step.
‘Liberty’ and ‘Mitaka’ releases.
Step #4: Edge filtering. By investigating the novel edges (i.e., new
or deleted) in the dependency graph, the developers can better Root cause. As reported in [30], the issue is due to a 5× increase in
focus on understanding which component might be more relevant authentication token validation time. The ultimate cause of the bug
11
Table 5. Components deployed by Openstack Kolla during RCA Table 6. Details about Rally tasks used in RCA evaluation.
evaluation.
Bug # Benchmark # Runs Concurr. Details
Component Purpose # Microservices boot and delete VMs up for
1533942 100 5
Nova VM computing 8 (B&D) 15-25 sec
Neutron VM networking 6 VMs up for
Keystone Identity 3 B&D 25 5
15-25 sec
Glance VM image manag. 2
Heat - 3 authenticate user
Horizon Web UI 1 and validate 100 5 -
Ceilometer Telemetry 5 token (AU&VT)
1590179
Heka Logging 3 create and delete
Cron Job scheduling 3 networks 50 5 -
Open vSwitch VM networking (aux.) 4 (C&DN)
Elasticsearch Search engine 1 create and delete Cirros 0.35
Kibana Data visualiz. 1 image (C&DI) 50 2 x86_64
Auth. token caching image
Memcached 1
(among others)
Openstack
Mariadb 1
parameter storage
RabbitMQ Message broker 1 Table 7. OpenStack components, sorted by the number of novel
MongoDB Ceilometer data storage 1 metrics between correct (C) and faulty (F) versions.
Telegraf Metric collection 1
InfluxDB 1
Metric storage Changed Total Final
PostgreSQL 1 Component
(New/Discarded) (per component) ranking
Totals - 47
Nova API 29 (7/22) 59 1
Nova libvirt 21 (0/21) 39 2
Nova scheduler 14 (7/7) 30 -
is a change in the token caching strategy in-between Openstack Neutron server 12 (2/10) 42 3
‘Liberty’ and ‘Mitaka’ releases. RabbitMQ 11 (5/6) 57 4
In the context of Openstack, tokens represent the authenticated Neutron L3 agent 7 (0/7) 39 5
identity of a specific requester (e.g. a system user) and grants au- Nova novncproxy 7 (0/7) 12 -
Glance API 5 (0/5) 27 6
thorization for a specific Openstack action (e.g. starting a VM) [10]. Neutron DHCP ag. 4 (0/4) 35 7
Openstack supports different types of tokens, but this issue is par- Nova compute 3 (0/3) 41 8
ticular to Fernet tokens [10], which do not require persistence in a Glance registry 3 (0/3) 23 9
database: validation is based on symmetric encryption, with secret Haproxy 2 (1/1) 14 10
Nova conductor 2 (0/2) 29 -
keys kept by the Openstack identity component. Other 3 components 0 (0/0) 59 -
Experimental setup. Similarly to the bug #1533942 use case, we Totals 113 (22/91) 508 -
deployed OpenStack components as containerized microservices us-
ing Kolla [28]. We deployed 7 Openstack components, along with 12
auxiliary components, as listed in Table 5. The Openstack versions
used for the correct (C) and faulty (F) versions are listed in Table 8. the reported root cause of bug #1590179 is related to Key-
The configurations for the Kolla deployments of each version are stone [30], this might appear as a ‘dishonest’ test. However,
publicly available 3 , as well as the monitoring infrastructure and we argue that Keystone is a central component of Open-
evaluation scripts 4 . stack - as are the compute and networking components -
The infrastructure consists of 3 t2.large Amazon EC2 VM in- and as such a natural candidate for testing.
stances to run OpenStack components (2 vCPUs, 8 GB RAM and 30 • C&DN: Creates and deletes network VM network resources.
GB Amazon EBS storage) and 2 t2.medium VM instances (2 vCPUs, Loads Openstack’s networking component - Neutron - as
4GB RAM and 30 GB EBS storage) for the supporting components well as related components (e.g. Open vSwitch).
(metric collection and storage). • C&DI: Creates and deletes VM images. Loads Openstack’s
Results. Since this bug manifests itself as a general performance image component, Glance.
degradation issue, we run the Rally tasks below to load 4 essential
Metric analysis and component rankings. Table 9 shows that
Openstack components, thus giving us a reasonably large ‘search
solely based on metric novelty scores, the ‘authenticate user and
space’ for RCA (details in Table 6):
validade token’ (AU&VT) Rally task - which directly relates to to-
• B&D: Same Rally task used in bug #1533942. Loads Open- ken validation - ranks Memcached 2nd place as a possible root
stack’s compute (Nova) and networking (Neutron) compo- cause (metric novelty rankings correspond to column ‘N’ in Table 9,
nents. Memcached is ranked 1st after edge filtering, i.e. the ‘F’ column).
• AU&VT: Authenticates a stream of user’s tokens in Key- Other tasks rank Nova conosoleauth (B&D) and Neutron Open-
stone, Openstack’s identity component. In hindsight, since vSwitch agent (C&DN and C&DI) as 1st, which are unrelated to
3 https://github.com/sieve-microservices/kolla bug #1590179 (at least not directly related to it, according to the
4 https://github.com/sieve-microservices/rca-evaluation description given in [30]).
12
 # components, clusters
80
a) Cluster novelty 30
b) Edge novelty c) # of components, clusters and metrics
63 New Components
100
# of clusters

and metrics
60

# of edges
20 Discarded Clusters
75 62
40 Lag change Metrics
10
24 Unchanged 50
17
10 34 34 34
20 2 2 2 25
6
1 0 1 0 0 1 0 0 1 0 0 1 0 9 11 4 4 4 4 4 4
0 0 0
New Discarded New Changed Total 0.0 0.5 0.6 0.7 0.0 0.5 0.6 0.7
and Similarity threshold Similarity threshold
Discarded

Figure 9. AU&VT Rally task results: (a) cluster novelty score. (b) edge novelty score. (c) no. of components & clusters after edge filtering w/
varying thresholds.
Table 8. Openstack versions used in RCA evaluation. Final rankings. In the case of the AU&VT task, Memcached -
which effectively is related to the bug - is ranked 1st after the
Bug # Correctness Openstack ver. Comments edge filtering steps. However, none of the metrics filtered in the
Adapted for Ceilometer edge filtering steps seems to directly relate to bug #1590179. Also,
Mitaka (EOL) support (not available by an intuitive isolation of lag changes in edges involving Keystone
Correct (C)
default)1 clusters did not occur. Further experimentation is required to assess
1533942
Kolla 2.0.0.0b3 - Sieve’s effectiveness for RCA of performance degradation issues
Mitaka (EOL) (1) (as opposed to crashes, such as bug #1533942) such bug #1590179.
Faulty (F)
Kolla 2.0.0.0b2 -
Liberty (EOL) (1) 7 Related Work
Correct (C) Adapted to Fernet tokens
Keystone 8.1.0 Scalable Monitoring. With the increasing number of metrics ex-
1590179 based on blueprint in [26]2
posed by distributed cloud systems, the scalability of the monitoring
Mitaka (EOL) (1)
Faulty (F)
Keystone 9.0.2 (2)
process becomes crucial. Meng et al. [73] optimize monitoring scal-
ability by choosing appropriate monitoring window lengths and
adjusting the monitoring intensity at runtime. Canali et al. [42]
Still regarding metric novelty, Keystone - one of the Openstack achieve scalability by clustering metric data. A fuzzy logic approach
components directly related to bug #1590179 [30] - is lowly ranked is used to speed up clustering, and thus obtain data for decision
according to the AU&VT Rally task. This result is also intuitive: making within shorter periods. Rodrigues et al. [46] explore the
since bug #1590179 relates to performance degradation, metrics trade-off between timeliness and the scalability in cloud monitor-
should not appear or disappear, rather their values should differ ing, and analyze the mutual influence between these two aspects
in-between C and F versions, an effect which may be captured by based on the monitoring parameters. Our work is complementary
relationships in-between the clusters the metrics belong to. to existing monitoring systems since Sieve aims to improve the
Since we load Openstack with the ‘boot and delete’ (B&D) tasks efficiency by monitoring less number of metrics.
in both bugs #1533942 and #1590179, we can compare how the Distributed Debugging. Systems like Dapper [84] and Pip [80]
rankings change between bugs, and verify if there is evidence of a require the developers to instrument the application to obtain its
a dependency between the way in which the system is loaded and causal model. X-trace [50] uses a modified network stack to propa-
the ‘metric novelty’ rankings produced by Sieve. As shown in Ta- gate useful information about the application. In contrast, Sieve
ble 10, the average change in rank is ∼ 5 positions, providing initial does not modify the application code to obtain the call/dependency
evidence about the lack of such dependency. Further validation of graph of the application.
this dependency is left to future work. Systems such as Fay [49] and DTrace [43] enable developers to
Edge filtering. The edge novelty statistics results for the AU&VT dynamically inject debugging requests by developers and require
task - depicted in Figures 9(b) - are of particular interest, since it no initial logs of metrics. Pivot Tracing [72] combines dynamic
directly loads Openstack with a task related to bug #1590179. instrumentation with causality tracing. Sieve can complement these
The edge filtering step identifies 10 new edges no inter-version approaches, because it can provide information about interesting
cluster similarity threshold is applied. These reduce to 2 once the components and metrics, so that the developers can focus their
similarity threshold is raised to 0.5. The 2 new edges are between efforts to understand them better. Furthermore, Sieve’s dependency
the following components: RabbitMQ > Nova conductor and Mem- graph is a general tool that can not only be used for debugging, but
cached > Nova libvirt. The Memcached metrics associated with the also for other purposes such as orchestration [89–91].
edge are not related to cache access metrics, e.g. cache hits or misses. Data provenance [37, 53, 86] is another technique that can be
The single edge isolated due to a causality ‘lag change’ pertains used to trace the dataflow in the system. Sieve can also leverage
to a relationship between RabbitMQ and and Nova libvirt, with the existing provenance tools to derive the dependence graph.
no apparent relation to the bug. None of the filtered relationships Metric reduction. Reducing the size and dimensionality of the
involves Keystone, and as such metrics such as ‘keystone identity bulk of metric data exposed by complex distributed systems is
authenticate success rate’, which are known to be related to the essential for its understanding. Common techniques include sam-
bug. pling, and data clustering via k-means and k-medoids. Kollios et
13
Table 9. Bug #151590179 results: OpenStack components, sorted by the number of novel metrics between correct (C) and faulty (F) versions,
for different Rally tasks. Metric novelty rankings correspond to column ‘N’, final rankings (after edge filtering steps) correspond to column ‘F’.
Components which are ranked 1st according to the ‘metric (N)ovelty’ ranking are underlined, those ranked 1st after edge filtering (‘(F)inal’
rankings) are emphasized in bold.

Rally tasks
Component AU&VT B&D C&DN C&DI
(C)hanged (T)otal Rankings
C T N F C T N F C T N F
(New/Disc.) N F
Neutron OvSwitch 42 (15/27) 53 1 - 0 (0/0) 38 14 - 40 (14/26) 52 1 1 42 (15/27) 53 1 -
Memcached 15 (14/1) 30 2 1 6 (4/2) 33 5 3 7 (3/3) 33 10 4 5 (4/1) 31 11 3
Nova API 10 (10/0) 25 3 - 3 (3/0) 44 10 6 20 (15/5) 32 3 - 18 (13/5) 30 3 1
Nova conductor 9 (0/9) 15 4 2 2 (2/0) 30 12 7 27 (0/27) 33 2 - 0 (0/0) 6 18 -
Nova libvirt 8 (0/8) 17 5 3 2 (2/0) 39 13 8 7 (1/6) 17 8 - 12 (6/6) 23 5 2
Neutron server 6 (6/0) 21 5 - 12 (5/7) 52 2 1 6 (4/2) 34 11 5 18 (13/5) 30 4 -
Neutron L3 agent 4 (0/4) 19 7 - 6 (4/2) 46 4 - 0 (0/0) 15 16 - 2 (2/0) 15 13 -
Nova SSH 3 (3/0) 3 8 - - (-/-) - - - 3 (3/0) 3 14 - - (-/-) - - -
Glance registry 3 (1/2) 45 9 - 0 (0/0) 30 17 - 7 (7/0) 7 9 - 8 (8/0) 8 7 -
RabbitMQ 3 (3/0) 3 10 4 4 (2/2) 63 6 4 8 (8/0) 54 6 2 10 (9/1) 57 6 3
Neutron DHCP ag. 2 (2/0) 15 11 - 0 (0/0) 42 15 9 0 (0/0) 15 18 - 0 (0/0) 15 17 -
Nova compute 2 (0/2) 19 12 - 0 (0/0) 45 16 - 6 (2/4) 23 12 - 2 (0/2) 19 14 -
Nova novncproxy 1 (1/0) 4 13 - 3 (0/3) 17 11 - 8 (7/1) 12 7 3 8 (8/0) 13 8 -
Nova scheduler 0 (0/0) 6 14 - 4 (1/3) 31 8 - 14 (14/0) 14 5 - 0 (0/0) 14 15 -
Nova consoleauth 0 (0/0) 6 15 - 18 (4/14) 36 1 - 0 (0/0) 6 17 - 5 (5/0) 20 10 -
Keystone 0 (0/0) 36 16 - 8 (6/2) 42 3 2 2 (1/1) 37 15 7 0 (0/0) 36 16 -
Neutron metadata ag. 0 (0/0) 16 17 - 0 (0/0) 24 18 - 16 (16/0) 16 4 - 20 (20/0) 20 2 -
Glance API - (-/-) - 18 - 4 (0/4) 34 7 5 5 (5/0) 15 13 6 6 (6/0) 6 9 -
Totals 103 (50/53) 332 - - 76 (34/42) 657 - - 136 (61/75) 378 - - 128 (81/47) 392 - -

Table 10. Change in ‘metric novelty’ rankings for bugs #1533942 among time series. Essentially, Sieve uses two steps of data reduc-
and #1590179, considering the ‘boot and delete’ Rally task. tion for better reduction. Furthermore, Sieve’s time series process-
ing method extracts other useful information such as the time delay
Rank (bug #) of the causal relationship, which can be leveraged in different use
Component Ranking change
1533942 1590179 cases (e.g., root cause analysis).
Nova API 1 10 -9 Orchestration of autoscaling. Current techniques for autoscal-
Nova libvirt 2 11 -11 ing can be broadly classified into four categories [71]: (i) static and
Nova scheduler 3 8 -5
Neutron server 4 2 +2 threshold-based rules (offered by most cloud computing providers [3,
RabbitMQ 5 6 -1 4, 17, 27]); (ii) queuing theory [34, 60, 93]; (iii) reinforcement learn-
Neutron L3 agent 6 4 +2 ing [79, 85, 92]; and (iv) time series analysis [44, 65, 82]. Existing
Nova novncproxy 7 11 -4 systems using these techniques can benefit from the selection of
Glance API 8 7 +1
Neutron DHCP ag. 9 15 -6
better metrics and/or from the dependencies between components.
Nova compute 10 16 -6 In this regard, our work is complementary to these techniques: it
Glance registry 11 17 -6 is intended to provide the developers with knowledge about the
Nova conductor 13 12 +1 application as a whole. In our case study, we showed the benefits
Keystone 15 3 +12 of Sieve for an autoscaling engine using threshold-based rules.
Root Cause Analysis (RCA). Large and complex distributed sys-
tems are susceptible to anomalies, whose root causes are often hard
to diagnose [62]. Jiang et al. [64] compare “healthy" and “faulty"
metric correlation maps, searching broken correlations. In contrast,
al. [67] employ biased sampling to capture the local density of Sieve leverages Granger causality instead of simple correlation,
datasets. Sampling based approaches argues for approximate com- allowing for richer causality inference (e.g., causality direction,
puting [68, 77, 78] to enable a systematic trade-off between the time lag between metrics). MonitorRank [66] uses metric collection
accuracy, and efficiency to collect and compute on the metrics. for RCA in a service-oriented architecture. It only analyzes pre-
Zhou et al. [94] simply use random sampling due to its simplicity established (component, metric) relations according to a previously-
and low complexity. Ng et al. [74] improved the k-medoid method generated call graph. Sieve also uses a call graph, but does not fix
and made it more effective and efficient. Ding et al. [47] rely on metric relations between components, for a richer set of potential
clustering over sampled data to reduce clustering time. root causes. There are other application-specific solutions for RCA
Sieve’s approach is unique because of its two-step approach: (e.g., Hansel [83], Gretel [55]). In contrast, Sieve uses a general
(1) we first cluster time series to identify the internal dependency approach for understanding the complexity of microservices-based
between any given metrics and then (2) infer the causal relations applications that can support RCA as well as other use cases.
14
8 Experience and Lessons Learned 9 Conclusion and Future Work
While developing Sieve, we set ourselves ambitious design goals This paper reports on our experiences with designing and building
(described in §2.2). However, we learned the following lessons while Sieve, a platform to automatically derive actionable insights from
designing and deploying Sieve for real-world applications. monitored metrics in distributed systems. Sieve achieves this goal
by automatically reducing the amount of metrics and inferring inter-
Lesson#1. When we first designed Sieve, we were envisioning component dependencies. Our general approach is independent
a dependency graph that was clearly showing the relationships of the application, and can be deployed in an unsupervised mode
between components (e.g., a tree). As a result, not only would the without prior knowledge of the time series of metrics. We showed
number of metrics that needed to be monitored be reduced, but that Sieve’s resulting model is consistent, and can be applied for
also the number of components: one would only need to observe common use cases such as autoscaling and root-cause debugging.
the root(s) of the dependency graph, and make the actions of the An interesting research challenge for the future would be to
dependent components according to the established relationships integrate Sieve into the continuous integration pipeline of an appli-
between the root(s) and them. Such a dependency graph would give cation development. In this scenario, the dependency graph can be
the orchestration scenario a huge benefit. Unfortunately, our expe- updated incrementally [39–41], which would speed up the analytics
rience has shown us that the relationships between components part. In this way, the developers would be able to get real-time pro-
are usually not linear, making the dependency graph more complex. file updates of their infrastructure. Another challenge is to utilize
Also, there was no obvious root. Consequently, we had to adjust already existing traffic to generate the dependency graph without
our thinking and utilize some application knowledge regarding requiring the developers to load the system. Using existing traffic
components and their relations with others. Nevertheless, in our would alleviate the burden of developers to supply a workload gen-
experience, Sieve provides the developer with a good starting point erator. On the other hand, existing traffic traces might not always
to improve their workflows. capture the stress points of the application. A hybrid approach,
in which workload generation is only used for these corner cases,
Lesson#2. Sieve is designed for “blackbox” monitoring of the eval- might help to overcome this problem.
uated application, where Sieve can collect and analyze generic Software availability. The source code of Sieve is publicly avail-
system metrics that are exposed by the infrastructure (e.g., CPU able: https://sieve-microservices.github.io/.
usage, disk I/O, network bandwidth). However, in our experience, Acknowledgments. We would like to thank Amazon AWS for
a system for monitoring and analyzing an application should also providing the required infrastructure to run the experiments.
consider application-specific metrics (e.g., request latency, number
of error messages) to build effective management tools. Fortunately, References
many microservices applications we analyzed already export such [1] Amazon AWS - Scaling Based on Metrics. https://docs.aws.amazon.com/
metrics. However, given the number of components and exported autoscaling/latest/userguide/policy_creating.html. Last accessed: September,
metrics, this fact can easily create an “information overload” for the 2017.
[2] Amazon CloudWatch. https://aws.amazon.com/de/cloudwatch/. Last accessed:
application developers. In fact, the main motivation of Sieve was September, 2017.
to deal with this “information overload”. Our experience showed [3] Amazon Web Services. https://aws.amazon.com/documentation/autoscaling/.
Last accessed: September, 2017.
that Sieve can still monitor the application in the blackbox mode [4] Google Cloud Platform. https://cloud.google.com/developers/articles/
(i.e., no instrumentation to the application), but also overcome the auto-scaling-on-the-google-cloud-platform. Last accessed: September, 2017.
barrage of application-specific metrics. [5] Google Stackdriver. https://cloud.google.com/stackdriver/. Last accessed: Sep-
tember, 2017.
[6] Http Trace of WorldCup98. http://ita.ee.lbl.gov/html/contrib/WorldCup.html.
Lesson#3. To adapt to the application workload variations, Sieve Last accessed: September, 2017.
[7] InfluxDB. https://influxdata.com/time-series-platform/influxdb/. Last accessed:
needs to build a robust model for the evaluated application. This re- September, 2017.
quires a workload generator that can stress-test the application thor- [8] Introducing Vector. http://techblog.netflix.com/2015/04/
oughly. To meet this requirement, there are three approaches: (1) In introducing-vector-netflixs-on-host.html. Last accessed: September, 2017.
[9] Kapacitor. https://influxdata.com/time-series-platform/kapacitor/. Last ac-
many cases the developers already supply an application-specific cessed: September, 2017.
workload generator. For instance, we employed the workload gen- [10] Keystone tokens. https://docs.openstack.org/keystone/latest/admin/
identity-tokens.html. Last accessed: September, 2017.
erator shipped with the OpenStack distribution. (2) For cases where [11] Locust - An Open Source Load Testing Tool. http://locust.io/. Last accessed:
we did not have an existing workload generator, we implemented September, 2017.
a custom workload generator for the evaluated application. For [12] Manualpage of Apache Benchmark. https://httpd.apache.org/docs/2.4/programs/
ab.html. Last accessed: September, 2017.
example, we built a workload generator for ShareLatex. Although [13] Microsoft Azure Monitor. https://docs.microsoft.com/en-us/azure/
we were able to faithfully simulate user actions in ShareLatex, such monitoring-and-diagnostics/monitoring-overview. Last accessed: September,
an approach might not be feasible for some applications. Having 2017.
[14] Monitoring at Quantcast. https://www.quantcast.com/wp-content/uploads/
the ability to utilize existing production traces (e.g., by replaying 2013/10/Wait-How-Many-Metrics_-Quantcast-2013.pdf. Last accessed: Sep-
the trace or by reproducing similar traces) or working in an online tember, 2017.
[15] Nginx. https://nginx.org/. Last accessed: September, 2017.
fashion to generate the model of the application would certainly [16] Observability at Uber Engineering: Past, Present, Future. https://www.youtube.
help Sieve. Custom workload generation can then be used to close com/watch?v=2JAnmzVwgP8. Last accessed: September, 2017.
the gaps in the model for certain workload conditions not covered [17] OpenStack. https://wiki.openstack.org/wiki/Heat. Last accessed: September,
2017.
by the existing traces. (3) We could also explore some principled [18] Openstack: API References (Response parameters). https://developer.openstack.
approaches for automatic workload generation, such as symbolic org/api-ref/. Last accessed: September, 2017.
execution in distributed systems [36]. [19] Openstack: Open source Software for Creating Private and Public Clouds. https:
//www.openstack.org/. Last accessed: September, 2017.
15
[20] Openstack: Telemetry. https://docs.openstack.org/admin-guide/ [52] K. P. F.R.S. LIII. On Lines and Planes of Closest Fit to Systems of Points in
telemetry-measurements.html. Last accessed: September, 2017. Space. In Proceedings of the 17th ACM SIGACT-SIGMOD-SIGART symposium on
[21] Rally. https://wiki.openstack.org/wiki/Rally. Last accessed: September, 2017. Principles of database systems (SIGMOD), 1901.
[22] Rancher Container Management. http://rancher.com/. Last accessed: September, [53] A. Gehani and D. Tariq. Spade: Support for provenance auditing in distributed
2017. environments. In Proceedings of the 13th International Middleware Conference
[23] Sharelatex - A Web-based Collaborative LaTeX Editor. https://sharelatex.com. (Middleware), 2012.
Last accessed: September, 2017. [54] D. Giles. Testing for Granger Causality. https://davegiles.blogspot.de/2011/04/
[24] Sysdig. http://www.sysdig.org/. Last accessed: September, 2017. testing-for-granger-causality.html. Last accessed: September, 2017.
[25] Telegraf: Time-series Data Collection. https://www.influxdata.com/ [55] A. Goel, S. Kalra, and M. Dhawan. GRETEL: Lightweight Fault Localization for
time-series-platform/telegraf/. Last accessed: September, 2017. OpenStack. In Proceedings of the 12th International on Conference on Emerging
[26] Use Fernet tokens with Keystone. https://blueprints.launchpad.net/kolla/+spec/ Networking EXperiments and Technologies (CoNEXT), 2016.
keystone-fernet-token. Last accessed: September, 2017. [56] C. Granger and P. Newbold. Spurious Regressions in Econometrics. Journal of
[27] Windows Azure. http://msdn.microsoft.com/en-us/library/hh680945(v=pandp. Econometrics, 2(2):111–120, 1974.
50).aspx. Last accessed: September, 2017. [57] C. W. J. Granger. Investigating Causal Relations by Econometric Models and
[28] Openstack Kolla. http://docs.openstack.org/developer/kolla/, 2016. Last accessed: Cross-spectral Methods. Econometrica, 1969.
September, 2017. [58] W. H. Greene. Econometric Analysis. Prentice Hall, 5. edition, 2003.
[29] Openstack Kolla Launchpad: neutron-openvswitch-agent Bug. https://bugs. [59] E. Haddad. Service-Oriented Architecture: Scaling the uber Engineering Code-
launchpad.net/kolla/+bug/1533942, 2016. Last accessed: September, 2017. base As We Grow. https://eng.uber.com/soa/, 2015. Last accessed: September,
[30] Openstack Kolla Launchpad: neutron-openvswitch-agent Bug. https://bugs. 2017.
launchpad.net/keystone/+bug/1590179, 2016. Last accessed: September, 2017. [60] R. Han, M. M. Ghanem, L. Guo, Y. Guo, and M. Osmond. Enabling Cost-aware
[31] ptrace(2) Linux User’s Manual, 4.07 edition, Aug 2016. and Adaptive Elasticity of Multi-tier Cloud Applications. 2014.
[32] Scikit Documentation: sklearn.metrics.silhouette_score. http://scikit-learn.org/ [61] B. Harrington and R. Rapoport. Introducing Atlas: Netflix’s Pri-
stable/modules/generated/sklearn.metrics.silhouette_score.html, 2016. Last ac- mary Telemetry Platform. http://techblog.netflix.com/2014/12/
cessed: September, 2017. introducing-atlas-netflixs-primary.html, 2014. Last accessed: September, 2017.
[33] M. K. Aguilera, J. C. Mogul, J. L. Wiener, P. Reynolds, and A. Muthitacharoen. [62] V. Heorhiadi, S. Rajagopalan, H. Jamjoom, M. K. Reiter, and V. Sekar. Gremlin:
Performance Debugging for Distributed Systems of Black Boxes. In Proceedings Systematic Resilience Testing of Microservices. In Proceedings of the 2016 IEEE
of the 19th ACM Symposium on Operating Systems Principles (SOSP), 2003. 36th International Conference on Distributed Computing Systems (ICDCS), 2016.
[34] A. Ali-Eldin, M. Kihl, J. Tordsson, and E. Elmroth. Efficient Provisioning of [63] M. A. Jaro. Advances in record-linkage methodology as applied to matching the
Bursty Scientific Workloads on the Cloud Using Adaptive Elasticity Control. In 1985 census of tampa, florida. 84(406):414–420, 1989.
Proceedings of the 3rd Workshop on Scientific Cloud Computing Date (ScienceCloud), [64] M. Jiang, M. A. Munawar, T. Reidemeister, and P. A. S. Ward. Dependency-
2012. aware Fault Diagnosis with Metric-correlation Models in Enterprise Software
[35] Amemiya, Takeshi. Advanced econometrics. 1985. Systems. In Proceedings of 2010 International Conference on Network and Service
[36] R. Banabic, G. Candea, and R. Guerraoui. Finding trojan message vulnerabilities Management (NSDI), 2010.
in distributed systems. In Proceedings of the 19th International Conference on Ar- [65] S. Khatua, A. Ghosh, and N. Mukherjee. Optimizing the Utilization of Virtual
chitectural Support for Programming Languages and Operating Systems (ASPLOS), Resources in Cloud Environment. In Proceedings of the 2010 IEEE International
2014. Conference on Virtual Environments, Human-Computer Interfaces and Measure-
[37] A. Bates, D. J. Tian, K. R. Butler, and T. Moyer. Trustworthy whole-system ment Systems (CIVEMSA), 2010.
provenance for the linux kernel. In 24th USENIX Security Symposium (USENIX [66] M. Kim, R. Sumbaly, and S. Shah. Root Cause Detection in a Service-oriented
Security, 2015. Architecture. In Proceedings of the ACM SIGMETRICS/international conference on
[38] R. Bellman and R. Corporation. Dynamic Programming. Rand Corporation Measurement and modeling of computer systems (SIGMETRICS), 2013.
research study. Princeton University Press, 1957. [67] G. Kollios, D. Gunopulos, N. Koudas, and S. Berchtold. Efficient Biased Sam-
[39] P. Bhatotia. Incremental Parallel and Distributed Systems. PhD thesis, Max Planck pling for Approximate Clustering and Outlier Detection in Large Data Sets. In
Institute for Software Systems (MPI-SWS), 2015. Proceedings of the 2003 IEEE Transactions on Knowledge and Data Engineering
[40] P. Bhatotia, P. Fonseca, U. A. Acar, B. Brandenburg, and R. Rodrigues. iThreads: (TKDE), 2003.
A Threading Library for Parallel Incremental Computation. In Proceedings of the [68] D. R. Krishnan, D. L. Quoc, P. Bhatotia, C. Fetzer, and R. Rodrigues. IncApprox: A
20th International Conference on Architectural Support for Programming Languages Data Analytics System for Incremental Approximate Computing. In proceedings
and Operating Systems (ASPLOS), 2015. of International Conference on World Wide Web (WWW), 2016.
[41] P. Bhatotia, A. Wieder, R. Rodrigues, U. A. Acar, and R. Pasquini. Incoop: MapRe- [69] J. M. Liu. Nonlinear Time Series Modeling Using Spline-based Nonparametric
duce for Incremental Computations. In Proceedings of the ACM Symposium on Models. In Proceedings of the 15th American Conference on Applied Mathematics
Cloud Computing (SoCC), 2011. (AMATH), 2009.
[42] C. Canali and R. Lancellotti. An Adaptive Technique To Model Virtual Ma- [70] R. Lomax and D. Hahs-Vaughn. Statistical Concepts: A Second Course, Fourth
chine Behavior for Scalable Cloud Monitoring. In Proceedings of the 19th IEEE Edition. Taylor & Francis, 2012.
Symposium on Computers and Communications (ISCC), 2014. [71] T. Lorido-Botran, J. Miguel-Alonso, and J. A. Lozano. A Review of Auto-scaling
[43] B. Cantrill, M. W. Shapiro, A. H. Leventhal, et al. Dynamic instrumentation Techniques for Elastic Applications in Cloud Environments. In Proceedings of
of production systems. In Proceedings of the 2004 USENIX Annual Technical Grid Computing (CCGrid), 2014.
Conference (ATC), 2004. [72] J. Mace, R. Roelke, and R. Fonseca. Pivot tracing: Dynamic causal monitoring for
[44] G. Chen, W. He, J. Liu, S. Nath, L. Rigas, L. Xiao, and F. Zhao. Energy-aware distributed systems. In Proceedings of the 25th Symposium on Operating Systems
Server Provisioning and Load Dispatching for Connection-intensive Internet Principles (SOSP), 2015.
Services. In Proceedings of the 5th USENIX Symposium on Networked Systems [73] S. Meng and L. Liu. Enhanced Monitoring-as-a-service for Effective Cloud
Design and Implementation (NSDI), 2008. Management. In Proceedings of IEEE Transactions on Computers (TC), 2013.
[45] M. Chow, D. Meisner, J. Flinn, D. Peek, and T. F. Wenisch. The Mystery Machine: [74] R. T. Ng and J. Han. Efficient and Effective Clustering Methods for Spatial Data
End-to-end Performance Analysis of Large-scale Internet Services. In Proceedings Mining. In Proceedings of the 19st International Conference on VERY LARGE DATA
of the 11th USENIX Symposium on Operating Systems Design and Implementation BASES (VLDB), 1994.
(OSDI), 2014. [75] C. H. Papadimitriou, P. Raghavan, H. Tamaki, and S. Vempala. Latent semantic
[46] G. da Cunha Rodrigues, R. N. Calheiros, M. B. de Carvalho, C. R. P. dos Santos, indexing: A probabilistic analysis. Journal of Computer and System Sciences, 2000.
L. Z. Granville, L. Tarouco, and R. Buyya. The Interplay Between Timeliness [76] J. Paparrizos and L. Gravano. k-Shape: Efficient and Accurate Clustering of Time
and Scalability In Cloud Monitoring Systems. In Proceedings of the 20nd IEEE Series. In Proceedings Of the 2016 ACM SIGMOD/PODS Conference (SIGMOD),
Symposium on Computers and Communications (ISCC), 2015. 2016.
[47] R. Ding, Q. Wang, Y. Dang, Q. Fu, H. Zhang, and D. Zhang. Yading: Fast Clustering [77] D. L. Quoc, M. Beck, P. Bhatotia, R. Chen, C. Fetzer, and T. Strufe. PrivApprox:
of Large-scale Time Series Data. In Proceedings of the 41st International Conference Privacy-Preserving Stream Analytics. In Proceedings of the 2017 USENIX Annual
on VERY LARGE DATA BASES (VLDB), 2015. Technical Conference (USENIX ATC), 2017.
[48] C. W. S. Emmons, and B. Gregg. A Microscope on Microservices. http://techblog. [78] D. L. Quoc, R. Chen, P. Bhatotia, C. Fetzer, V. Hilt, and T. Strufe. StreamApprox:
netflix.com/2015/02/a-microscope-on-microservices.html, 2015. Last accessed: Approximate Computing for Stream Analytics. In Proceedings of the International
September, 2017. Middleware Conference (Middleware), 2017.
[49] U. Erlingsson, M. Peinado, S. Peter, and M. Budiu. Fay: Extensible distributed [79] J. Rao, X. Bu, C.-Z. Xu, L. Wang, and G. Yin. VCONF: A Reinforcement Learning
tracing from kernels to clusters. In Proceedings of the 23th ACM Symposium on Approach to Virtual Machines Auto-configuration. In Proceedings of the 6th
Operating Systems Principles (SOSP), 2011. International Conference on Autonomic Computing (ICAC), 2009.
[50] R. Fonseca, G. Porter, R. H. Katz, S. Shenker, and I. Stoica. X-trace: A pervasive [80] P. Reynolds, C. E. Killian, J. L. Wiener, J. C. Mogul, M. A. Shah, and A. Vahdat. Pip:
network tracing framework. In Proceedings of the conference on Networked systems Detecting the unexpected in distributed systems. In Proceedings of the conference
design & implementation (NSDI), 2007. on Networked systems design & implementation (NSDI), 2006.
[51] M. Fowler. Microservices. http://martinfowler.com/articles/microservices.html.
Last accessed: September, 2017.
16
[81] P. J. Rousseeuw. Silhouettes: A graphical aid to the interpretation and validation [88] N. X. Vinh, J. Epps, and J. Bailey. Information Theoretic Measures for Clusterings
of cluster analysis. Journal of Computational and Applied Mathematics, 20:53 – Comparison: Is a Correction for Chance Necessary? In Proceedings of the 26th
65, 1987. Annual International Conference on Machine Learning (ICML), 2009.
[82] N. Roy, A. Dubey, and A. Gokhale. Efficient Autoscaling in the Cloud Using [89] A. Wieder, P. Bhatotia, A. Post, and R. Rodrigues. Brief Announcement: Modelling
Predictive Models for Workload Forecasting. In Proceedings of the 2011 IEEE 4th MapReduce for Optimal Execution in the Cloud. In Proceedings of the 29th ACM
International Conference on Cloud Computing (CCIS), 2011. SIGACT-SIGOPS symposium on Principles of Distributed Computing (PODC), 2010.
[83] D. Sharma, R. Poddar, K. Mahajan, M. Dhawan, and V. Mann. Hansel: Diagnosing [90] A. Wieder, P. Bhatotia, A. Post, and R. Rodrigues. Conductor: Orchestrating the
Faults in OpenStack. In Proceedings of the 11th ACM Conference on Emerging Clouds. In Proceedings of the 4th international workshop on Large Scale Distributed
Networking Experiments and Technologies (CoNEXT), 2015. Systems and Middleware (LADIS), 2010.
[84] B. H. Sigelman, L. A. Barroso, M. Burrows, P. Stephenson, M. Plakal, D. Beaver, [91] A. Wieder, P. Bhatotia, A. Post, and R. Rodrigues. Orchestrating the Deployment
S. Jaspan, and C. Shanbhag. Dapper, a Large-Scale Distributed Systems Tracing of Computations in the Cloud with Conductor. In Proceedings of the 9th USENIX
Infrastructure. Technical report, Google, 2010. symposium on Networked Systems Design and Implementation (NSDI), 2012.
[85] G. Tesauro, N. K. Jong, R. Das, and M. N. Bennani. A Hybrid Reinforcement [92] L. Yazdanov and C. Fetzer. Lightweight Automatic Resource Scaling for Multi-tier
Learning Approach to Autonomic Resource Allocation. In Proceedings of the Web Applications. In Proceedings of the 2014 IEEE 7th International Conference
2006 IEEE International Conference on Autonomic Computing (ICAC), 2006. on Cloud Computing (CLOUD), 2014.
[86] J. Thalheim, P. Bhatotia, and C. Fetzer. Inspector: Data Provenance using In- [93] Q. Zhang, L. Cherkasova, and E. Smirni. A Regression-Based Analytic Model for
tel Processor Trace (PT). In proceedings of IEEE International Conference on Dynamic Resource Provisioning of Multi-Tier Applications. In Proceedings of
Distributed Computing Systems (ICDCS), 2016. the Fourth International Conference on Autonomic Computing (ICAC), 2007.
[87] J. Thalheim, A. Rodrigues, I. E. Akkus, P. Bhatotia, R. Chen, B. Viswanath, L. Jiao, [94] S. Zhou, A. Zhou, J. Cao, J. Wen, Y. Fan, and Y. Hu. Combining Sampling
and C. Fetzer. Sieve: Actionable Insights from Monitored Metrics in Distributed Technique With DBSCAN Algorithm for Clustering Large Spatial Databases. In
Systems. In Proceedings of Middleware Conference (Middleware), 2017. Proceedings of the 4th Pacific-Asia Conference on Knowledge Discovery and Data
Mining (PAKDD), 2000.

17