Understanding Web Application Workloads and Their Applications: Systematic Literature Review and Characterization

We conduct our literature search on two highly reputable research databases: IEEE library ²²2https://ieeexplore.ieee.org/ and ACM library ³³3https://dl.acm.org/. We aim to identify papers from both conference proceedings and journals, written in English and published between January 2014 and December 2023, a time span of 10 years. To pinpoint the articles that either introduce or utilize web application workloads, we use the following query to search the databases:

[[Title: web application] OR [Title: workload] OR [Title: trace]] AND [Abstract: dataset] AND [E-Publication Date: (2014/01/01 TO 2023/12/31)]

Upon conducting this query on the IEEE library, we identify a total of 545 articles that satisfy our criteria. Simultaneously, our search from the ACM library yields an additional 217 articles after deduplication. Combining papers from both sources, we successfully compile a total of 762 candidate articles.

We screen each paper, beginning with its abstract, and then proceed through the entire article until a decision is made based on the inclusion and exclusion criteria.

Inclusion criteria:

• •

  Papers must be written in English.

• •

  Papers must be in conference proceedings or journals.

• •

  Papers must introduce or utilize web application workloads that are publicly available and representative of real-world applications.

• •

  The utilized workloads must include timestamp information, as this data will enable us to conduct subsequent analysis and characterization of the workloads.

Exclusion criteria:

• •

  Papers not publicly available.

• •

  Papers introducing or utilizing private data (e.g., with Non-Disclosure Agreement (NDA) contracts).

• •

  Papers introducing or utilizing non-web application data such as cloud-based batch-jobs (e.g., Google cluster trace), social networks data, mobile devices workloads, and synthetic workloads. Overall, we exclude articles that leverage data unrelated to user interaction workloads.

After performing the article selection step, we end up selecting 41 out of the 762 candidate articles.

To ensure the comprehensiveness of our study and to capture potentially relevant articles, we initiate the forward and backward snowballing phase on the selected 41 papers. During this phase, for each of the selected articles, we inspect the list of papers that have cited them (i.e., forward snowballing) and their reference papers (i.e., backward snowballing) and perform the article selection step (as described in Section III-B2) to decide whether the new article matches with our inclusion and exclusion criteria. After performing this step, we end up having 33 new articles. Combining the articles retrieved through our initial search of research databases and the subsequent snowballing process, we obtain a total of 78 articles, each of which introduces or utilizes public web application workloads.

We manually examine each of the selected articles to extract its dimensions (i.e., workloads, techniques, objectives). This labeling process involves reviewing all sections of each paper to identify the relevant information, though most of this information is typically found in the methodology sections. We use an open coding approach [58] to extract the desired information. To label the articles, the first two authors of the paper (i.e., coders) jointly perform a coding process, determining each article’s workloads, techniques, and objectives. We perform a five-step coding process as follows.

Step 1: Coding. Each coder independently analyzes the first batch of articles (i.e., the initial 37) and assigns labels for each dimension (i.e., workloads, techniques, objectives) of each paper. Multiple labels can be assigned to each dimension.

Step 2: Discussion. The coders share their responses and discuss the labels they created, aiming for a common understanding. We join related labels and refine high-level ones, then revise the coding of the first batch accordingly.

Step 3: Coding. Each coder analyzes the second batch of articles (i.e., the final 37) based on the discussion results.

Step 4: Resolving disagreements. The coders compare their final results from step 3 and discuss any remaining conflicts, attempting to resolve them. If an agreement cannot be reached, the third author makes the final decision.

Step 5: Final revision. In the final stage, we create a mind map from all the produced labels. We then discuss the labels and form a hierarchy, change some labels’ names for clarity, and merge some small categories to be cohesive.

Ensuring reliability is crucial for validating coding results [59]. The coding results are reliable when there is a specific level of agreement between coders, referred to as inter-coder agreement. In this study, we employ Cohen’s kappa [60] as a metric to quantify the reliability of agreements between two coders. We evaluate our coding reliability for the second batch of the articles (i.e., after the discussion session) and achieve a Cohen’s kappa of 0.94. A value of kappa $\geq$ 0.80 indicates a strong agreement [61].

Table I presents workload datasets used in the literature, showcasing details such as duration, data instances, description, frequency of usage in articles, and example references. Our review identifies 12 distinct workloads commonly used in the literature: Wikipedia [1], Worldcup98 [29], NASA [30], Saskatchewan [62], Calgary [63], EPA [64], Clarknet [65], Retailrocket [66], Boston [67], SDSC [68], Youtube [31], and Madrid [69].

Some workloads, such as the NASA dataset, appeared in many articles (35 in total), while others, such as RetailRocket, SDSC, and Madrid, were used much less frequently, each appearing in just one article. These datasets vary significantly in duration and the number of instances. For instance, the WorldCup98 workload, characterized by high demand, consists of 1.3 billion instances. In contrast, the SDSC workload spans only one day with 28 thousand instances.

Table II presents objectives and primary techniques from literature articles, along with their frequency of usage and example references. A more detailed list of objectives and techniques along with all the reference articles can be found in our replication package⁴⁴4https://github.com/mooselab/web-app-workloads. Below we define the objective categories.

The first objective theme is Resource Management, which aims to ensure the efficient allocation, utilization, and optimization of computing resources within a system or network. It includes three objectives: Load Balancing, Resource Provisioning, and Caching Optimization.

Load balancing is the distribution of incoming network traffic or computational tasks across multiple resources to optimize resource utilization and ensure high availability. Various approaches are employed for this purpose. For example, Riska et al. [70] apply queueing theory to evaluate load balancing policies in distributed multi-server systems, modeling each server as a queue using Markov chains.

Resource provisioning is the process of initially allocating the resources and services from a cloud provider to a customer to meet the requirements of applications. Shahidinejad et al. [28] employ classical machine learning models to design their Resource Provisioning system. They propose a system to first cluster the workloads using K-Means and then use Decision Tree Regression (i.e., DTR) to determine scaling decisions for efficient resource provisioning. In another work, Zhou et al. [5] combines deep learning models (i.e., Long Short-Term Memory (i.e., LSTM)) with classical machine learning models (i.e., XGBoost) to provide proactive resource provisioning. They center their work on microservice systems.

Caching optimization is the techniques and strategies aimed at improving the efficiency and effectiveness of caching mechanisms. Time-series and statistical analysis are the main techniques used for this category. For example, Bairavasundaram et al. [12] create an image of the file system cache using inferred disk traffic data and propose an array caching mechanism.

The second objective theme is Workload Analysis. It is the examination and understanding of patterns, trends, and characteristics of system or network usage. This objective also includes three objectives: Workload Prediction, Workload Classification, and Workload Characterization.

Workload prediction involves using historical data and trends to forecast future workload demands and patterns, aiming to predict resource requirements and maintain optimal system performance. This objective represents the most common theme across our literature articles, comprising 45% of the total objectives outlined in these papers. Due to the number of articles aiming to improve workload prediction approaches, various techniques have been used. Kim et al. [53] build an ensemble architecture encompassing various time-series models such as Weighted Moving Average (i.e., WMA) and Auto-Regressive Integrated Moving Average (i.e., ARIMA). Their ensemble architecture chooses a subset of these models based on the characteristics of the upcoming workload. Deep learning is another popular solution. Shi et al. [14] use Deep Reinforcement Learning (i.e., DRL) and Attia et al. [71] use deep learning with a Differential Evolution (i.e., DE) algorithm as their optimization technique to predict the workloads. In another work, Saravanan et al. [39] use markov models in order to filter redundant historical data before passing them to a Generative Adversarial Network (i.e., GAN) model.

Workload classification involves using classification techniques to categorize workloads. In [72], time-series models such as Sample Entropy and classical machine Learning models such as K-Nearest Neighbors (i.e., KNN) are employed to classify workloads. In [2], Urdaneta et al. use time-series analysis techniques to classify Wikipedia workload, emphasizing challenges from its large, heterogeneous dataset.

Workload characterization is the understanding and describing the behavior and characteristics of workloads, including patterns, trends, and variations. The majority of articles within this category rely on time-series and statistical analysis due to the ability of time-series analysis to capture temporal dependencies and identify recurring patterns in workload data.

Self Adaptation is the third objective theme. It is the automatic adjustment of system parameters and resources in response to changing conditions or system faults, aiming to improve performance and reliability without human intervention. It includes two categories of Auto-scaling and Self-healing.

Auto-scaling is the dynamic adjustment of the number of resources allocated to an application based on its workload, ensuring optimal performance and cost efficiency. While resource provisioning focuses on the initial allocation and management of resources, auto-scaling continuously monitors system metrics and automatically adjusts resource levels to match current demand. Control Theory techniques such as Proportional Integrative Derivative (i.e., PID) [73] have been applied to automatically scale out public-cloud resources, utilizing only the customer’s existing knowledge base. In another attempt to create an auto-scaling system, Kumar et al. [74] use queueing theory and design a system that dynamically performs resource corrections at the virtual machine level by considering both underutilization and over-utilization scenarios.

Self-healing systems are designed to autonomously detect, diagnose, and recover from failures or performance degradation without human intervention, ensuring continuous operation and reliability. In both works of [10] and [11], the authors employ a Markov model along an optimization technique (i.e., Stochastic Programming) to design a self-healing system compatible with various adaptation objectives.

As the last objective, Benchmarking evaluates the performance, reliability, and scalability of systems by measuring and comparing their performance. In[75], Papadopoulos et al. propose a performance evaluation framework that utilizes Control Theory and optimization to evaluate different auto-scaling systems. In another work, Kumar et al.[57] evaluate optimization algorithms such as Genetic Algorithm and DE in the task of workload prediction. In a similar attempt, Kumar et al. [36] propose a performance evaluation framework that assesses the performance of six prediction models such as ARIMA and Exponential Smoothing on Workload Prediction.

Initially, we compare the publication years of the studied articles with those of the workload datasets they employ. Our findings indicate that a significant proportion of the articles utilize relatively dated workload datasets. Specifically, while the median publication year of the articles is 2018, the median publication year of the datasets is 1995. To provide further insight, over 82% of articles published in 2015 and later rely on datasets that were published in 2000 or earlier. This analysis emphasizes the frequent utilization of older workloads in research studies. Figure 2 presents the scatter plot of this phenomenon.

Subsequently, we analyze the popularity of different objectives over time. Figure 3 presents the cumulative number of articles associated with each objective throughout the years. The analysis highlights workload prediction, auto-scaling, and workload characterization as the most prominent objectives. Particularly, there is a noticeable increase in articles focusing on workload prediction, especially after 2014. However, objectives such as self-healing and caching optimization have received relatively less attention, indicating potential areas for further exploration. This observation aligns with the conclusions drawn by Aghili et al. [76].

As mentioned in Section III-C1, the selected 78 articles utilize 12 web application workloads. A comprehensive description of these workloads, along with key features for each, is presented in Table I.

Most of these 12 web application workloads are accessible for download. However, some, such as Wikipedia, require scripting for data extraction. For the Wikipedia workload, we develop a script to retrieve data spanning 5.5 years, from January 2018 to August 2023.

After gathering all the workloads, we realize that each of them possesses a unique data format. Some are summary-based and only provide information on the number of users who accessed a particular server (e.g., the Wikipedia workload) per time interval, while the others are in the format of log lines and are event-based (e.g., the Worldcup98 workloads). To illustrate this contrast, we provide a comparative example of the raw data from Wikipedia and Worldcup98 workloads in Table III. We write separate scripts for each of these data types, extracting the number of users interacting with the web application within one-hour time intervals.

The preprocessing phase involves three stages: aggregation, standardization, smoothing.

Variability and burstiness analysis is performed before clustering to understand the inherent dynamics of web application workloads. This analysis helps identify distribution differences that might influence clustering results. By examining raw datasets for variability and burstiness after aggregation but before standardization, we retain the original data characteristics.

Variability is defined as the extent to which data points in a time series differ from each other and from the mean value [81]. We calculate variability by measuring the coefficient of variation (CV) of the workload intensities within one day (i.e., 24 hourly workload intensity values) or one week (i.e., seven daily intensity values) using the below formula:

where $\sigma$ is the standard deviation and $\mu$ is the mean.

Burstiness refers to the tendency of a time series to exhibit sudden, irregular increases in activity or intensity over short periods. To quantify burstiness, we calculate the mean and standard deviation of workloads per day and per week. Burstiness is then determined using the formula [82]:

Burstiness ranges from -1 to 1, where 1 indicates high irregularity, 0 indicates no significant burstiness, and -1 suggests a regular pattern. This quantification highlights periods with significant deviations from overall trends.

To investigate general workload patterns across a comprehensive dataset, we employ workload clustering. We first develop a script to combine the standardized and smoothed daily and weekly workloads from all 12 workload datasets. This combination process results in two unified datasets: the daily workload dataset and the weekly workload dataset. This combination enables us to construct a unified dataset that can effectively capture robust and generalized patterns. The daily dataset contains 3191 data points, each representing one day, and the weekly dataset contains 466 data points, each representing one week.

After constructing the combined daily and weekly datasets, we employ K-Means clustering. K-Means clustering is a widely recognized and employed clustering method, designed to divide n observations into k clusters, in which analyzed data sets are partitioned in relation to the selected parameters and grouped around cluster centroids [83].

Determining the optimal number of clusters (a.k.a., k value) is critical in applying the K-Means algorithm. Various methods exist for this purpose, including the elbow method [84] and the silhouette score [85]. In this study, we opt for the silhouette score, which measures the cohesion and separation of clusters, with values ranging from -1 to 1. A higher value indicates well-separated clusters and a near-zero or negative score suggests overlapping or incorrectly assigned data points.

We experiment with Euclidean [86], Dynamic Time Warping (DTW) [87], and Soft-DTW [88] distance metrics to determine the most suitable metric for our data. Our evaluation indicates that Euclidean distance yieldes the most meaningful cluster separations based on the silhouette score. We execute the K-Means algorithm for k values ranging from 1 to 20 and determine the optimal k based on the silhouette score. Additionally, to validate our findings, we visually inspect the clustering results using t-SNE illustration [89].

Beyond identifying workload patterns, understanding their underlying characteristics and interrelationships is crucial. We analyze the clustering results from various perspectives: analyzing centroids, studying associations between daily and weekly patterns, and exploring the time dependence of workload patterns.

Figure 6 illustrates the variability analysis for daily and weekly granularities across each workload. The number of workloads depicted in this figure (from 1 to 12) corresponds to the “ID” column listed in Table I. A CV close to 0 indicates low variability, suggesting stable workloads, while a CV greater than 1 suggests high relative variability. Figure 6 reveals two key observations: 1. Daily variability is significantly higher than weekly variability, and 2. Although most workloads exhibit relatively stable workloads, high variability is observed in two daily workloads (i.e., Boston and EPA) and one weekly workload (i.e., YouTube).

Figure 7 presents the burstiness analysis for daily and weekly granularities. Similar to variability findings, daily workloads exhibit higher burstiness. Specifically, while none of the weekly workloads exhibit burstiness values exceeding 0.5, two of the daily workloads, Boston and EPA, reach this threshold. The Wikipedia workload shows the lowest burstiness, nearing -1, indicating regular patterns. This regularity may result from its large global user base, which balances bursts across different regions.

The clustering results are presented in Figure 8, showcasing the identified patterns. As mentioned in Section IV-B5, we obtain 3191 datapoints at the daily granularity of web application workloads. After clustering, we identify three distinct patterns. Specifically, Figure 8(a) illustrates the first cluster (i.e., D1) with 2262 instances, Figure 8(c) shows the second cluster (i.e., D2) with 406 instances, and Figure 8(e) depicts the third cluster (i.e., D3) with 523 instances. The x-axis of these figures represents a 24-hour day.

As discussed in Section IV-B5, 466 datapoints exist at the weekly granularity of web application workloads. After clustering, we uncover three patterns presented in Figures 8(b),  8(d), and 8(f), and are donated as W1, W2, and W3, with 283, 64, and 119 instances, respectively. The x-axis of these figures corresponds to a 7-day week, commencing from 1 (Monday) and concluding at 7 (Sunday).

Looking at daily and weekly patterns, it is evident that they exhibit unique patterns while having commonalities. Most of the clusters (i.e., D1, D2, D3, W1, W3) are non-monotonic and have curvy shapes. On the other hand, W2 has a sublinear shape with a slightly ascending pattern. All the patterns show a distinct peak and low period, yet the rate of ascending/descending to/from these peaks differs.

Figure 9 shows the centroids of daily and weekly patterns. These centroids serve as fundamental representations of the underlying temporal dynamics captured within each cluster. Utilizing both quadratic polynomial and cubic polynomial approaches, we discover that the cubic polynomial model best fits our daily clusters and can successfully model the underlying nuances. On the other hand, for weekly patterns, due to their simpler shapes, the quadratic model is enough to fit the centroids. The coefficients of the cubic model (for daily patterns) and the quadratic model (for weekly patterns) are presented in Figure 9.

Complementing our mathematical modeling, we assign descriptive names to the clusters based on their centroid patterns.

Daytime active with rapid decline (D1): Substantial activity during daytime hours with a rapid decrease in activity from peak to off-peak hours.

Nighttime active (D2): Primary activity during the nighttime with gradual workload changes.

Daytime active with gradual decline (D3): Substantial activity during daytime hours with a gradual decrease in activity from peak to off-peak hours.

Weekday Active (W1): Higher activity on weekdays with consistent decrease.

Weekend Active (W2): Increased activity on weekends with steady rise.

Midweek Active (W3): Activity concentrated in midweek, exhibiting smooth changes.

Table V shows the relative frequency of daily and weekly patterns. Notably, D1 emerges as the predominant daily pattern, while W1 stands out as the most common weekly pattern. Moreover, the combination of D1 and W1 represents the most frequent occurrence, encompassing over 43% of the workloads. In Table V, we include the count of workload datasets associated with each pattern within parentheses. For instance, the combination of D1 and W1 appears in six workload datasets. This observation suggests that the identified patterns are not specific to individual datasets, rather, they exist in multiple workload datasets.

We evaluate how time affects workload patterns. Figure 10 illustrates each pattern’s distribution across different time periods. Figure 10(a) compares patterns between weekdays and weekends for each daily cluster. Analyzing the figure reveals that while D1 follows a typical distribution, D2 and D3 diverge from this trend, with D2 showing more weekend data and D3 displaying a higher proportion of weekday data, suggesting variations in workload patterns between weekdays and weekends.

In Figure 10(b), we examine the workload pattern distribution across weekly clusters throughout the seasons. Similar to daily patterns, there are noticeable variations in the distribution of weekly patterns across different seasons. Specifically, cluster W1 shows a higher proportion during the Summer and Winter periods, whereas cluster W3 displays more variability, with a notable increase in percentage during the Fall season.