Abstract
This paper presents a comprehensive statistical analysis of a variety of workloads collected on production clusters and Grids. The applications are mostly computational-intensive and each task requires single CPU for processing data, which dominate the workloads on current production Grid systems. Trace data obtained on a parallel supercomputer is also included for comparison studies. The statistical properties of workloads are investigated at different levels, including the Virtual Organization (VO) and user behavior. The aggregation procedure and scaling analysis are applied to job arrivals, leading to the identifications of several basic patterns, namely pseudo-periodicity, long range dependence (LRD), and multifractals. It is shown that statistical measures based on interarrivals are of limited usefulness and count based measures should be trusted when it comes to correlations. Other job characteristics like run time and memory consumption are also studied. A “bag-of-tasks” behavior is empirically evidenced, strongly indicating temporal locality. The nature of such dynamics in the Grid workloads is discussed. This study has important implications on workload modeling and performance predictions, and points out the need of comprehensive performance evaluation studies given the workload characteristics.
Article PDF
Similar content being viewed by others
Avoid common mistakes on your manuscript.
References
Abry P, Veitch D (1998) Wavelet analysis of long-range dependent traffic. IEEE Trans Inf Theory 44(1):2–15
Abry P, Baraniuk R, Flandrin P, Riedi R, Veitch D (2002) The multiscale nature of network traffic: discovery, analysis, and modelling. IEEE Signal Process Mag 19(3):28–46
Abry P, Taqqu MS, Flandrin P, Veitch D (2000) Self-similar network traffic and performance evaluation. In: Park K, Willinger W (eds) Wavelets for the analysis, estimation, and synthesis of scaling data. Wiley, New York
Abry P, Veitch D, Flandrin P (1998) Long-range dependence: revisiting aggregation with wavelets. J Time Ser Anal 19(3):253–266
Barabasi A-L (2005) The origin of bursts and heavy tails in human dynamics. Nature 435:207–211
Chainais P, Riedi RH, Abry P (2005) On non-scale-invariant infinitely divisible cascades. IEEE Trans Inf Theory 51(3):1063–1083
Cirne W, Berman F (2001) A comprehensive model of the supercomputer workload. In: Proceedings of IEEE 4th annual workshop on workload characterization
Faubechies I (1992) Ten lectures on wavelets. BMS-NSF reg. conf. series in applied math. SIAM, Philadelphia
Feitelson DG (2002) Workload modeling for performance evaluation. In: Lecture notes in computer science, vol 2459. Springer, Berlin, pp 114–141
Feitelson DG (2006) Workload modeling for computer systems performance evaluation. draft version 0.7
Feldmann A, Gilbert AC, Willinger W (1998) Data networks as cascades: Investigating the multifractal nature of Internet WAN traffic. In: SIGCOMM, pp 42–55
Iosup A, Dumitrescu C, Epema D, Li H, Wolters L (2006) How are real grids used? The analysis of four grid traces and its implications. In: Proceedings of 7th IEEE international conference on grid computing (Grid’06)
Jagerman DL, Melamed B, Willinger W (1996) Stochastic modeling of traffic processes. In: Frontiers in queueing: models, methods and problems
Leland W, Taqqu M, Willinger W, Wilson D (1994) On the self-similar nature of Ethernet traffic (extended version). IEEE/ACM Trans Netw 2(1):1–15
Li H, Muskulus M (2006) Analysis and modeling of job arrivals in a production grid. In: ACM SIGMETRICS performance evaluation review, Dec 2006
Li H, Groep D, Wolters L (2005) Workload characteristics of a multi-cluster supercomputer. In: Lecture notes in computer science, vol 3277. Springer, Berlin, pp 176–193
Lowen SB, Teich MC (2005) Fractal-based point processes. Wiley, New York
Lublin U, Feitelson DG (2003) The workload on parallel supercomputers: modeling the characteristics of rigid jobs. J Parallel Distrib Comput 63(11):1105–1122
Medernach E (2005) Workload analysis of a cluster in a grid environment. In: Proceedings of 11th workshop on job scheduling strategies for parallel processing
Paxson V (1997) Fast, approximate synthesis of fractional Gaussian noise for generating self-similar network traffic. Comput Commun Rev 27(5):5–18
Polana R, Nelson R (1993) Detecting activities. In: Proceedings of IEEE CVPR
Riedi RH (2002) Long range dependence: theory and applications. In: Doukhan, Oppenheim, Taqqu (eds) Multifractal processes. Birkhauser, Basel, pp 625–715
Riedi RH, Willinger W (2000) Self-similar network traffic and performance evaluation. In: Park K, Willinger W (eds) Toward an improved understanding of network traffic dynamics. Wiley, New York
Riedi RH, Crouse MS, Ribeiro VJ, Baraniuk RG (1999) A multifractal wavelet model with application to network traffic. IEEE Trans Inf Theory 45(3):992–1019
Ross SM (2003) Introduction to probability models, 8th edn. Academic Press, San Diego
Song B, Ernemann C, Yahyapour R (2004) Parallel computer workload modeling with Markov chains. In: Lecture notes in computer science, vol 3277. Springer, Berlin, pp 47–62
Squillante MS, Yao DD, Zhang L (1999) The impact of job arrival patterns on parallel scheduling. ACM SIGMETRICS Perform Eval Rev 26(4):52–59
Strang G, Nguyen T (1996) Wavelets and filter banks. Wellesley-Cambridge Press, Cambridge
Vehel JL, Riedi R (1997) Fractional Brownian motion and data traffic modeling: The other end of the spectrum. In: Fractals in engineering. Springer, Berlin, pp 185–202
Veitch D, Abry P (1999) A wavelet based joint estimator of the parameters of long-range dependence. IEEE Trans Inf Theory 45(3):878–897. Special issue on “Multiscale statistical signal analysis and its applications”
Wornell GW (1993) Wavelet-based representations of the 1/f family of fractal processes. Proc IEEE 81(10):1428–1450
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
Open Access This is an open access article distributed under the terms of the Creative Commons Attribution Noncommercial License (https://creativecommons.org/licenses/by-nc/2.0), which permits any noncommercial use, distribution, and reproduction in any medium, provided the original author(s) and source are credited.
About this article
Cite this article
Li, H. Workload dynamics on clusters and grids. J Supercomput 47, 1–20 (2009). https://doi.org/10.1007/s11227-008-0189-x
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11227-008-0189-x