Skip to content
BY-NC-ND 3.0 license Open Access Published by De Gruyter Open Access June 28, 2011

Evaluating distributed real-time and embedded system test correctness using system execution traces

  • James Hill EMAIL logo , Pooja Varshneya and Douglas Schmidt
From the journal Open Computer Science

Abstract

Effective validation of distributed real-time and embedded (DRE) system quality-of-service (QoS) properties (e.g., event prioritization, latency, and throughput) requires testing system capabilities in representative execution environments. Unfortunately, evaluating the correctness of such tests is hard since it requires validating many states dispersed across many hardware and software components. To address this problem, this article presents a method called Test Execution (TE) Score for validating execution correctness of DRE system tests and empirically evaluates TE Score in the context of a representative DRE system. Results from this evaluation show that TE Score can determine the percentage correctness in test execution, and facilitate trade-off analysis of execution states, thereby increasing confidence in QoS assurance and improving test quality.

[1] S. E. Institute, Ultra-Large-Scale Systems: Software Challenge of the Future, Technical report, Carnegie Mellon University, Pittsburgh, PA, USA, June 2006 Search in Google Scholar

[2] Snow A., Keil M., The Challenges of Accurate Project Status Reporting, Proceedings of the 34th Annual Hawaii International Conference on System Sciences (3–6 January 2001, Maui Hawaii), IEEE Computer Society, 2001, 8043 Search in Google Scholar

[3] Mann J., The Role of Project Escalation in Explaining Runaway Information Systems Development Projects: A Field Study, PhD thesis, Georgia State University, Atlanta, GA, 1996 Search in Google Scholar

[4] Smith C., Williams L., Performance Solutions: A Practical Guide to Creating Responsive, Scalable Software, Addison-Wesley Professional, Boston, MA, USA, 2001 Search in Google Scholar

[5] Technical I.M., Majzik I., Software Monitoring and Debugging Using Compressed Signature Sequences, In: Milligan P., Kuchcinski K. (Eds.), Proceedings of the 22nd EUROMICRO Conference (2–5 September 1996, Prague, Czech Republic), IEEE Computer Society, 1996, 311–318 Search in Google Scholar

[6] Cheon Y., Leavens G.T., A Simple and Practical Approach to Unit Testing: The JML and JUnit Way, In: Magnusson B. (Ed.), Proceedings of the 16th European Conference on Object-Oriented Programming (10–14 June 2002, Malaga, Spain), Springer-Verlag, 2002, 231–255 10.1007/3-540-47993-7_10Search in Google Scholar

[7] Chang F., Ren J., Validating system properties exhibited in execution traces, In: Stirewalt K., Egyed A., Fischer B. (Eds.), ASE’ 07: Proceedings of the twenty-second IEEE/ACM international conference on Automated software engineering (5–9 November, New York, NY, USA), ACM, 2007, 517–520 Search in Google Scholar

[8] Loyall J.P., Gillen M., Paulos A., Bunch L., Carvalho M., Edmondson J., Varshneya P., Schmidt D.C., Martignoni A., Dynamic Policy-Driven Quality of Service in Service-Oriented Systems, Proceedings of the 13th International Symposium on Object/Component/Service-oriented Real-time Distributed Computing (5–6 May 2010, Parador of Carmona, Spain), IEEE Computer Society, 2010 10.1109/ISORC.2010.13Search in Google Scholar

[9] T. N. S. Agency, Global Information Grid, The National Security Agency, www.nsa.gov/ia/industry/gig.cfm?MenuID=10.3.2.2. Search in Google Scholar

[10] Rittel H., Webber M., Dilemmas in a General Theory of Planning, POLICY SCI, 1973, 4, 155–169 http://dx.doi.org/10.1007/BF0140573010.1007/BF01405730Search in Google Scholar

[11] Hill J.H., Turner H.A., Edmondson J.R., Schmidt D.C., Unit Testing Non-functional Concerns of Component-based Distributed Systems, Proceedings of the 2nd International Conference on Software Testing, Verification, and Validation (1–4 April 2009, Denver, Colorado, USA) IEEE Computer Society, 2009, 406–415 10.1109/ICST.2009.44Search in Google Scholar

[12] Hill J.H., Slaby J., Baker S., Schmidt D.C., Applying System Execution Modeling Tools to Evaluate Enterprise Distributed Real-time and Embedded System QoS, Proceedings of the 12th International Conference on Embedded and Real-Time Computing Systems and Applications (August 2006 Sydney, Australia) IEEE Computer Society, 2006, 350–362 10.1109/RTCSA.2006.17Search in Google Scholar

[13] Lédeczi Á., Bakay Á., Maróti M., Völgyesi P., Nordstrom G., Sprinkle J., Karsai G., Composing Domain-Specific Design Environments, COMPUTER, 2001, 34, 44–51 http://dx.doi.org/10.1109/2.96344310.1109/2.963443Search in Google Scholar

[14] Czarnecki K., Eisenecker U.W., Generative Programming: Methods, Tools, and Applications, Addison-Wesley, Reading, Massachusetts, 2000 Search in Google Scholar

[15] Liu N.X., Baras J.S., Modelling multi-dimensional QoS: Some fundamental constraints: Research articles, INT J COMMUN SYST, 2004, 17, 193–215 http://dx.doi.org/10.1002/dac.65210.1002/dac.652Search in Google Scholar

[16] Hill J.H., Schmidt D.C., Edmondson J., Gokhale A., Tools for Continuously Evaluating Distributed System Qualities, IEEE SOFTWARE, 2010, 27, 65–71 http://dx.doi.org/10.1109/MS.2009.19710.1109/MS.2009.197Search in Google Scholar

[17] Downs E., Clare P., Coe I., Structured Systems Analysis and Design Method: Application and Context, Prentice Hall International (UK) Ltd., Hertfordshire, UK, 1988 Search in Google Scholar

[18] Singhal M., Shivaratri N.G., Advanced Concepts in Operating Systems, McGraw-Hill, Inc., New York, NY, USA, 1994 Search in Google Scholar

[19] Hill J.H., Gokhale A.S., Model-driven engineering for early QoS validation of component-based software systems, J SOFTW, 2007, 2, 9–18 10.4304/jsw.2.3.9-18Search in Google Scholar

[20] Moe J., Carr D.A., Understanding Distributed Systems via Execution Trace Data, The Proceedings of the Ninth International Workshop on Program Comprehension (12–13 May 2001, Toronto, Canada) IEEE Computer Society, 2001, 60–67 Search in Google Scholar

[21] Baker B.S., Parameterized Pattern Matching by Boyer-Moore-type Algorithms, Proceedings of the sixth annual ACM-SIAM symposium on Discrete algorithms (22–24 January 1995, San Francisco, California) Society for Industrial and Applied Mathematics, 1995, 541–550 Search in Google Scholar

[22] Fowler M., Continuous Integration, www.martinfowler.com/articles/continuousIntegration.html Search in Google Scholar

[23] Massol V., Husted T., JUnit in Action, Manning Publications Co., Greenwich, CT, USA, 2003 Search in Google Scholar

[24] Tian J., Lu P., Palma J., Test-execution-based reliability measurement and modeling for large commercial software, IEEE T SOFTWARE ENG, 1995, 21, 405–414 http://dx.doi.org/10.1109/32.38747010.1109/32.387470Search in Google Scholar

[25] Lee J., Kuo J.-Y., New Approach to Requirements Trade-Off Analysis for Complex Systems, IEEE T KNOWL DATA EN, 1998, 10, 551–562 http://dx.doi.org/10.1109/69.70605610.1109/69.706056Search in Google Scholar

Published Online: 2011-6-28
Published in Print: 2011-6-1

© 2011 Versita Warsaw

This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 3.0 License.

Downloaded on 22.9.2024 from https://www.degruyter.com/document/doi/10.2478/s13537-011-0012-2/html
Scroll to top button