Presentation of a PhD dissertation with the following contributions: (i) a robustness test framework for HA middleware systems, (ii) analysis of the semantics of UML 2 Sequence Diagrams, (iii) definition of a test framework and test language (TERMOS) for mobile computing systems.
Languages and frameworks for specifying test artifacts
1. Languages and frameworks for
specifying test artifacts
PhD dissertation public defense
Zoltán Micskei
Advisors: István Majzik (BME)
Hélène Waeselynck (LAAS-CNRS)
2. Software testing concepts
Test framework = test artifacts + supporting tools
2
Testing process:
Test artifacts:
Specification,
requirements
Test cases Verdicts
Test
execution
Specification,
requirements
Test
requirements
Test purposes
Test suite
(test cases) Test traces Verdicts
Test approach
Test
execution
Test
evaluation
Test oracle
3. Test languages
Specific languages for each artifact:
o E.g. U2TP, TTCN-3, ATML …
Requirements and purposes:
o Graphical scenario languages
o MSC, LSC, UML 2 Sequence Diagrams
3
4. New application domains (1)
High availability (HA) middleware systems
Standardizing HA functions
o SA Forum AIS specification
Characteristics
o State-based nature
o Robustness is a key factor
Challenges
o Test languages do not address robustness testing
o HA middleware is a complex system
4
5. New application domains (2)
Context-aware mobile computing systems
Mobile computing system
o laptop, phone, intelligent car…
Characteristics
o Context awareness
o Dynamic, evolving environment
o Communication with unknown partners
Challenges
o Describing dynamic configurations
o Extensions for describing contexts
5
8. Contents
Preliminaries and objectives
New scientific results
o Robustness testing of standard-based HA middleware
o Semantic choices in UML 2 Sequence Diagrams
o A test language and framework for mobile systems
Conclusion
8
9. Research question
Review of previous API robustness testing
Identifying sources for activating failures
9
11. Systematic design of test tools
1. Collecting test artifacts’
requirements
2. Designing the new
test languages
3. Developing test tools
11
E.g.: type-specific testing
Required:
• Definition of functions
• Definition of types
• Selecting test values
12. Case studies
Objective: Can the test framework be used to
assess the robustness of HA middleware systems?
o Qualitative observations
Unit of analysis: OpenAIS, OpenSAF, SAFE4TRY
Data collection: exit and error codes, logs
Case studies performed:
1. Can the complex techniques detect more failures?
2. Can the three techniques uncover different failures?
3. Can the different implementations be compared?
12
13. Thesis 1
Thesis 1: Robustness testing of HA middleware
o Following a systematic method I identified potential
activation modes of robustness faults (including
activation through stateless API, stateful API and
underlying services), designed languages to represent
the related test artifacts, and developed algorithms for
tools that use these languages to generate test data.
o I implemented the languages and tools in a test
framework, which can compare the robustness of
standard specification-based middleware
implementations.
Collaboration: Francis Tam (Nokia Research Center)
13
14. Related publications
Z. Micskei, H. Madeira, A. Avritzer, I. Majzik, M. Vieira, and N. Antunes. “Robustness Testing
Techniques and Tools”. In: Resilience Assessment and Evaluation of Computing Systems. Ed. by K.
Wolter, A. Avritzer, M. Vieira, and A. van Moorsel. Springer, 2012, pp. 323–339. doi: 10.1007/978-3-
642-29032-9_16
Z. Micskei, I. Majzik, and F. Tam. “Comparing Robustness of AIS-Based Middleware Implementations”.
In: International Service Availability Symposium (ISAS 2007). Ed. by M. Malek, M. Reitenspieß, and A.
van Moorsel. Vol. 4526. LNCS. Springer, May 2007, pp. 20–30. doi:10.1007/978-3-540-72736-1_3
A. Kövi and Z. Micskei. “Robustness Testing of Standard Specifications-Based HA Middleware”. In:
30th International Conference on Distributed Computing Systems Workshops (ICDCSW). IEEE, June
2010, pp. 302–306. doi: 10.1109/ICDCSW.2010.73
Z. Micskei, I. Majzik, and F. Tam. “Robustness Testing Techniques for High Availability Middleware
Solutions”. In: International Workshop on Engineering of Fault Tolerant Systems (EFTS2006).
Luxembourg, Luxembourg: University of Luxembourg, June 2006, pp. 55–66
Z. Micskei. “Robustness Comparison of High Availability Middleware Systems”. In: 14th PhD Mini-
Symposium. Budapest University of Technology and Economics. Budapest, Hungary: Department of
Measurement and Information Systems, Feb. 2007, pp. 70–73
Z. Micskei. “Robustness Testing of High Availability Middleware Solutions”. In: 13th PhD Mini-
Symposium. Budapest University of Technology and Economics. Budapest, Hungary: Department of
Measurement and Information Systems, Feb. 2006, pp. 36–37
Z. Micskei. “Nagy Rendelkezésre Állást Biztosító Köztes Rétegek Robosztusság Tesztelése”. In: Proc. of
Tavaszi Szél 2006. Kaposvár, Hungary: Doktoranduszok Országos Szövetsége, May 2006, pp. 295–298
14
15. Contents
Preliminaries and objectives
New scientific results
o Robustness testing of standard-based HA middleware
o Semantic choices in UML 2 Sequence Diagrams
o A test language and framework for mobile systems
Conclusion
15
16. Research question
Review of the OMG spec. and 13 formal semantics
Categorizing semantic choices
Evaluating the different options for the choices
16
17. Goal: easy to use presentation of results
Notation inspired by feature models:
„Test” diagrams and traces:
17
sd c2
a : A b : B
m1
Are the following traces valid, invalid or
inconclusive?
!m1.?m1.!m2.?m2.!m3.?m3.!m1.?m1
!m1.?m1.!m4.?m4.!m2.?m2.!m3.?m3
!m1.?m1.!m3.?m3.!m2.?m2.!m3.?m3
!m1.?m1.!m1.?m1.!m2.?m2.!m3.?m3
m3
m2
19. Thesis 2
Thesis 2: I identified and categorized the semantic
choices and available options in UML 2 Sequence
Diagrams. I gave a structured framework with an
easy to use feature-model like representation of
the available options that can be used to adapt
the semantics of the language to a specific
purpose.
19
20. Related publications
Z. Micskei and H. Waeselynck. “The many meanings of UML 2
Sequence Diagrams: a survey”. In: Software and Systems Modeling
10.4 (2011), pp. 489–514. doi: 10.1007/s10270-010-0157-9
Z. Micskei and H. Waeselynck. A survey of UML 2.0 sequence
diagrams’ semantics. Tech. rep. 08389. Laboratoire d’Analyse et
d’Architecture des Systemes (LAAS), Aug.2008, pp.1–37
20
21. Contents
Preliminaries and objectives
New scientific results
o Robustness testing of standard-based HA middleware
o Semantic choices in UML 2 Sequence Diagrams
o A test language and framework for mobile systems
Conclusion
21
22. Research question
Case study:
Mobile Group
Membership
Protocol (GMP)
22
Test language and framework for mobile systems
Characteristics
Research question
Context awareness
Communication with
local broadcast
How can dynamic communication structures be specified in testing artifacts?
Dynamic, evolving
environment
23. Proposed notation for context information
Two views: event (left) and spatial (right)
Extending graphical scenario languages
23
24. Test approach and framework
24
Requirements
TERMOS language
- event view: messages
- spatial view: configurations
Execution environment
Network
simulator
Context
controller
Application execution
SUT
Test
components
Execution trace
Trace evaluation Verdict
GraphSeq tool
Configuration matching
25. TERMOS language
Test Requirement language for Mobile Setting
Defining the modeling language
o Abstract syntax: profile + restricting UML
o Concrete syntax: existing graphical elements
o Well-formedness: constraints for verification
o Semantics: based on Thesis 2’s framework
(goal: evaluating test traces),
formal operational semantics
25
27. Thesis 3
Thesis 3: I designed a test requirement language that can
be used in the domain of mobile systems.
o I defined the syntax of the language using extensions to the
UML Sequence Diagrams’ metamodel, and its semantics using
an automaton-based formal operational semantics.
o The language is capable of expressing local broadcasts and
changes in the communication topology, and has the necessary
syntactic and semantic choices to make the specified
requirements checkable.
Collaboration: Nicolas Rivière, Minh Duc Nguyen (LAAS)
27
28. Related publications
G. Pintér, Z. Micskei, A. Kövi, Z. Égel, I. Kocsis, G. Huszerl, and A. Pataricza.
“Model-Based Approaches for Dependability in Ad-Hoc Mobile Networks and
Services”. In: Architecting Dependable Systems V. Springer, 2008, pp. 150–174.
doi: 10.1007/978-3-540-85571-2_7
H. Waeselynck, Z. Micskei, N. Rivière, Á. Hamvas, and I. Nitu. “TERMOS: a Formal
Language for Scenarios in Mobile Computing Systems”. In: Mobile and
Ubiquitous Systems: Computing, Networking, and Services (MobiQuitous 2010).
Ed. by P. Sénac, M. Ott, and A. Seneviratne. Sydney, Australia, Dec. 2010, pp.
285–296. doi: 10.1007/978-3-642-29154-8_24
H. Waeselynck, Z. Micskei, M.D. Nguyen, and N.Rivière. “Mobile Systems from a
Validation Perspective: a Case Study”. In: Sixth International Symposium on
Parallel and Distributed Computing (ISPDC ’07). Ed. by D. Kranzlmüller, W.
Schreiner, and J. Volkert. Hagenberg, Austria: IEEE Computer Society, July 2007,
pp. 85–92. doi: 10.1109/ISPDC.2007.37
Z. Micskei. “Specifying Tests for Ad-Hoc Mobile Systems”. In: 15th PhD Mini-
Symposium. Budapest University of Technology and Economics. Budapest,
Hungary: Department of Measurement and Information Systems, Feb. 2008, pp.
32–35
Z. Micskei, H. Waeselynck, M. D. Nguyen, and N. Rivière. Analysis of a group
membership protocol for Ad-hoc networks. Tech. rep. 06797. Laboratoire
d’Analyse et d’Architecture des Systemes (LAAS), Nov. 2006, pp. 1–42
28
29. Contents
Preliminaries and objectives
New scientific results
o Robustness testing of standard-based HA middleware
o Semantic choices in UML 2 Sequence Diagrams
o A test language and framework for mobile systems
Conclusion
29
30. Application of new results
Comparing robustness of HA middleware
o OpenAIS, OpenSAF, SAFE4TRY implementations
o Reporting faults to the developers
o Publication of the test suite and the results
Testing mobile systems
o TERMOS language: HIDENETS EU FP6 project
o Extending results in the EU ARTEMIS R3-COP project
30