● Software Testing Laboratory @ MDH
● Test Automation
● Quality of Tests
● Developers Testing
● AGENTS - KKS funded project
● Bombardier Transportation
● Involved in the TOCSYC
● Intelligent Embedded Systems
Master Program Coordinator
● ATAC research project
● Funded by ITEA2/Vinnova
● Advanced Test Automation for Complex Software-
Cooperation with Bombardier
“The project has brought test automation
on the agenda of European industry.”
- ITEA3 Final Review Report
● Train Control Management System (TCMS)
● a high capacity, infrastructure backbone
● the center of the distributed system
● Functions controlled by TCMS include
● collecting line voltage,
● controlling the train engines,
● opening and closing the train doors,
● upload of diagnostic data.
● Vehicle Integration testing
● System Integration testing
● Modules integration testing
● Modules testing
● “Get to the driver’s perspective as soon as possible”
… and their testing
of a Developer!
● IEC 61131-3
● Function Block Diagram
● Graphical programming language
● Based on element composition
● Various predefined elements
● Strict measurements mandated by safety standards
● No tool support to measure coverage on FBD
● Creating tests just to reach the coverage leaves very little
time for functional testing
… with module testing
Transformation 1 Annotation2
Test traces Test cases
(Inputs) (0 1 23)
(Outputs) (1 2 543,9)
”Automated Test Generation using Model-Checking: An Industrial Evaluation”, Eduard Paul Enoiu, Adnan
Causevic, Thomas J. Ostrand, Elaine J. Weyuker, Daniel Sundmark, Paul Pettersson. International Journal
on Software Tools for Technology Transfer, 2014, Springer.
● Time required to generate tests
● satisfying the DC, CC and MC/DC logic coverage criteria
Case Study at BT
● For 34 of the 157 programs, the tool did not terminate
after running for a substantial period of time.
● Cut-off time was set to 10 minutes.
● There is an indication that as the number of decisions
increases, the performance deteriorates and the cost of
using the tool may become prohibitive.
● But, if you have so many decisions, how would you test
Case Study at BT (2)
● Should a developer generate the tests from scratch?
● Should a developer provide manual tests first?
● How many (or how much in terms of coverage)?
● When should a developer use the Validate Test button?
● Should developers use this tool at all or should we
automate that as well (nightly build, etc.) ?
● Internal behavior of each function block has to be
● Currently, blocks from the standard language are modeled
● and a few custom made by Bombardier which are often used
● State-space explosion
● a known problem of model-checking
● large number of Boolean or Integer input values
● but, again, how would you do it manually ?
● Complementing model-checking with:
● Static analysis of FBD’s
● Search-based software testing
● Trying some new approaches:
● Different model-checkers, SMT solvers, …
● Measuring effectiveness of generated tests
● In terms of how good they are in finding faults
● Experimenting with FBD programs from
On-going and Future work