1. Notes on Teaching Software
Testing
Alpen-Adria-Universität, Klagenfurt, 28 March 2013
Ladislav Samuelis, Technical University of Košice, Slovakia

2. Motivation
• For years I was notoriously curious about the latest
updates in program development.
• The main question:
– How a program “is born”. Later changed to “what is software
evolution”.
• Challenge: grant of the Slovak Ministry of Education:
“Implementation of Current Software Engineering
Courses for Computer Science Programs …”
Ladislav Samuelis: Notes on Teaching
2 of 23
Software Testing

3. Dawn of software testing
• P. Naur, 1969: “If you develop programs
systematically, you can dispense with testing
altogether.”
• This hope to achieve the goal is still present.
• However, in practice testing cannot be avoided by
any other means, whether it is the most careful
design, construction, or formal analysis of the
requirements.
• Elimination of testing from the development
process just postpones the test with reality until
the product is put into production.
Ladislav Samuelis: Notes on Teaching
3 of 23
Software Testing

4. Lehman’s law
• E-type software, software implementing real
world computer applications, must be
continually evolved to maintain user
satisfaction, generally over several years and
releases.
• It is likely that the responsible organisation
will, sooner or later, consider its replacement
through redevelopment or acquisition. (1974)
Ladislav Samuelis: Notes on Teaching
4 of 23
Software Testing

5. Specific of software testing
• Testing is a part of the development
process, not the manufacturing process.
• This testing is different from many other
engineering products, where the
emphasis is on testing the correct
reproduction.
Ladislav Samuelis: Notes on Teaching
5 of 23
Software Testing

6. What are the questions?
• What are we teaching in
software testing courses and
why?
• What are the scientific
foundations of testing?
Ladislav Samuelis: Notes on Teaching
6 of 23
Software Testing

7. Why should we have courses on
software testing?
• Developers spend 50% of their time testing,
and reliability requirements are steadily
increasing.
• All programmers spend a significant amount
of time to trial their software.
• It gets negligible significance in most software
development curricula.
• Accidents, ARIANE …
Ladislav Samuelis: Notes on Teaching
7 of 23
Software Testing

8. What are we teaching in software
testing courses?
• Techniques, measurements, management …
– Plenty of comprehensive books …
• ISTQB (International Software Testing
Qualification Board)
– Effective for undergraduate informatics study-
programmes
– Required by the industry
Ladislav Samuelis: Notes on Teaching
8 of 23
Software Testing

9. Notes on obstacles in building
software testing course
• How to avoid the phenomenon of “bag
of tricks”?
• It is not easy to design a specific
curriculum with adequate labs and
assessment procedures.
• The border between the program
analysis and testing is fuzzy.
Ladislav Samuelis: Notes on Teaching
9 of 23
Software Testing

10. How do we cooperate with the
industry?
• RWE IT local company provides selected
lectures on:
– testing of large projects
– testing techniques
– test cases by examples
• Diploma theses and internships
• Experimental data published in papers
• Workshop biannually for prospective
employees
Ladislav Samuelis: Notes on Teaching
10 of 23
Software Testing

11. Dijkstra’s “law”
• “Program testing can be used to show the presence
of bugs, but never to show their absence”. (1969)
• Donald Knuth:
– Beware of bugs in the above code; I have only
proved it correct, not tried it.
This law is expresses the limitations of black-box
testing.
Question:
How do we explain the theory of this “law” to
students?
Ladislav Samuelis: Notes on Teaching
11 of 23
Software Testing

12. On the limits of formal methods
(D.L.Parnas)
Simplification (model) is achieved by ignoring
certain facts such as the limits in the sizes of
data elements and the errors in arithmetic
operations. Unfortunately, these are exactly the
type of details that can cause faults and lead to
failures.
No formal analysis of such a model that leaves
out critical limits can reveal faults attributable
to those limits.
Ladislav Samuelis: Notes on Teaching
12 of 23
Software Testing

13. The theory of program testing
• The idea of program testing is as old as
programming.
• 1970 – new field of research called testing theory
emerged – efficiency and effectivity
• What are the main questions?
– How to design test cases from different sources:
specification, source code, input and output domains
of programs
– How to select a subset of test cases from the set of all
possible test cases?
Ladislav Samuelis: Notes on Teaching
13 of 23
Software Testing

14. A scheme of testing
(a mathematical approach)
Ladislav Samuelis: Notes on Teaching
14 of 23
Software Testing

15. The theory of Goodenough and
Gerhart
(very simplified explanation of the ideal test)
P “runs” and gives B number of bugs.
We create new set Tk until Bk ≥ B. Then
it is an ideal test.
• D = input domain, T≤D, d є D
• P= program, result is P(d)
• d is true, if output P(d) is acceptable
Ladislav Samuelis: Notes on Teaching
15 of 23
Software Testing

16. Other testing theories
• Theory of Weyuker and Ostrand, Gourlay
• Dines Bjørner: On a Triptych of Software
Development
– Domain engineering,
– Requirements engineering,
– Design
• SEMAT initiative
Question:
How much testing theory do we have to teach?
Ladislav Samuelis: Notes on Teaching
16 of 23
Software Testing

17. Power law
The notion of power laws has been around for
more than a century.
P(x) ~ c.x−k
Question:
What does it mean in software testing?
Ladislav Samuelis: Notes on Teaching
17 of 23
Software Testing

18. Power law and software testing
• Object-oriented software ~ network of
objects.
• It is worth to test less complex classes than
many trivial classes.
Note:
This is an empirical “law” or concept , rarely
taught in testing courses.
Ladislav Samuelis: Notes on Teaching
18 of 23
Software Testing

19. Gödel’s incompleteness theorem
• No computer program is capable of
proving all programs.
• For any consistent system of axioms
there will always be statements that
are improvable within the system.
Ladislav Samuelis: Notes on Teaching
19 of 23
Software Testing

20. Teaching of Gödel's incompleteness
theorems?
• Elaine Weyuker - Evaluating Software Test
Data Adequacy (1988)
Question:
Where are the places of Gödel’s theorems in
software engineering courses?
Ladislav Samuelis: Notes on Teaching
20 of 23
Software Testing

21. Note on testing and deductive
reasoning
• Deductive reasoning;
– Major premise – initial set of facts
– Minor premise – goal fact
• For example, PROLOG program –
the inference engine gives the verdict.
• Not-PROLOG program – no inference engine,
program is executed and tester “gives a
verdict” against requirements.
Ladislav Samuelis: Notes on Teaching
21 of 23
Software Testing

22. Questions on the role of observation in
software testing
Questions:
Why do we select observations?
Where do we perform the observation?
In which phase of the product lifecycle do
we perform observations?
Ladislav Samuelis: Notes on Teaching
22 of 23
Software Testing

23. Notes on the history of testing
concurrent programs
• Before object-oriented programming:
– mechanisms for ensuring mutual exclusion
(Dijkstra),
– monitors (Hoare) and
– critical regions (Brinch-Hansen)
Question:
What topics to teach in concurrent program
testing?
Ladislav Samuelis: Notes on Teaching
23 of 23
Software Testing