Доклад Mike Smith на конференции SQA Days-19, 20-21 мая 2016 г., Санкт-Петербург

1. Are you in control of Testing, or does
Testing control you?
Mike Smith
Independent Consultant

2. 40th
Year in IT!

3. I’ve lived through a few technology eras!

4. Getting the balance right can be difficult!

5. Trying to stay relevant
• Tool Author – Test Process Management (T-Plan)
• Businessman/Entrepreneur/Tester/Innovator
• UK Representative to ISTQB
• Chairman ISTQB Advanced Level Syllabus Working Group
• Founder member of CAT (Certified Agile Tester) SIG
• Practical Assessments (R&D Project)
• Web TV & Digital Marketing in sport
• Professional sports governance (in my spare time!)
• Writing a book!

6. Presentation Background
• Lack of control over projects/initiatives in IT, Business &
Sport
• Common faults over this broad perspective
• This presentation focuses on software testing as a form
of measurement
• But it uses analogies and common characteristics across
other perspectives
• I will use some examples and my experiences to help
highlight key points

7. Presentation Background
Concerns over effectiveness of software testing as
a measurement process
The subject of my book – Measurement & Testing!

8. My theme for a book
“Effective Measurement and Testing Delivers
Control of Anything”

9. Measurement & Testing
My Axioms
• Testing is more than just Testing!
• Software Testing can learn from other forms of
Measurement & Testing
• Other forms of Measurement & Testing can learn
from Software Testing
• Agile Development is helping the case for
Measurement & Testing
• Software Testing took a wrong turn when it
became obsessed with Test Cases!

10. Another Testing Axiom
•The answer to the ultimate question of Life, The Universe
and Everything?
Answer: “42”
(Hitchhikers Guide to the Galaxy – Douglas Adams)
•The answer to any question about Software Testing?
Answer: “It Depends!”
(Dale Perry – SQE Inc. USA)

11. 3 Key Words
Control
Measurement
Testing (& Software Testing)

12. Control – Some definitions
•The power to influence or direct people's behaviour or the
course of events. (lead)
•Determine the behaviour or supervise the running of. (lag
& lead)

13. Control vs Management
What’s the difference?
Corporate Governance Definition
1.Strategic Direction of the Business
1.Effective Monitoring and Control of the
Management of the Business
2.Accountable to the Shareholders of the Business

14. Control vs Management?

15. Being in Control – Can you answer
these questions?
•What are my measures of success?
•How will I know when I’ve met them?
•How sure will I be that I’m going to meet them at
any point in time?
•How do I know where I am against them at any
point in time?
•Can I manage change effectively?
•For an IT project, how useful are Test Cases to
help me keep in control?

16. Successful Outcomes?
• Reach destination port
• Get there on schedule
• All passengers & crew are safe
• Passengers have not been (too) seasick!
• Within budget (economical with fuel)
• Obstacles avoided!

17. How will I know when I’ve met the
successful outcomes?
• What are the key measures & targets?
• Speed?
• Direction?
• Position?
• Quality Attributes?

18. Keeping the ship under control!
Key Drivers
Initiatives taken based these
key goal & performance drivers

19. Other measures to support the key drivers for
Control

20. There may be many management systems to
support these measures
A complex measurement
& management framework
is needed

21. Simple Hierarchical Model does not support
the complex relationships

22. What is Business Performance
Measurement & Management?
• ‘Translating Strategy into Action’
– Kaplan & Norton
• Use of Business Balanced Scorecard
– Different perspectives
• Drives behaviour
• Measures outcomes
• Links actions to strategy
• Leads to predictable outcomes

23. Dangers of Performance Measurement &
Management?
Measures that are badly constructed can
destroy an organisation’s performance… effort
may be misdirected into unproductive actions
and a great deal of time and effort can be
wasted.
‘Balanced Scorecard in a week – Mike Bourne, Pippa Bourne’

24. 5 Key principles
1. Generic application
2. Objectives, Measures & Targets, Initiatives
3. Separate “What” & “How”
4. Cascading Scorecards
– One person’s ‘How’ is another person’s ‘What’
– Measures & Targets become objectives for next
person
1. Lead & Lag Indicators
– Goal Indicators (reactive)
– Performance Indicators (predicative)

25. Scorecard Objectives, Measures, Targets &
Initiatives
InitiativesObjectives TargetsMeasures
WHY WHAT HOW

26. Scorecards linked by
cascading ‘whats’ & ‘hows’
HOWWHAT
HOWWHAT
HOWWHAT
• Could be
1:many or
many:many
• Simplest way
is 1:1
HOW
HOWWHAT
HOWWHAT
HOWWHAT
WHAT

27. Summary of Outcome Driven Measurement
Framework/Model
• Separate “What” & “How”
• Complex Relationships in Model
• Information Traceability matrix Top to
Bottom
• Control of Complex Organisations &
Systems
• Good fit with Agile Development
Practices!

28. Now consider Software Testing Model
• What are the Measures & Targets that support
the Software Testing Process?
• Test Cases?
• Test Objectives?
• Test Requirements?
• Test Conditions?
• Testers focus on Test Cases, but are they the
best Measures/Targets to drive behaviour?
• Are they the best to manage risk and change?

29. The problem with Test Cases

30. Test(ing) & Software Testing
•The means by which the presence, quality, or genuineness of anything
is determined; a means of trial (lag)
•Software testing is a process of executing a program or application with
the intent of finding the software bugs. (lag) It can also be stated as the
process of validating and verifying that a software program or application
or product meets the business and technical requirements that guided it's
design and development. (lag & lead)
•Testing is a risk reduction process (lead)

31. The problem with Test Cases
Test case
A set of input values, execution preconditions,
expected results and execution postconditions,
developed for a particular objective or test
condition, such as to exercise a particular
program path or to verify compliance with a
specific requirement. [After IEEE 610]

32. The problem with Test Cases
• How big?
• When can they be defined?
• They may help me Test, but do they help me
Measure?
• Whose Measures are they?
• Do they separate the What from the How?
• History of Test Cases (‘Bottom-up’)
• What else fails if my Test Case fails - traceability?
• Tool problems – wrong model
• What if none defined?

33. The problem with Test Cases

34. 1479 test cases: but are they “useful”?
Now, let’s
start with a
classification
tree
1479
test cases, so
it must be good,
right?
Test
specification
process
Documentation
to agree
coverage

35. A Simple Hierarchical World

36. A Complex Relational World!

37. A simple world became a complex world!

38. Test Model described by ‘methods’ people at large pan-
European financial institution.
Test
Requirements
Requirements
WHY
Test Design
Specifications
Test Case
Specifications
WHAT WHAT HOW
• “From Requirements to Test Case Specification”
in test management tool

39. A complex model to support software testing
May or may not
Need complex
model:
IT DEPENDS!

40. Guidelines for a better model
• Need to separate What & How
• Model needs Complex Relationships – many:many
• Test Conditions can be Considered targets for
success, organised by Measurement Attributes
– Functional, Behavioural
• Can and should be defined early
• Ability to cope with change
• Know where you are at any point in time
• Delivers Control of Complex Systems
• Good fit with Agile Development Practices!
• Business Outcome Driven Development

41. What factors affect the level of measures
and targets in a measurement framework?
• Risk
• Size & Complexity
• Dev Method
• Test Level
• Manageability & Maintainability
• Tool usage
• Ability to cope with change
• Quality and level of artefacts
• Regulatory requirements…....

42. What factors affect the level of measures
and targets in a measurement framework?
IT DEPENDS!!
Setting good measures and targets is
difficult - it requires strong analytical skills
People often neglect the –ve measures
and focus on the +ve

43. What can the world of Measurement learn
from Software Testing?
• Software Testing has been developed as a
discipline for many years
• Very detailed processes and techniques
compared with other forms of Measurement &
Testing
• Software Test Techniques
– Support Analysis and Design Activities
• Airport Example

44. Other Experiences
• Making a new Financial
Exchange Live
• An Office Move
• ISTQB Syllabus Development
• Practical Assessments
• Governance in Sport
• Mindset change and silos!

45. WHAT’S MISSING?
General Model
Control
Management &
Information
Systems
Measurement
Framework
Measures &
Targets
Objectives
Initiatives

46. WHAT’S MISSING?
Testing Model
Control
Tests and Test
Cases
Test Requirements/
Test Conditions
Measures & Targets
Separated from Tests
Many:Many Reqs:Tests
Objectives/
Requirements
Initiatives

47. WHAT’S MISSING?
New Financial Exchange
New Market
Tests and Test
Cases
Regulator Approval
SPECIFICATION
Measurement
Framework
Retrospective
Test Conditions in
Complex Model

48. WHAT’S MISSING?
ISTQB Syllabi (Presented in Prague 2011)
Certificates
(FL,AL,EL)
Learning
Objectives &
Syllabus
VISION / STRATEGY
SPECIFICATION
Business
Outcomes
REQUIREMENTS

49. WHAT’S MISSING?
Sports Club Constitution Review
Need to Change
Old Subject Matter
Experience
VISION / STRATEGY
CONSTITUTION/
RULES
Terms of
Reference
REQUIREMENTS

50. My ‘Axioms’
“Test Driven Everything”
“Business Outcome Driven Development”
“Effective Measurement and Testing Delivers
Control of Anything”
“Top Down, meets Bottom Up”

51. References
• Testing Entities Paper – John Kent
• IEEE Paper – ‘Keystone Testing Entity’ Mike
Smith & Neil Thompson
• Value Flow Scorecards – Mike Smith & Neil
Thompson
• Balanced Scorecard in a week – Mike & Pippa
Bourne
• My book on Measurement & Testing – when?!

52. What can you take away from this?
• If you are building a pure hierarchical model to
manage testing, think again
• Can I optimise my model to give better control
without it becoming unmanageable?
• Am I relying too much on my testing tools – do I
need something else?
• Can I use my analytical skills to engage with the
business better?
• I encourage you to think about these concepts,
even if you think you are in control…...

53. If you think you’ve found a good move,
look for a better one!

54. My Measures of Success for this talk?!
1.Standing Ovation?
2.Loud Applause?
3.Polite Applause?
4.Silence?
5.I won’t be let into Russia again?!!