Slides for Keynote: Experiences With Exploratory Testing Delivered at TestIstanbul May 2013 by Alan Richardson

1. Experiences with
Exploratory Testing
Alan Richardson
@eviltester
www.eviltester.com
www.compendiumdev.co.uk
www.seleniumsimplified.com

2. Future of Testing: New
Techniques and Methodologies
●
Exploratory Testing is not new,
●
but people still don't do it.
●
The future will be the same, as now,
●
unless we take responsibility for it.
●
Learn from experience,
●
base experiments on observation.

3. Some things I've learned by doing
and teaching Exploratory Testing
●
Adopt a personal approach
●
Exploration & experiment is natural
●
Environment can kill exploratory testing
●
Evidence based documentation
●
Use Automation in your exploration

4. What does "Exploratory
Testing" mean?
"An Interactive Process of
Simultaneous learning, test
design, and test execution"
James Bach
http://www.satisfice.com/glossary.shtml

5. "If you master the source, but not
the teachings, whenever you open
your mouth you will speak at
random; if you master the
teachings but not the source, you
will be like a one-eyed dragon."
The story of Chinese Zen,
by Nan Huai-Chin

6. Adopt a personal
approach to Testing
Take responsibility for your test
approach:
●
What does it mean to me?
●
How do I do it?
●
Why did I start?
●
Why do I do it?

7. What do I mean by
exploratory testing?
●
Continual
Evaluation &
Re-evaluation
●
Observation &
Evidence
●
Tool Augmented
●
Low Overhead
& Fast
●
Skill Based
●
Team Based
●
Intent
Characterized by:

8. Low Overhead & Fast
●
Fast paced environments, becoming norm
●
Success based on outcome, not compliance
●
Maximize the time spend "doing"
Techniques:
●
Think through risks and target those. System
risks, business risks.
●
Chunk testing into 'charters' and 'sessions'.
Log the testing against those. Raise defects.

9. Skill Based
Rather than just relying on 'Techniques'...
●
Assume the tester knows the techniques
and applies them on an adhoc skill based
basis
●
Train people where risks and bugs exist and
how to look for them
●
Train people to be adaptable
●
Trust people, give them responsibility

10. Team Based
●
Use the whole team to review everyone
else's work
●
Write logs that others can read
●
Review coverage of others testing and
suggest new scope
●
Pair to come up with new ideas
●
Debrief rather than report
●
The Manager tests as well

11. Continual Evaluation &
Re-evaluation
●
Start with minimal scope
●
Test to learn what you need to test
●
Write down what you know and don't know
●
Continually revisit your approach and
testing scope
●
Revisit your use of tools
●
Question your effectiveness
●
Model the system and the process

12. Intent
●
Know what you are testing for.
●
Decide on what to test next.
●
What risk are you concerned about?
●
What requirement are you
investigating?
●
What data are you covering?
●
Don't do it because you planned to.
●
Do it because it is important now.

13. Observation & Evidence
●
Focus / Defocus
●
Go Meta
●
Go Deeper
●
Peripheral Observation
●
Relate to existing evidence
●
Produce Evidence
●
screenshots, system logs,
●
recordings, manual logs

14. Tool Augmented
●
Tools augment our testing to help us
test better
●
Tools should never drive or dictate our
process
●
Exploratory Testing has heavy use of
tools for:
Observation, Data Creation,
Logging, State Manipulation

15. My Exploratory Process
●
Modelling
●
Observing
●
Reflective
●
Intentional
●
Manipulative

16. Why should you do
exploratory testing?
•
Because you liked the definition?
•
Because you liked the characteristics?
•
Because you want to test more effectively?
Personal Approach
I don't know why you should do it. I only know why
I do it, and why I started doing it.

17. Why did I start doing
exploratory testing?
"There is a risk that the test team will
waste a lot of time and money by
creating scripts that are wrong and
will never be run"
•
Frustration Mainly
•
Heavily Scripted environment which wasn't
working well

18. In order to be effective we had
to Subvert the process
Step
#
Description Expected Result
1 I've only
added this
because the
tool made
me
I'll test the conditions
in the case and write
down my
observations

19. Why do I keep doing
Exploratory Testing?
Because:
●
It works
●
I get better at it, the more I do it
●
I can train people to do it
●
It adapts to different environments,
and processes

20. ● Stop Following
● Do and Learn
● Change What you do
Take Responsibility to succeed

21. Develop Ability & enable
Response Ability
Response Able Adapt Able
Adapt Ability
Account Ability
Account Able

22. What stops
Exploratory Testing?
●
Environment
●
Personal Fear
●
Existing Learning

23. Automation &
Exploratory Testing
●
Tool Augmented
●
Exploratory Automated Testing
●
Models & Checks
●
Automation leads to Exploration

24. An example: Random
Coverage of Defined Scope
●
New Registration Process
●
Manual Testing kept finding problems,
●
Every app change introduced new errors
●
Randomize Coverage of a Model
●
Model registration as a process of equivalent steps
●
Fill Name, Fill Password, Fill PostCode etc.
●
Model data as a set of templates to randomly fill
●
{a-z,A-Z, }*3-25, {a-z,A-Z,0-9,!"£$%^&*()}*6-18
●
Leave to run, Automated Checks flag issues
●
When run is clean, perform exploratory testing

25. When Automation Leads
to exploration
●
Sometimes you want to get
somewhere known fast,
●
so that you can explore unknowns
●
Automate the 'getting to'
●
Randomise to maximise chance that
this shows a problem
●
When there, engage brain and test

26. Explore and Experiment
with your Testing
●
The Future of Testing requires us to Unlearn.
●
Change our existing beliefs based on evidence.
●
Make it easy to experiment with new
approaches.
●
Evaluate Qualitatively as well as Quantitatively.
●
Take responsibility, own your test approach.

27. 27
Blogs and Web Sites
● SeleniumSimplified.com
● EvilTester.com
● Testing Papers and Tools
– CompendiumDev.co.uk
@eviltester
Online Training Courses
● Technical Web Testing 101
– Unow.be/at/udemy101
● Intro to Selenium
– Unow.be/at/udemystart
● Selenium 2 WebDriver API
– Unow.be/at/udemyapi
Videos
youtube.com/user/EviltesterVideos
Books
Selenium Simplified
Unow.be/rc/selsimp
Alan Richardson is an Independent Test Consultant
based in the UK. He offers training and consultancy
in Selenium WebDriver, exploratory and technical
web testing.
– uk.linkedin.com/in/eviltester
Contact Alan for training and consultancy
tailored to your needs:
alan@compendiumdev.co.uk