Rapid Software Testing

Rapid Software Testing Copyright © 1995-2007, Satisfice, Inc.

V2.1.2

James Bach, Satisfice, Inc. Michael Bolton, DevelopSense
james@satisfice.com mb@developsense.com
www.satisfice.com www.developsense.com
+1 (540) 631-0600 +1 (416) 656-5160

Copyright © 1995-2007, Satisfice, Inc.
1


Acknowledgements

Some of this material was developed in collaboration with Dr.
Cem Kaner, of the Florida Institute of Technology. See
www.kaner.com and www.testingeducation.org.
Doug Hoffman (www.softwarequalitymethods.com) also
contributes to and teaches from this material.
Many of the ideas in this presentation were also inspired by or
augmented by other colleagues including Jonathan Bach, Bret
Pettichord, Brian Marick, Dave Gelperin, Elisabeth Hendrickson,
Jerry Weinberg, Noel Nyman, and Mary Alton.
Some of our exercises were introduced to us by Payson Hall,
Jerry Weinberg, Ross Collard, James Lyndsay, Dave Smith, Earl
Everett, Brian Marick, Cem Kaner and Joe McMahon.
Many ideas were improved by students who took earlier versions
of the class going back to 1996.
2

Although the value of the class is greater because of the people who helped us, the sole responsibility for the
content of this class belongs to the authors, James Bach and Michael Bolton. This class is essentially a long
editorial opinion about testing. We have no authority to say what is right. There are no authorities in our field.
We don’t peddle best practices. We don’t believe in them.

What we will do is share our experiences and challenge your mind. We hope you will challenge us back.
You’ll find the class more rewarding, we think, if you assume that our purpose is to make you stronger,
smarter and more confident as a tester. Our purpose is NOT to have you listen to what the instructor says and
believe it, but to listen and then think for yourself.

All good testers think for themselves. In a way, that’s what testing is about. We look at things differently than
everyone else so that we can find problems no one else will find.


Assumptions

You test software.
You have at least some control over the design of your tests
and some time to create new tests.
You are worried that your test process is spending too much
time and resources on things that aren’t important.
Good testing requires thinking.

You test under uncertainty and time pressure.
Your major goal is to find important problems quickly.
You want to get very good at software testing.
3

If any of these assumptions don’t fit, this class might not be for you.

We don’t make any assumptions about how experienced you are. The class can work well for novices or
experts.


Triangle

Rapid Testing

Rapid testing is a mind-set
and a skill-set of testing
focused on how to do testing
more quickly,
less expensively,
with excellent results.

This is a general testing
methodology. It adapts to
any kind of project or product.
4

Rapid testing involves considerations of skill, personal integrity, improved focus on the underlying need for
testing tasks, improved appreciation for the stakeholders of testing tasks, and knowledge of the possible
techniques and tools that could be brought to bear to improve efficiency.


Our Method of Instruction

The Class Presents Our Editorial Opinions: We do not make
appeals to authority; we speak only from our experiences, and we appeal to
your experience and intelligence.
Not All Slides Will be Discussed: There is much more material
here than we can cover in detail, so we may skip some of it. (If you want me to
go back to something that I skipped, just ask.)
We Need to Hear from You: You control what you think and do, so
we encourage your questions and challenges the lecture. (Talk to me during the
break, too.)
If You Want Specifics, Bring Specifics: We invite you to bring
real examples of testing problems and test documents to class. (I am happy to
show you how a rapid tester would work through them.)
The Exercises are the Most Important Part: We use immersive
socratic exercises that are designed to fool you if you don’t ask questions. We
usually do not provide all the information you need. Asking questions is a
fundamental testing skill!
5


Our Method of Instruction

If I call on you,
and you don’t want to be put on the spot,
say “Pass!”

6


Primary Goal of this Class

To teach you how to test a product
when you have to test it right now,
under conditions of uncertainty,
in a way that stands up to scrutiny.

7

Without scrutiny, this would be easy to do. You could test badly, but no one would ever know or care. What
makes testing hard is that someone, perhaps your employer, perhaps the users, or perhaps you yourself, will be
judging your work. Maybe this scrutiny will be indirect, such as someone unhappy with the product and
cursing your company for building it so poorly. Or perhaps it will be the most intimate form of scrutiny–
which is your feeling of pride or disappointment in your own work.

If you want your testing to pass muster, you need to work at it.


Secondary Goal of this Class

To help you practice
thinking like an expert tester.

Do that well, and you’ll be
able to handle any testing situation.

8


KEY IDEA

9


The Universal Testing Method, v1.0

“Try it and see if it works.”
Get it Choose Read the spec
Set it up where to look
See if the product
Run it See what’s matches the spec
there
Run it again, Find problems…
maybe See what’s
…especially the
not there
bad ones

Procedures Coverage Oracles 10



“Try it and see if it works.”

“Try it to learn,
sufficiently, everything that matters
about whether it can work and
how it might not work.” 11



If you don’t have an understanding and an
agreement on what is the mission of your testing,
then doing it “rapidly” would be pointless.

“everything that matters”

12


A Heuristic Test Strategy Model

Project
Environment

Tests

Quality Product
Criteria Elements

Perceived
Quality 13


A Heuristic Test Strategy Model

Project
Environment

Tests

Quality Product
Criteria Elements

Perceived
Quality 14


Calendar
“Try it to learn…how it might not work.”

1. Model the test space.
2. Determine coverage.
3. Determine oracles. There are
numerous ways
4. Determine test procedures. of doing these
5. Configure the test system. things. Patterns that
describe how to do
6. Operate the test system. this are called
7. Observe the test system. “test techniques.”
8. Evaluate the test results.
9. Report test results.
15

By “modeling”, we mean that we must build a representation in our minds of what the product must do. If our
model is incorrect or limited, then we automatically will not test what should be tested.

The first four items on the list are usually called “test design”. The latter five items are usually called “test
execution”.


KEY IDEA

16


Testers are Not Savage Beasts

Though sometimes we seem a bit “negative”
to developers or wishful marketers.
17


Testers light the way.

This is our role.
We see things for what they are.
We make informed decisions about quality possible,
because we think critically about software. 18



Quality is value to some person (who matters).

A bug is anything about the product
that threatens its value.

These definitions are designed to be inclusive.
Inclusive definitions minimize the chance that you will
inadvertently overlook an important problem.
For a video lecture, see:
http://www.satisfice.com/bbst/videos/BugAdvocacy.wmv 19

The first question about quality is always “whose opinions matter?” If someone who doesn’t matter thinks
your product is terrible, you don’t necessarily change anything. So, an important bug is something about a
product that really bugs someone important.

One implication of this is that, if we see something that we think is a bug and we’re overruled, we don’t
matter. That’s a fact: testers don’t matter. That is to say, our standards are not the standards to which the
product must adhere; we don’t get to make decisions about quality. This can be hard for some testers to accept,
but it’s the truth. We provide information to managers; they get to make the hard decisions. They get the big
bucks, and we get to sleep at night.



Testing is questioning a product
in order to evaluate it.
The “questions” consist of ordinary questions about the idea or
design of the product, or else questions implicit in the various
ways of configuring and operating the product. (Asking
questions about the product without any intent to operate it is
usually called review rather than testing.)
The product “answers” by exhibiting behavior, which the tester
observes and evaluates.
To evaluate a product is to infer from its observed behavior how
it will behave in the field, and to identify important problems in
the product.
20

Cem Kaner prefers this definition:
“Software testing is a process of empirical, technical investigation of the product under test conducted to
provide stakeholders with quality-related information.”

Kaner’s definition means the same thing as our definition in every material respect. However, we sometimes
prefer more concise wording.

One surprising implication of both our definitions is that you can test a product that doesn’t yet exist in
operable form. The questioning process that precedes what we call test execution is still part of testing if it
serves the process of test design and test execution.


The Themes of Rapid Testing

Put the tester's mind at the center of testing.
Learn to deal with complexity and ambiguity.
Learn to tell a compelling testing story.
Develop testing skills through practice, not just talk.
Use heuristics to guide and structure your process.
Be a service to the project community, not an obstacle.
Consider cost vs. value in all your testing activity.
Diversify your team and your tactics.
Dynamically manage the focus of your work.
Your context should drive your choices, both of
which evolve over time.
21


A Diversified Strategy Minimizes Tunnel Vision

le
isib ms y ou
inv ble lems ith your
pro Prob nd w
fi
can s…
e
bias

le
isib ms
inv ble
pro
22


Assess Testing Cost vs. Value.

No!
More Work & Time

ybe
ma
(Cost)

Yes!

Better Thinking & Better Testing
(Value)

http://www.satisfice.com/bbst/videos/QualityCost.mp4 23

VALUE
Coverage. The value of the elements of the product that are exercised by the test.
Power. The probability that if a problem exists in the area covered, the test will reveal it.
Integrity. The degree to which the test can be trusted, because its results are valid and consistent
across multiple executions of that test.
Technical Necessity. The degree to which the information we want to discover exists; and the
degree to which that information matters.
Business Necessity. The degree to which a test is expected to be performed by your clients.

COSTS (remember that time = money)
Opportunity Cost. Performing it may prevent you from doing other tests.
Development. The cost of getting ready to perform the test.
Execution. The cost of performing it as needed.
Transfer. The cost of getting someone else ready to run that test.
Maintenance. The cost of keeping it running.
Accounting. The cost of explaining the test, justifying it, show that it was performed.


How does Rapid Testing compare
with other kinds of testing?
When testing is turned into an Management likes to talk
elaborate set of rote tasks, about exhaustive testing, but
it becomes ponderous without they don’t want to fund it and
really being thorough. they don’t know how to do it.
More Work & Time

Exhaustive Rapid testing may
Ponderous Slow, very expensive,
not be exhaustive,
Slow, expensive, and difficult but it is thorough
(Cost)

and easier enough and quick
enough. It’s less work
Slapdash Rapid than ponderous
Faster, less expensive, testing. It might be
Much faster, cheaper, less work than
still challenging
and easier
slapdash testing.

You can always test It fulfills the mission
quickly... Better Thinking & Better Testing of testing.
But it might be poor (Value)
testing.
24

There are big differences between rapid testing and testing that is merely fast. Testing can be fast, but might
be shallow or sloppy or ill-considered or unprofessional. If so, we call it slapdash testing. Many
organizations start with slapdash testing.

It might be wonderful to test products exhaustively or completely or comprehensively. However, we can never
test a product completely. To do all the testing that we could possibly imagine would take an enormous
amount of time—and even if we had that time available, management wouldn’t want to pay for all of the
resources that it would take. Even though the value of exhaustive testing might be high, the prohibitive cost
wouldn’t justify the value.

Still, there is a perception that exhaustive testing is a good thing, so people sometimes try to achieve it by
adding work and time to the testing process. When exhaustive testing is carelessly scaled down, or slapdash
testing is made to appear rigorous by piling on mounds of badly designed documentation, it becomes
ponderous. It’s expensive, and it’s slow, but it isn’t very good testing.

Rapid testing is as exhaustive as it needs to be, and no more.


KEY IDEA

25


The Test Project Context
Where are you and what are you doing there?
Mission
Find Important Problems Advise about QA
Assess Quality Advise about Testing
Certify to Standard Advise about Quality
Fulfill Process Mandates Maximize Efficiency
Satisfy Stakeholders Minimize Cost
Assure Accountability Minimize Time

Development Requirements
Product Product Mission
Project Lifecycle Stakeholders
Project Management
Configuration Management
Test Quality Criteria
Reference Material
Defect Prevention Process
Development Team
Strategy
Logistics
Work-products

Test Team Test Lab
Expertise Test Platforms
Loading Test Tools
Cohesion Test Library
Motivation Problem Tracking System
Leadership Office Facilities
Project Integration
26


Modern Project Cycle

Project Start “Code Freeze”
Alpha Beta Beta Beta

Release to
Ongoing Requirements Definition
Manufacturing

Incremental Design Design...

Incremental Coding

Testing and Retesting

27


Rapid Testing Starts Right Away
Cycles of Work with Ongoing Status Reports
•Getting ready to test
•Designing & doing tests
•Processing test results

Do a burst of
START testing

Make sense of Focus on what
Report needs doing
your status

Compare status

STOP
against mission

28


The Test Cycle
Ship it!

Quality Typical Quality Growth

Time 29

Near the beginning of the project, rapid testing is often a process of very rapid cycles of determining
that there are major problems. Our focus is mostly on learning and developing the infrastructure for
better testing, later on.

In the middle of the project, we are find a lot of problems. We feel productive.

It’s near the end of the project that rapid testing becomes very important, because we have very little
time to discover whether the product has been seriously degraded by that most recent change. A year
long project can come down to an hour or two of retesting at the end. This is where rapid testing skill
becomes very important.


Do You Test Under
Time Pressure?

You have less time than you’d like.
Faster is better.
Your plans are often interrupted.
You are not fully in control of your time.

What is this like?

30

When you’re watching a televised professional team sport, have you noticed that there are no Gantt charts on
the sidelines? Soccer teams, baseball teams, and yacht racing teams don’t manage by Gantt charts. That’s
because they have to respond in real time to the conditions set up by the other team, and that can’t be predicted
or charted.


Excellent Rapid Technical Work
Begins with You

When the ball comes to you…
Do you know you have the ball?

Can you receive the pass? Do you know your options?

Do you know what your Is your equipment ready?
role and mission is?
Can you read the
Do you know where situation on the field?
your teammates are?
Are you aware of the
Can you let your teammates help you? criticality of the situation?

Are you ready to act, right now?
31

A good sports team has to depend on the skills of the players, rather than on rote procedures. Professional
sports teams begin with the skills of each player, and then work outwards from that. Skills get practiced
offline, but every game is part of the practice too. Strategy matters, plans matter, techniques matter, but the
focus is on the player to choose the appropriate techniques and to execute them skillfully.


…but you don’t have to be great at
everything.

Rapid test teams are about diverse talents cooperating
− We call this the elliptical team, as opposed to the team of perfect circles.
− Some important dimensions to vary:
− Technical skill
− Domain expertise
− Temperament (e.g. introvert vs. extrovert)
− Testing experience
− Project experience
− Industry experience
− Product knowledge
− Educational background
− Writing skill
− Diversity makes exploration far more powerful
− Your team is more powerful because of your unique individual contribution
32


Skill vs. Alternatives

supervision
The greater the skill of
the tester, the less
prescribed procedure,
supervision, or
favorable environment
is required.

skill

environment procedure
33

How do we deal with people who don’t have the necessary skills?

Set an Undemanding Context: We can give them a supportive environment by giving them easy problems to
solve, easily tested technology, users with low quality standards, management that doesn’t worry about quality
or liability, or developers that don’t put bugs in the software. The quality control school argues that you don’t
need to test, or that testing should be super-easy. This is how unskilled tester survive; if they’re not given hard
problems to solve, it doesn’t matter if they’re skilled. This is the approach that the quality control school takes.

Assign Rote Tasks: We can give unskilled testers comprehensive procedures to follow. We can give them
step-by-step procedures to follow, and we can turn testing into more of a clerical task than a bureaucratic one.
This is the approach that the factory school takes.

Give Personal Support: We can personally guide a tester. In the positive view, we can give a less-
experienced tester hands-on mentoring and coaching and relax the supervision as he gains experience and
skill. The pathological version of this is to supervise at a distance, or to guide the tester so thoroughly that we
end up doing all his thinking for him.

But we prefer to train testers. It’s cheaper, easier,
and the more skill they have, the less of these other
things are needed.


What might be in a tester’s head?

34


KEY IDEA

35


Sphere

Welcome to Epistemology
the study of knowledge

Epistemology is the study of how we know
what we know. The philosophy of science
and the theory of testing comes from Epistemology.

All good scientists and testers study
Epistemology.

36

Why do we talk about Epistemology? Just in case you want to become a true expert in testing. Reading about
Epistemology can be a heady, theoretical undertaking. It’s something you do in a café while sipping espresso.
Maybe that’s not for you, but if you’re the kind of person who likes to have a deep understanding of your work,
get thee to a Starbucks and start reading. Even if you read five pages and stop, it will exercise your mind in
important ways.

What specifically should you read? We would recommend The Structure of Scientific Revolutions, by Kuhn, or
the first 30 pages of Conjectures and Refutations, by Karl Popper. Or try Proofs and Refutations, by Imre
Lakatos. These are all books about how scientists come up with theories and know if they are valid.


Test Project Epistemology 101
Spot traps and avoid them.

The essential purpose served
Mission by the strategy.

What we do in this situation
Strategy to accomplish the mission.

The conditions under which
Situation at Hand
we must work.
37


Test Project Epistemology 101
Spot traps and avoid them.

Unachievable
Strategies
Mission Review assumptions.
Check for overlooked
resources.
Strategy Consider radical shortcuts.
can’t work Do something now.
Share the problem.
Make a counteroffer.
Situation won’t Promise good faith effort,
support strategy not unconditional success.
38


Scientific Skills

posing useful questions
observing what’s going on
describing what you perceive
thinking critically about what you know
recognizing and managing bias
designing hypotheses and experiments
thinking despite already knowing
analyzing someone else’s thinking
reasoning about cause and effect
treating “facts” as merely what we believe we know
as of this moment
39


Testing is about questions…
posing them and answering them

Product Tests
− What is this product? − What would constitute a diversified
− What can I control and observe? and practical test strategy?
− What should I test? − How can I improve my understanding
of how this product works?
Problems − If there were an important problem
− What quality criteria matter? here, how would I uncover it?
− What kinds of problems might I − What document to load? Which button
find in this product? to push? What number to enter?
− Is what I see a problem? Why? − How powerful is this test?
− How important is this problem? − What have I learned from this test that
Why should it be fixed? helps me perform powerful new tests?
− What just happened? How do I
Sometimes stating an examine that more closely?
assumption is a useful proxy
for asking a question.
40


..also, explanations and predictions.
what happened, what might happen, and why

1. Collect data.
2. Find several explanations that account for the data.
3. Find more data that is either consistent or
inconsistent with explanations.
4. Choose the best explanation that accounts for the
important data, or keep searching.

This is called abductive inference. To do it well,
you must consider a variety of possible explanations.

When you think you know what something is, stop and ask,
“What else could it be?”
41


Confronting Complexity:
General Systems Thinking

We must be able to simplify things without oversimplifying them.
We must know when to say:
− “it worked the same way” even when there are many differences.
− “that’s not the same bug” or “that’s a different test”, even when there are
many similarities.
− “that quality is worse, now” or “the product improved, significantly”, even
when we have made only a handful of observations.

GST is about the relationship between simplicity and
complexity, especially in dynamic and open systems.

Testers practice GST when we confront complex systems
and try to test them with relatively simple tests.
42

General Systems Thinking has been called the science of simplification (Weinberg, 1974). It is concerned
with the study of complex and open systems in general, rather than with any specific system. The aim of GST
is to help us understand the general relationships between systems and our simplified models of them, so that
we can better observe, learn about, interact with and build specific systems.

I think the basic textbook of testing is Introduction to General Systems Thinking, by Gerald M. Weinberg. This
book doesn’t specifically talk about software testing, but everything in it pertains to what testers do and how
they think.


Magic

Magic Tricks are Like Programs

Our thinking is limited
− We misunderstand probabilities Magic tricks work
for the same reasons
− We use the wrong heuristics
that bugs exist
− We lack specialized knowledge
− We forget details
− We don’t pay attention to the right things Studying magic can
help you develop
The world is hidden the imagination
− states to find better bugs.
− sequences
− processes Testing magic is
− attributes indistinguishable from
testing sufficiently
− variables
advanced technology
− identities 43


Observation vs. Inference

Observation and inference are easily confused.
Observation is direct sensory data, but on a very low level it is
guided and manipulated by inferences and heuristics.
You sense very little of what there is to sense.
You remember little of what you actually sense.
Some things you think you see in one instance may be confused
with memories of other things you saw at other times.
It’s easy to miss bugs that occur right in front of your eyes.
It’s easy to think you “saw” a thing when in fact you merely
inferred that you must have seen it.

44


Observation vs. Inference

Accept that we’re all fallible, but that we can learn to be better
observers by learning from mistakes.
Pay special attention to incidents where someone notices something
you could have noticed, but did not.
Don’t strongly commit to a belief about any important evidence you’ve
seen only once.
Whenever you describe what you experienced, notice where you’re
saying what you saw and heard, and where you are instead jumping to
a conclusion about “what was really going on.”
Where feasible, look at things in more than one way, and collect more
than one kind of information about what happened (such as repeated
testing, paired testing, loggers and log files, or video cameras).
45

Our colleague, Jeremy Kominar, is an expert amateur magician. We asked him to provide us with some
remarks on the links between software testing and magic. Here’s an edited version of what he had to say--we
can’t quote him exactly, because he asked us not to give away his secrets!

•Learning to perform magic causes the magician to be aware of not only his own perspective, but also (and
more importantly) others’ perception of the trick. This becomes invaluable for testers because it helps us to
recognize that the interests and perspectives of many stakeholders must be considered when testing software.
The trick may “look good” or “work” from your angle, but there are always other angles to cover.

•When learning to perform magic tricks, you generally want to simplify things--minimize the number of
moves and figure out how you can reach the same end result without as many steps to get there. In a testing
context this idea can be used for creating different usage scenarios—taking multiple paths and variables to
attain the same goal. (We might want to minimize the number of steps, but we might also want to vary the
paths to shake out more bugs.)

•Learning to perform magic and learning to figure out magic tricks require us to develop observational skills,
and to recognize things that deceive us. Most non-magicians fail to consider possibilities that come easily to
magicians; devices, coins, or cards can be gimmicked. Non-magicians only think about normal cards. You
need to think outside of the box to be a magician or to figure out how the magician does his work. This kind
of reasoning and deduction are key assets to being a tester. There isn’t really such a thing as magic but there
clearly is such a thing as deception. As the magician becomes more experienced and more wise to the
practice, he should be able to reverse-engineer tricks by using patterns that he already knows about
magic. Once you’ve seen one kind of trick, similar tricks aren’t so mysterious because you have heuristics
that you can use to recognize how it’s done.


Screen Wine Glass Wason

Models Link Observation and Inference

A model is an idea, activity, or object…
such as an idea in your mind, a diagram, a list of words, a spreadsheet,
a person, a toy, an equation, a demonstration, or a program

…that represents another idea, activity, or object…
such as something complex that you need to work with or study.

…whereby understanding the model may help you
understand or manipulate what it represents.
- A map helps navigate across a terrain.
- 2+2=4 is a model for adding two apples to a basket that already has two apples.
- Atmospheric models help predict where hurricanes will go.
- A fashion model helps understand how clothing would look on actual humans.
- Your beliefs about what you test are a model of what you test.
46


Thinking Like A Tester:
Seeing the Words that Aren’t There
Among other things, testers question premises.
A suppressed premise is an unstated premise that
an argument needs in order to be logical.
A suppressed premise is something that should be
there, but isn’t…
(…or is there, but it’s invisible or implicit.)
Among other things, testers bring suppressed
premises to light and then question them.
A diverse set of models can help us to see the
things that “aren’t there.”
47


Thinking Like A Tester:
Spot the missing words!

“I performed the tests. All my tests passed.
Therefore, the product works.”
“The programmer said he fixed the bug. I
can’t reproduce it anymore. Therefore it
must be fixed.”
“Microsoft Word frequently crashes while I
am using it. Therefore it’s a bad product.”
“Nobody complained about that bug after
we shipped it. Therefore it wasn’t
important.”
48


Heuristics
bring useful structure to problem-solving skill.

adjective:
“serving to discover.”
noun:
“a fallible method for solving a problem or
making a decision.”

“Heuristic reasoning is not regarded as final and strict
but as provisional and plausible only, whose purpose
is to discover the solution to the present problem.”
- George Polya, How to Solve It
49


Types of Heuristics

Guideword Heuristics: Words or labels that help you access
the full spectrum of your knowledge and experience as you
analyze something.
Trigger Heuristics: Ideas associated with an event or condition
that help you recognize when it may be time to take an action or
think a particular way. Like an alarm clock for your mind.
Subtitle Heuristics: Help you reframe an idea so you can see
alternatives and bring out assumptions during a conversation.
Heuristic Model: A representation of an idea, object, or system
that helps you explore, understand, or control it.
Heuristic Procedure or Rule: A plan of action that may help
solve a class of problems.
50


Some Everyday Heuristics

It’s dangerous to drink and drive.
A bird in hand is worth two in the bush.
Nothing ventured, nothing gained.
Sometimes people stash their passwords near their
computers. Try looking there.
Stores are open later during the Holidays.
If your computer is behaving strangely, try
rebooting. If it’s very strange, reinstall Windows.
If it’s a genuinely important task, your boss will
follow-up, otherwise, you can ignore it.
51


How Heuristics Differ from
Other Procedures or Methods

A heuristic is not an edict. Heuristics require
guidance and control of skilled practitioner.
Heuristics are context-dependent.
Heuristics may be useful even when they
contradict each other– especially when they do!
Heuristics can substitute for complete and
rigorous analysis.

52


KEY IDEA

http://www.satisfice.com/bbst/videos/BBSTORACLES.mp4 53


Oracles

An oracle is the principle or mechanism
by which you recognize a problem.

“..it works”

“...it appeared at least once to meet some
requirement to some degree.”

54

Jerry Weinberg has suggested that “it works” may mean “We haven't tried very hard to make it fail, and we
haven't been running it very long or under very diverse conditions, but so far we haven't seen any failures,
though we haven't been looking too closely, either.” In this pessimistic view, you have to be on guard for
people who say it works without checking even once to see if it could work.


Learn About Oracles

Without an oracle you cannot recognize a problem

If you think you see a problem,
you must be using an oracle.

55


Consistency (“this agrees with that”)
an important theme in oracles
History: The present version of the system is consistent with past versions of it.
Image: The system is consistent with an image that the organization wants to project.
Comparable Products: The system is consistent with comparable systems.
Claims: The system is consistent with what important people say it’s supposed to be.
Users’ Expectations: The system is consistent with what users want.
Product: Each element of the system is consistent with comparable elements in the
same system.
Purpose: The system is consistent with its purposes, both explicit and implicit.
Statutes: The system is consistent with applicable laws.
Familiarity: The system is not consistent with the pattern of any familiar problem.

Consistency heuristics rely on the quality of your
models of the product and its context. 56

Note that we’re saying “system” here, instead of “product”. A product could behave perfectly as a part of
some system, but when something else within the system changes, that change could undermine the product’s
usefulness or perceived quality.

Seeing the product as a component or participant of some larger system opens our eyes to a greater range of
potential problems.


Word Save File Notepad Save File Diskmapper

All Oracles Are Heuristic

We often do not have oracles that establish a definite correct or
incorrect result, in advance.
That’s why we use abductive inference.
No single oracle can tell us whether a program (or a feature) is
working correctly at all times and in all circumstances.
That’s why we use a variety of oracles.
Any program that looks like it’s working, to you, may in fact be
failing in some way that happens to fool all of your oracles.
That’s why we proceed with humility and critical thinking.
You (the tester) can’t know the deep truth about any result.
That’s why we report whatever seems likely to be a bug.
57


Coping With Difficult Oracle Problems

Ignore the Problem
− Ask “so what?” Maybe the value of the information doesn’t justify the cost.
Simplify the Problem
− Ask for testability. It usually doesn’t happen by accident.
− Built-in oracle. Internal error detection and handling.
− Lower the standards. You may be using an unreasonable standard of correctness.
Shift the Problem
− Parallel testing. Compare with another instance of a comparable algorithm.
− Live oracle. Find an expert who can tell if the output is correct.
− Reverse the function. (e.g. 2 x 2 = 4, then 4/2 = 2)
Divide and Conquer the Problem
− Spot check. Perform a detailed inspection on one instance out of a set of outputs.
− Verify by parts. Check specific, risky aspects of the output, instead of everything.
− Blink test. Compare or review overwhelming batches of data for patterns that stand out.
− Easy input. Use input for which the output is easy to analyze.
− Easy output. Some output may be obviously wrong, regardless of input.
− Unit test first. Gain confidence in the pieces that make the whole.
− Test incrementally. Gain confidence by testing over a period of time.
58


“Easy Input”

Fixed Markers. Use distinctive fixed input patterns that are easy to
spot in the output.
Statistical Markers. Use populations of data that have
distinguishable statistical properties.
Self-Referential Data. Use data that embeds metadata about itself.
(e.g. counterstrings)
Easy Input Regions. For specific inputs, the correct output may be
easy to calculate.
Outrageous Values. For some inputs, we expect error handling.
Idempotent Input. Try a case where the output will be the same as
the input.
Match. Do the “same thing” twice and look for a match.
Progressive Mismatch. Do progressively differing things over time
and account for each difference. (code-breaking technique) 59

Fixed Markers. Some programmers use the hex values 0xDEADBEEF as a flag to indicate uninitialized data.
Statistical Markers. If you create input that has a specific statistical property, then you may be able to detect
missing or corrupted data by examining the statistical properties of the output.
Self-referential data. Counterstrings are strings that identify their lengths. *3*5*7*9*12*15* is an example
of a counterstring. (Satisfice’s PERLCLIP tool creates counterstrings of arbitrary lengths.) Another example is
using JAMES_FIRSTNAME and BACH_LASTNAME as sample data in the first name and last name fields of
a database.
Easy Input Regions. A number times one is always itself; a number plus zero is always itself; a multiple of
ten always ends in 0, and so on.
Outrageous values. Examples include letters in fields that expect only numbers, extremely long strings in
fields that expect short ones, huge numbers where modest ones are expected, or zero or null values where
some positive value is required. Each example should trigger some kind of error handling.
Idempotent input. Take some data and process it in some way, such that the output from the first run can be
used as input for subsequent runs but shouldn’t go through additional changes. Examples include spell-
checking (after all the corrections have been accepted for the first run, no further changes should be suggested)
or code formatters (after the code has been formatted once, the same code should pass through the process
unchanged).
Match. Given the same input data and the same function on that data, we would typically expect the output to
be the same. This approach is sometimes used to check various kinds of encoding or checksums.
Progressive mismatch. Check for consistency in a function by varying the input in some limited or controlled
way, and observe the effect on the output. This is an exploratory approach, sometimes used as a code-breaking
technique.


Quality Criteria

Capability Compatibility
Reliability Supportability
Usability Testability
Security Maintainability
Scalability Portability
Performance Localizability
Installability

Many test approaches focus on Capability
(functionality) and underemphasize the other criteria 60

Try pronouncing CRUSSPIC STMPL; sometimes we think of it as the name of an Eastern European hockey
player, but however you think of it, it makes this list easier to remember. The point is to be able to cycle
through the list at any time to ask a new question about something that we might value in the product, or
something that might represent a vulnerability if it were missing or broken.


KEY IDEA

61


Coverage

Product coverage is the proportion of the
product that has been tested.

Structure
Function Performance Maintainability
Capability Portability
Data Installability
Reliability Localizability
Compatibility
Platform Usability
Supportability
Operations Security
Testability
Scalability
Time

62


Product Elements:
Structural Coverage

input output
Test what it’s
made of.
platform

Print testing example
− Files associated with printing
− Code modules that implement printing
− Code statements inside the modules
− Code branches inside the modules


Product Elements:
Functional Coverage

input functions
functions output
Test what
it does.
platform

− Print, page setup and print preview
− Print range, print copies, zoom
− Print all, current page, or specific range


Product Elements:
Data Coverage

functions
input & output
Test what structure
it does it to.
platform

− Types of documents
− Items in documents, size and structure of documents
− Data about how to print (e.g. zoom factor, no. of copies)


Product Elements:
Platform Coverage

functions
input & output
Test what it structure
depends upon.
platform

− Printers, spoolers, network behavior
− Computers
− Operating systems
− Printer drivers


Product Elements:
Operations Coverage

input output
Test how
it’s used.
platform

− Use defaults
− Use realistic environments
− Use realistic scenarios
− Use complex flows


Profiler

Product Elements:
Time Coverage

Test how
input output
it’s affected
by time.
platform

− Try different network or port speeds
− Print one document right after another, or after long intervals
− Try time-related constraints--spooling, buffering, or timeouts
− Try printing hourly, daily, month-end, and year-end reports
− Try printing from two workstations at the same time
− Try printing again, later.


Sometimes your coverage is disputed…

“No user would do that.”

“No user I can think of, who I like,
would do that on purpose.”
Who aren’t you thinking of?
Who don’t you like who might really use this product?
What might good users do by accident? 69


Sometimes it’s really hard to cover…
Ask for testability!

Controllability Scriptable
Observability Interface!
Availability
Log files!
Simplicity
Stability
Information

70


KEY IDEA

71


Exploratory Testing Is…

an approach to software testing…
(applicable to any test technique)

that emphasizes the personal freedom and
responsibility of each tester to continually
optimize the value of his work…
(optimize how?)

by treating learning, test design and test
execution as mutually supportive activities that
run in parallel throughout the project.
72


IP Address

ET is a Structured Process

Exploratory testing, as I teach it, is a structured process
conducted by a skilled tester, or by lesser skilled testers or users
working under supervision.
The structure of ET comes from many sources:
− Test design heuristics
− Chartering
− Time boxing In other words,
− Perceived product risks it’s not “random”,
− The nature of specific tests but systematic.
− The structure of the product being tested
− The process of learning the product
− Development activities
− Constraints and resources afforded by the project
− The skills, talents, and interests of the tester
− The overall mission of testing 73


ET is a Structured Process

In excellent exploratory testing, one structure
tends to dominate all the others:

Exploratory testers construct a compelling
story of their testing. It is this story that
gives ET a backbone.

74


To test is to compose, edit, narrate,
and justify a story.

You must tell a story about the product…
…about how it failed, and how it might fail...
…in ways that matter to your various clients.

But also tell a story about testing…
…how you configured, operated and observed it…
…about what you haven’t tested, yet…
…or won’t test, at all…
…and about why what you did was good enough.
75


The Process of Test Design

Testing Story
- Test Plan/Report
- Work Products
- Status

Informs Produces Question

Knowledge Analysis Experiment
- Product Story - Risk Test
- Technical Knowledge - Coverage Procedure
- Domain Knowledge - Oracles
- Resources/Constraints - Configure
- General Knowledge
- Value/Cost - Operate
- Bugs Investigate - Observe
bugs - Evaluate

Answer
Produces Informs

76


How to Start?
Pick a Useful, Fun, or Easy Starting Point.
Testing Story

What do I need to produce for my client?
What has already been done?
Informs Produces

Knowledge Analysis Experiment

What do I know? What kind of testing Do I have a product?
What do I need to know? am I good at? What can I do with
What obstacles threaten the product?
my success?
Produces Informs

77


Test Design
Testing to Learn vs. Testing to Search
Testing (primarily) to Learn
− Forming a mental model of the product.
− Learning what the product is supposed to do.
− Inventorying the product elements you may want to test.
− Looking at consistency relationships and trying various oracles.
− Generating test data.
− Considering testability and trying potentially useful tools.
− Experimenting with different ways of testing it.
− Reporting bugs that you find.
Testing (primarily) to Search
− Using your detailed product knowledge, and any relevant tools, to
systematically exercise the product in every important way.
− Using careful observation and good oracles to detect important bugs.
− Modifying and diversifying your tests to make them more powerful.
− Reporting bugs that you find.
78


Test Design
Testing to Learn vs. Testing to Search

Compare these Situations:

L S − Starting a new project.
L S − Seeing a new feature for the first time.
L S − Testing a product deeply to reveal important bugs.
L S − Investigating a particular bug.
L
S − Re-testing a product after a change.
S − Repeated execution of detailed procedural test scripts.

79


Lyndsay Machine

High Learning ET:
Explore These Things
Composition
− Affordances: Interfaces to the product.
− Dimensions & Variables: Product space and what changes within it.
− Relationships & Interactions: functions that cooperate or interfere.
− Navigation: Where things are and how to get to them.
Conformance
− Benefits: What the product is good for-- when it has no bugs in it.
− Consistencies: Fulfillment of logical, factual, and cultural expectations.
− Oracles: Specific mechanisms or principles by which you can spot bugs.
− Bugs and Risks: Specific problems and potential problems that matter.
Context (of the Product)
− History: Where the product has been, and how it came to be.
− Operations: Its users and the conditions under which it will be used.
Conditions (of Testing)
− Attitudes: What your clients care about and what they want from you.
− Complexities & Challenges: Discover the hardest things to test.
− Resources: Discover tools and information that might help you test. 80


Contrasting Approaches

In scripted testing, tests are first
designed and recorded. Then they Test
may be executed at some later time Scripts
or by a different tester.

In exploratory testing, tests are Test Ideas
designed and executed at the
same time, and they are not Product
necessarily recorded, but may be.
81



Scripted testing is about Test
controlling test execution. Scripts

Exploratory testing is about Test Ideas
improving test design.
Product
82



Scripted testing is like
making a prepared speech. Test
It is guided by pre-conceived ideas. Scripts

Exploratory testing is like Test Ideas
having a conversation.
It is self-guided.
Product
83


Mixing Scripting and Exploration

freestyle exploratory
pure scripted fragmentary
vague scripts test cases charters roles

When I say “exploratory testing” and don’t qualify it, I mean anything
on the exploratory side of this continuum.

84


Blending Scripted & Exploratory

Generic scripts: specify general test procedures and apply them to different
parts of a test coverage outline.
Vague scripts: specify a test step-by-step, but leave out any detail that does not
absolutely need to be pre-specified.
Improvisation: have scripts, but encourage deviation from them, too.
Fragmentary cases: specify tests as single sentences or phrases.
Test Coverage Outline: use outline of product elements and have tester
construct tests from it on the fly.
Risk Catalog: specify types of problems to look for, then construct tests on the
fly to find each one.
Exploratory Charters: specify 90 minutes of testing in two sentences or less.
Roles: Give each tester a standing role to test a certain part of the product. Leave
the rest up to them.
Heuristics: Train exploratory testers to use standardized test design heuristics.
SBTM: Consider using Session-Based Test Management, a formalized method of
exploratory test management. (http://www.satisfice.com/sbtm).
85


Test Design and Execution

Achieve excellent test design by
exploring different test designs
while actually testing Product

Test
Ideas

Guide testers with personal supervision and
concise documentation of test ideas. Meanwhile,
Product train them so that they can guide themselves and
or spec be accountable for increasingly challenging work.
86


Allow some disposable time
Self-management is good!

How often do you account for your progress?

If you have any autonomy at all, you can risk
investing some time in
− learning
− thinking
− refining approaches
− better tests

87


Allow some disposable time

If it turns out that you’ve made a bad
investment…oh well
☺ If it turns out that you’ve made a good
investment, you might have
− learned something about the product
− invented a more powerful test
− found a bug
− done a better job
− surprised and impressed your manager

88


“Plunge in and Quit” Heuristic

Whenever you are called upon to test
something very complex or frightening, plunge in!
After a little while, if you are very confused
or find yourself stuck, quit!

This benefits from disposable time– that part of your
work not scrutinized in detail.
Plunge in and quit means you can start something
without committing to finish it successfully, and
therefore you don’t need a plan.
Several cycles of this is a great way to discover a plan. 89


Exploratory Branching:
Distractions are good!

New test
idea
New test
idea

New test
New test ideas occur idea

continually during an
ET session.

90


“Long Leash” Heuristic

Let yourself be distracted…
‘cause you never know
what you’ll find…

but periodically take stock
of your status against your mission

91

Rapid Software Testing

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Rapid Software Testing

Similar to Rapid Software Testing (20)

Rapid Software Testing