DataArt IT school in Odessa (Autumn/Winter 2014)

1. Test case and BUG report
Andrey Oleynik
2014

2. Agenda
1. Test Case.
2. Examples.
3. BDD.
4. Examples.
5. Defects
6. BUG Report
7. Examples
8. Good BUG
9. Additional links
2

3. About myself
Andrey Oleynik
• About 7 years of
work in Quality
Assurance area
• QA Automation
Engineer
• PhD, Theoretical
physics
3

4. What is a Test Case?
• A set of input values, execution preconditions,
expected results and execution postconditions,
developed for a particular objective or test condition,
such as to exercise a particular program path or to
verify compliance with a specific requirement. [IEEE]
• Documentation specifying inputs, predicted results, and
a set of execution conditions for a test item. [ANSI]
• A test case is a bunch of steps that lead to consider one
"item" as successfully working.
4

5. Why do we write
test cases?
• The basic objective of writing test cases
is to validate the testing coverage of the
application
• Usually QA engineers follow the test
cases standards
• So writing test cases brings some sort
of standardization and minimizes the
ad-hoc approach in testing
5

6. What information would
the test manager want
out of test case
document/s?
• Which features have been tested/ will be tested
eventually?
• How many user scenarios/ use cases have been
executed?
• How many features are stable?
• Which features need more work?
• Are sufficient input combinations exercised?
• Is software ready to ship? Is testing enough?
• What is the quality of the application?
6

7. Test Case design
• Specification based testing;
• Input Domain testing;
• Risk based testing;
• Scenario testing.
7

8. Test Case Fields
(the most important)
• Test Case id
• Title
• Priority
• Description
• Steps
• Expected result
• Feature/Area 8

9. Test Case fields
• Test case status
(Idea/Design/Ready/Update/etc.)
• Product version
• Responsible person
• Preparation notes
• Notes to clean the system
• Attachments
• Examples
• Linked feature/requirement/bug
• ...
9

10. HP ALM example
10

11. ALM example
(Test Case steps)
11

12. Execution example
(ALM)
12

13. Example
Title: Open file.
Description: a file can be opened in the Application.
1. Launch the Application.
2. Select "File" menu.
File menu pulls down.
3. Choose "Open“.
"Open" dialog box appears.
4. Select a file to open.
5. Click OK.
Expected result: File should be opened.
13

14. Important:
1. Template
2. Descriptive and specific
3. Reusable
4. Atomic
5. Positive and negative
6. Refactor
7. Test data
8. Setup and tear down
9. Test Case review
10. Screenshots
14

15. Test Suite
• A test suite is a set of related tests, usually
pertaining to a group of features or software
component and usually defined over the same
database. Suites are combined into groups.
[Beizer]
15

16. Test management tools
• Gemini
• HP ALM (QC)
• IBM Rational Quality Manager
• Meliora Testlab
• Seapine TestTrack
• TestLink
• Google Docs !
• …
16

17. Q & A
17

18. Testing pyramid
18

19. TDD
Test-driven development is a software
development methodology which
essentially states that for each unit of
software, a software developer must:
• define a test set for the unit first;
• then implement the unit;
• finally verify that the implementation of
the unit makes the tests succeed.
19

20. TDD
• Java (Junit, TestNG);
• C (Check, CUnit);
• C++ (Boost, CppUnit, CppUnitLite, Google
Test);
• C# (xUnit, NUnit, MSTest);
• Ruby (Test::Unit, Shoulda, RSpec);
• Python (pyUnit, py.test, doctest);
• …
20

21. BDD
• Behavior-driven development is a specialized version
of test-driven development which focuses on
behavioral specification of software units.
• Behavior-driven development specifies that tests of
any unit of software should be specified in terms of
the desired behavior of the unit.
21

22. BDD
22
Title
The story should have a clear, explicit title.
Narrative
A short, introductory section that specifies
•who
•which effect the stakeholder wants the story to have
•what business value the stakeholder will derive from
this effect
Acceptance criteria or scenarios
a description of each specific case of the narrative.
•It starts by specifying the initial condition
•It then states which event triggers the start of the
scenario.
•Finally, it states the expected outcome, in one or more
clauses.

23. BDD example
23
Feature: New account
In order to have new ready to use account
As a user
I want to get an account with balance $0
Scenario: Create new account
Given I have no account
When I create an account
Then I should have an account with balance $0

24. BDD
• Jbehave (Java)
• Thucidydes
• Rspec (Ruby)
• Cucumber
• SpecFlow (C#)
• …
24

25. Q & A
25

26. What is a BUG
(defect)?
• A flaw in a component or system
that can cause the component or
system to fail to perform its
required function, e.g. an
incorrect statement or data
definition. A defect, if
encountered during execution,
may cause a failure of the
component or system.
26

27. BUG
(Harvard Mark II, 1947)
27

28. Another BUG definition
• A software bug is an error,
flaw, failure, or fault in a
computer program or system
that produces an incorrect or
unexpected result, or causes
it to behave in unintended
ways.[Wikipedia]
28

29. BUG names
• Mistake
• Anomaly
• Fault
• Failure
• Error
• Exception
• Crash
• Bug
• Defect
• Incident
• Side Effect
• Change request
29

30. Types of BUGs
• UI
• Functional issues
• Programming logic
• Installation
• Memory allocation
• System resources
• Documentation
• Requirements
• Localization (L10N)
• Internalization (I18N)
• Performance
• Security
• …
30

31. How to find a bug?
• Understand the whole
application in depth
• Prepare good test cases and test
data before testing
• Execute tests with different test
environment
• Compare actual and expected
results
31

32. What to do next?
• Do monkey testing
• Use test data pattern from
previous versions/projects
• Check standard scenarios from
other products
• Try to break the application!
32

33. BUG report
Defect/BUG report: A document reporting on
any flaw in a component or system that can
cause the component or system to fail to
perform its required function. [IEEE]
33
“The point of writing problem
report(bug report) is to get bugs
fixed” [Cem Kaner].

34. Defect Fields
(the most important)
• Bug ID
• Summary
• Description/Steps to Reproduce
• Expected/Actual result
• Status
• Severity
• Priority
• Detected by
• Assigned to
• Module/Area
34

35. Defect Fields
• Environment
• Reproducible
• Fixed in Version
• Detected Date
• Modified/Last Updated
• Detected in Version/Build Version
• Comments
• Attachments
• Linked testcases/requirements/defects
35

36. Defect Life Cycle
36

37. Defect Resolution
37

38. Defect Resolution
• Fixed
• Won't Fix
• Not Reproducible
• Duplicate
• By Design
• External / 3d party
38

39. Defect Severity
1 - Urgent
• (Critical)
• Data loss or corruption. Product is down. Customer operations are
impacted very seriously. There is no any workaround.
• For documentation: copyright or offensive content issues,
documentation steps lead to data loss or corruption.
2- High
• (Serious)
• System / feature is still usable to a high degree but has a serious
issue. Defect has serious customer business impact.
• Some data is at risk. No obvious workaround.
• For documentation: core functionality is not described.
39

40. Defect Severity
3 - Medium
• Product operational
• Application customer data is not at risk.
• Obvious/easy workaround available.
• Non informative error reporting.
• For documentation: Conceptual information is incorrect or
incomplete.
4 - Low
• It is more of a cosmetic issue. No serious impedance to
system functionality is noted. Annoyances.
• For documentation: typography, formatting.
40

41. Priority
0. Show-stopper - this needs to be fixed right now,
everything else can wait, the build cannot be
released with this defect
1. Urgent - needs to be fixed before any other high,
medium or low defect should be fixed
2. High - should be fixed as early as possible
3. Medium - should take precedence over low
priority defects
4. Low - fixing can be deferred until all other priority
defects are fixed
41

42. JIRA example
42

43. ALM/QC example
43

44. BUG example
Title/Summary: Application crash on clicking the Browse button while file open.
Area: File operations Build Number: x.x.x
Severity: HIGH Priority: HIGH
Status: New/Open/Active
Environment: Windows 2008
Steps To Reproduce:
1) Launch the application
2) Open menu File -> Open.
3) Click on ‘Browse’ button
Actual result: Application crash.
Expected result: On clicking Browse button file browse dialogue should be
opened.
Notes: the screenshot and the logs are attached. 44

45. Good BUG
1. Short and precise title
2. Concise but complete description: Steps to
reproduce, consequences of the bug, and
suggested behavior
3. Good attachments
4. Complete definition of the categorizing fields
5. Correct Severity & Priority
6. Follow-up and comment
45

46. Good BUG
7. Include the Results you Expected
8. Include the Results you Actually Observed
9. Explain the Impact on the Customer
10. Avoid speculation
11. Be careful of the tone of your report
12. Avoid duplication - search first (!)
46

47. Collaboration
• QA engineers and developers are not in fight
but are partners
• The main thing is product and its quality
47

48. BUG Tracking Systems
• Gemini
• HP ALM (QC)
• IBM Rational Quality Manager
• JIRA
• Seapine TestTrack
• Bugzilla
• Google Docs !
• …
48

49. More details
49
• Software testing Help:
http://www.softwaretestinghelp.com/
• Про Тестинг: http://www.protesting.ru/
• http://software-testing.ru
• What is severity and priority in software testing?
http://istqbexamcertification.com/what-is-severity-
and-priority-in-software-testing/
• IEEE Std 1044-1993 IEEE Standard Classification for
Software Anomalies
http://standards.ieee.org/findstds/standard/1044-
1993.html
• ISTQB® Glossary
http://www.istqb.org/downloads/viewcategory/20
.html

50. More details
50
• Сэм Канер, Джек Фолк Тестирование программного
обеспечения
• Роман Савин Тестирование Дот Ком, или Пособие по
жестокому обращению с багами в интернет-стартапах
• Луиза Тамре Введение в тестирование программного
обеспечения

51. More details
51
• Introducing BDD by Dan North
http://dannorth.net/introducing-bdd/
• Портал об автоматизации тестирования ПО:
http://automated-testing.info/
• Конференции sqadays: http://www.sqadays.com
• Software testing Help: http://www.softwaretestinghelp.com/
• Про Тестинг: http://www.protesting.ru/
• http://software-testing.ru

52. More details
52
• Э.Дастин, Дж. Рэшка, Дж.Пол Автоматизированное тестирование
программного обеспечения. Внедрение, управление и эксплуатация
• Matt Wynne and Aslak Hellesoy The Cucumber Book: Behaviour-Driven
Development for Testers and Developers
• Криспин, Грегори Гибкое тестирование

53. Q & A
53

54. The End
• Thank you!
54