How we use Bugzilla, Testopia and Jenkins to build automated testing environment

1. Automated Testing Environment
by Bugzilla, Testopia and Jenkins
Chin Pin Chang / Chia Hung Kao
04/07/2014
1

2. Outline
• Bugzilla
• Testopia
• Jenkins
• Automated Testing Environment (Hypervisor
Testing as an example)
– Automated Testing Process
– Example
• References
2

3. Bugzilla Introduction
• Bugzilla is a Defect Tracking System and it is
server software designed to help you manage
software development
3

4. Bugzilla Features
• Excellent security to protect confidentiality
• Optimized database structure for increased
performance and scalability
• Integrated email capabilities
• Comprehensive permissions system
• Editable user profiles and comprehensive email
preferences
• Proven under fire as Mozilla's bug tracking
system
4

5. Bugzilla Requirements
• Hardware
– CPU : 3 GHz
– RAM : 4 GB or more recommended
– Hard Disk : 50 GB of free space is large enough
• Operating System : Including Windows, Linux, and
Mac OS X
• Software
– Perl : 5.12
– Database Server : Bugzilla supports MySQL 5.5,
PostgreSQL 8.3, Oracle and SQLite
– Web Server : Apache 2.2
5

6. MySQL Installation
Step 1. Update
6
Step 2. Install MySQL
Step 3. Set the password

7. MySQL Installation (cont.)
Step 4. Login MySQL
– user
– password
7

8. MySQL Configuration
Step 5. Create an user for Bugzilla
8
Step 6. List users

9. MySQL Configuration (cont.)
Step 7. Create database for Bugzilla
9

10. Apache Configuration
10
Step 1. Install Apache2
Step 2. Modify Apache2 configuration file

11. Bugzilla Installation
Step 1. Download Bugzilla : http://www.bugzilla.org/
11
Step 2. Decompress the file to /var/www/
Step 3. Modify folder name and switch to
/var/www/bugzilla

12. Bugzilla Installation (cont.)
Step 4. Execute checksetup.pl
12

13. Bugzilla Installation (cont.)
13
Step 5. Install module

14. Bugzilla Installation (cont.)
14
Step 6. Re-run checksetup.pl
Step 7. Need to modify localconfig

15. Bugzilla Installation (cont.)
15
Step 8. Modify Bugzilla configuration
Step 9. Modify permission

16. Bugzilla Installation (cont.)
16
Step 10. Re-run checksetup.pl
Step 11. Set up user information

17. Bugzilla Installation (cont.)
17
Step 12. Modify Apache configuration and restart
Apache2

18. Bugzilla Installation (cont.)
18
Step 13. Installation complete

19. Testopia Introduction
• Testopia is a test case management extension for
Bugzilla
• Testopia Requirements
– Bugzilla
– MySQL or PostgreSQL
– Mozilla compatible browser
– Additional Perl Modules
• Testopia Architecture
– Test Plans
– Test Cases
– Test Runs
19

20. Relationship between Bugzilla and Testopia
20
• Since Testopia is an extension to Bugzilla, it relies on many of the same
objects that Bugzilla uses to track bugs
• Testopia is comprised of several objects that are interdependent, so they
make managing the testing procedure possible

21. Relationship between Bugzilla and Testopia (cont.)
21
• Component
– A Bugzilla component
– An attribute of a product

22. Relationship between Bugzilla and Testopia (cont.)
22
• Product
– The product name belong to Bugzilla

23. 23
• Milestone
– A Bugzilla object
– Testopia builds are associated with milestones
Relationship between Bugzilla and Testopia (cont.)

24. 24
• Test Plan
– At the top of the Testopia hierarchy are Test Plans. Before you can do anything else in
Testopia, you need a Test Plan
– Test plan will serve as the storage point for all related Test Cases and Test Runs and it will
act as the Dashboard for your testing
Relationship between Bugzilla and Testopia (cont.)

25. Relationship between Bugzilla and Testopia (cont.)
25
• Test Case
– Test Cases are the heart of all testing
– Test Cases spell out what steps to take while running a test and what outcomes to expect
– Test Cases are associated with one or more Test Plans and with zero or more Test Runs

26. Relationship between Bugzilla and Testopia (cont.)
26
• Test Run
– Once you have defined a set of Test Cases, you are ready to run through those tests in a
Test Run
– Each run is associated with a single Test Plan and can consist of any number of Test
Cases from that plan

27. Relationship between Bugzilla and Testopia (cont.)
27
• Build
– Software development is usually an iterative affair. In Testopia, each iteration is called a build
– In software testing, a string denoting the compiled results of a period of development
– Builds are often associated with milestones of the project. This is reflected in the build's
relationship to Bugzilla's target milestones object

28. Relationship between Bugzilla and Testopia (cont.)
28
• Case Run
– A Test Case-Run is the record of how a particular Test Case fared in a particular run for a given
Build in a given Environment

29. Relationship between Bugzilla and Testopia (cont.)
29
• Category
– A property of a product that is used to classify Test Cases

30. Relationship between Bugzilla and Testopia (cont.)
30
• Environment
– A list of the surrounding conditions that a test run is performed in

31. Testing Process of Testopia
31
The testing process in
Testopia is as follows :
Step 1. Create a Product and
multiple Components
Step 2. Create a Test Plan
Step 3. Create Test Cases
Step 4. Create a Build

32. Testing Process of Testopia (cont.)
32
The testing process in
Testopia is as follows :
Step 5. Create Environment
Step 6. Create a Test Run
Step 7. Execute the Test Run
Step 8. Produce a test report
on Dashboard

33. Testopia Installation
Step 1. Download Testopia
– http://www.mozilla.org/projects/testopia/
33
Step 2. Decompress the file to Bugzilla folder
Step 3. Run checksetup.pl and install module

34. Access to Testopia
Step 4. Re-run checksetup.pl
34
Step 5. Installation complete

35. Jenkins Introduction
• Jenkins is an open source continuous integration
tool written in Java
• Jenkins provides a web-based user interface
35

36. Jenkins Features
• Easy installation
• Easy configuration
• Change set support
• E-mail integration
• Test reporting
• Plugin support
36

37. Jenkins Installation
Step 2. Access to Jenkins
– http://localhost:8080
37
Step 1. Install Jenkins

38. Install Testopia plugin on Jenkins
38

39. Integrate Jenkins with Testopia
39
• This plug-in integrates Jenkins with Testopia
and generates reports on automated test
execution
• With this plug-in you can manage your tests in
Testopia, schedule and control in Jenkins
• It reads TAP test report formats, used to
update Testopia test case's executions

40. Integrate Jenkins with Testopia(cont.)
40
• Testopia configuration section

41. Auto Test Process
Test Controller
Ubuntu 13.10
Bugzilla 4.4
Testopia
Jenkins 1.5
Shell scripts (expect)
Shell scripts (expect)
Shell scripts (expect)
Step 1. Retrieve Test Case from Testopia
41
1.
Step 2. Invoke corresponding script
to initialize the test environment
2. Auto testing environment for ITRI
ARM hypervisor :

42. Auto Test Process (cont.)
Test Controller
Ubuntu 13.10
Bugzilla 4.4
Testopia
Jenkins 1.5
Shell scripts (expect)
Shell scripts (expect)
Shell scripts (expect)
Ubuntu 12.04 LTS
ITRI ARM Hypervisor
Guest
VM
Guest
VM
Guest
VM
Test Target
Step 3. Initialize the test environment
Step 4. Launch necessary Guest VMs
42
3.
4.

43. Auto Test Process (cont.)
Test Controller
Ubuntu 13.10
Bugzilla 4.4
Testopia
Jenkins 1.5
Shell scripts (expect)
Shell scripts (expect)
Shell scripts (expect)
Ubuntu 12.04 LTS
ITRI ARM Hypervisor
Guest
VM
Guest
VM
Guest
VM
Test Target
Step 5. Execute testing tasks
43
5.

44. Auto Test Process (cont.)
Test Controller
Ubuntu 13.10
Bugzilla 4.4
Testopia
Jenkins 1.5
Shell scripts (expect)
Shell scripts (expect)
Shell scripts (expect)
Ubuntu 12.04 LTS
ITRI ARM Hypervisor
Guest
VM
Guest
VM
Guest
VM
Test Target
Step 6. Generate and return test
result (by TAP, Test Anything Protocol)
44
6.

45. Auto Test Process (cont.)
Test Controller
Ubuntu 13.10
Bugzilla 4.4
Testopia
Jenkins 1.5
Shell scripts (expect)
Shell scripts (expect)
Shell scripts (expect)
Step 7. Return test result to Jenkins
45
7.

46. Auto Test Process (cont.)
Test Controller
Ubuntu 13.10
Bugzilla 4.4
Testopia
Jenkins 1.5
Shell scripts (expect)
Shell scripts (expect)
Shell scripts (expect)
Step 8. Write corresponding result to
Test Case in Testopia based on TAP
46
8.

47. Auto Test Example
Step 1. Add user on Bugzilla to execute auto testing
47

48. The process to create a Test Plan
48
• To create a new Test Plan in Testopia
– Step 1. Click the "New Plan" link in the Bugzilla footer
– Step 2. Enter a name for this Test Plan
– Step 3. Select a product from the product list
– Step 4. Select a type for this plan
– Step 5. Select a product version
– Step 6. Type or paste your plan document into the Plan
Document Editor
– Step 7. Click the "Submit" button

49. Step 2. Create a new Test Plan
49
• Plan Name
– Test Plan name

50. Step 2. Create a new Test Plan (cont.)
50
• Product
– Bugzilla product name

51. Step 2. Create a new Test Plan (cont.)
51
• Product Type
– The testing type for this Test Plan

52. Step 2. Create a new Test Plan (cont.)
52
• Product Version
– The Bugzilla product version

53. Step 2. Create a new Test Plan (cont.)
53
• Plan Document
– This is document that spells out the type of testing and testing methods
used for this plan

54. The process to create a Test Run
54
• To create a new Test Run in Testopia
– Step 1. Click "Create a New Test Run" link on your Test Plan
or the New Run link in the footer
– Step 2. Select which Test Cases to include
– Step 3. Enter a Summary for this Test Run
– Step 4. Select a build from the Build list or type the name
for a new one
– Step 5. Select an environment from the Environment list
– Step 6. Click the "Submit" button

55. Step 3. Create a new Test Run
55
• Product Version
– The Bugzilla product version

56. Step 3. Create a new Test Run (cont.)
56
• Run Manager
– The user in charge of this Test Run

57. Step 3. Create a new Test Run (cont.)
57
• Target Completion Rate
– The expect completion rate of target

58. Step 3. Create a new Test Run (cont.)
58
• Build
– The default product build

59. Step 3. Create a new Test Run (cont.)
59
• Environment
– The environment in use for this Test Run

60. Step 3. Create a new Test Run (cont.)
60
• Target Pass Rate
– The expect pass rate of target

61. Step 3. Create a new Test Run (cont.)
61
• Summary
– A short description of this Test Run

62. Step 3. Create a new Test Run (cont.)
62
• Notes
– This field is a place to make notes and observations about this run

63. The process to create a Test Case
63
• To create a new Test Case in Testopia
– Step 1. Click "Create a New Test Case" on the plan page or
New Case from the footer
– Step 2. Enter a short description of your Test Case in the
Summary field
– Step 3. Select a Category
– Step 4. Add a default tester
– Step 5. List the steps for testing in the Action field
– Step 6. Provide the expected outcomes in the Expected
Results field
– Step 7. Click the "Submit" button

64. Step 4. Create a new Test Case
64
• Summary
– A short description of the Test Case

65. Step 4. Create a new Test Case (cont.)
65
• Default Tester
– The person is assigned to this Test Case

66. Step 4. Create a new Test Case (cont.)
66
• Alias
– A unique string that uses to identify test result

67. Step 4. Create a new Test Case (cont.)
67
• Priority
– The level of testing

68. Step 4. Create a new Test Case (cont.)
68
• Category
– The product category

69. Step 4. Create a new Test Case (cont.)
69
• Estimated Time
– The estimated time that this Test Case should take to complete

70. Step 4. Create a new Test Case (cont.)
70
• Bugs
– You can attach bugs to your Test Cases

71. Step 4. Create a new Test Case (cont.)
71
• Blocks
– If this Test Case prevents others from being run their ID numbers will appear
here

72. Step 4. Create a new Test Case (cont.)
72
• Status : Test Case status
– PROPOSED : This test case is a work in progress and has not been reviewed for
accuracy. It is therefore not ready to be included in test runs
– CONFIRMED : This test case has passed review and is ready to be included in future test
runs. Only test cases with this status can be included in new test runs
– DISABLED : This test case is no longer applicable to current testing

73. Step 4. Create a new Test Case (cont.)
73
• Add Tags
– A user defined string used to classify Test Cases

74. Step 4. Create a new Test Case (cont.)
74
• Requirements
– The requirement number or URL to a document containing the requirement
this Test Case is designed to test

75. Step 4. Create a new Test Case (cont.)
75
• Automated
– Automatic Test Cases are run by a script

76. Step 4. Create a new Test Case (cont.)
76
• Scripts
– The name of the script that runs this Test Case

77. Step 4. Create a new Test Case (cont.)
77
• Arguments
– The specific arguments sent to the script

78. Step 4. Create a new Test Case (cont.)
78
• Add to Run
– Entering a run ID here will include this Test Case in a Test Run if it is not
already included

79. Step 4. Create a new Test Case (cont.)
79
• Depends On
– If this Test Case requires other Test Cases be run before this one, their case
numbers should appear here

80. Step 4. Create a new Test Case (cont.)
80
• Action
– This field lists the steps of testing

81. Step 4. Create a new Test Case (cont.)
81
• Expected Results
– This details the expected outcomes of a test

82. Process to Launch Test Script
82
Configure
1. Clean Environment
2. Check Image
3. Launch Test Script
Start Test Script
1. 3.
2.
1. Login to Host
2. Start VM
3. Ping Host
4. Turn off VM
5. Return Result
Testopia
Contents :
1. Parameters
2. Functions
4.
1 . Retrieve Test Script

83. Set up a new project
83
Step 5. Create a project on Jenkins to invoke Test
Script

84. Step 6. Set Testopia configuration section
84
• Testopia Version
– The version name of Testopia

85. Step 6. Set Testopia configuration section (cont.)
85
• Test Run ID
– The Test Run ID of Testopia

86. Step 6. Set Testopia configuration section (cont.)
86
• Single Test Build Steps
– The command to execute by single step

87. Step 6. Set Testopia configuration section (cont.)
87
• Iterative Test Build Steps
– The parameters of Testopia are used to execute by test cases

88. Step 7. Set Result configuration section
88
• Test Anything Protocol (TAP)
– TAP is a simple text-based interface between testing
modules in a test harness
– It allows individual tests to communicate test results
to the testing harness

89. TAP File Example
89
• The path of the file
• Success contents
• Fail contents

90. Launch The Project
90
Step 8. Start this project on Jenkins

91. Jenkins Console
91
Step 9. Monitor the shell scripts

92. Jenkins Result
92

93. Return Result to Testopia
93

94. Question
94
• Mail:walkerchang@iii.org.tw

95. References
95
• http://www.bugzilla.org/
• https://developer.mozilla.org/en-
US/docs/Mozilla/Bugzilla/Testopia#Requirements
• http://blog.codylab.com/testcase-management-
using-testopia/
• https://wiki.jenkins-
ci.org/display/JENKINS/Testopia+Plugin
• http://testanything.org/
• https://wiki.jenkins-
ci.org/display/JENKINS/Meet+Jenkins