This document discusses different levels of software testing including unit testing, integration testing, functional testing, and acceptance testing. It provides details on tools and frameworks for each level including JUnit and Maven for unit testing, the Failsafe plugin for integration testing, Selenium for functional testing, and Cucumber for acceptance testing. Continuous testing and test automation best practices are also covered.

1. Continuous Test Automation
Richard Langlois P. Eng. and Yuri Pechenko, May 2018

2. Agenda
• What is Continuous Testing
• Maven Recap
• Unit Testing
• Integration Testing
• Functional Testing
• Acceptance Testing
2

3. Software Testing Levels
3
• SOFTWARE TESTING LEVELS are the different stages of the software development lifecycle where testing is conducted.
• There are four levels of software testing: Unit >> Integration >> System >> Acceptance.

4. Continuous Testing
Continuous Testing is the process of executing automated tests as part of the software delivery pipeline in order to obtain
feedback on the business risks associated with a software release candidate as rapidly as possible.
4

5. Maven Recap
• Maven divides execution into the following nested hierarchies:
• Lifecycle,
• Phase,
• Plugin,
• Goal
5

6. Maven Recap
Lifecycle
• A Build Lifecycle is a well defined sequence of phases which define the order in which the goals are to be executed.
• Lifecycle allows to execute multiple Goals sequentially.
• Maven ships with three lifecycles:
• Clean
• Default
• Site
6

7. Maven Recap
Phases
• When executing a phase, all phases and bound plugin goals up to that point in the lifecycle are also executed.
• Each step in a lifecycle flow is called a phase.
7

8. Maven Recap
Plugin
• A plugin is a logical grouping and distribution (often a single JAR) of related goals, such as JARing.
8

9. Maven Recap
Goal
• A goal, the most granular step in Maven, is a single executable task within a plugin.
• Zero or more plugin goals are bound to a phase.
9

10. Unit Testing
10
• A level of the software testing process where individual units of a software are tested.
• The purpose is to validate that each unit of the software performs as designed.

11. Unit Testing
Maven – Dependencies
<junit5-version>5.1.1</junit5-version>
<dependency>
<groupId>org.junit.platform</groupId>
<artifactId>junit-platform-launcher</artifactId>
<version>1.1.0</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.junit.jupiter</groupId>
<artifactId>junit-jupiter-engine</artifactId>
<version>${junit5-version}</version>
<scope>test</scope>
</dependency>
11
<dependency>
<groupId>org.junit.jupiter</groupId>
<artifactId>junit-jupiter-api</artifactId>
<version>${junit5-version}</version>
</dependency>
OR
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.12</version>
</dependency>

12. Unit Testing
Surefire Plugin
Surefire plugin runs unit tests during the test phase.
12

13. Unit Testing
Maven Test Lifecycle
Use Maven Surefire Plugin by calling the test phase of the build lifecycle:
13

14. Unit Testing
Maven Surefire Plugin – Dependency:
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.21.0</version>
</plugin>
</plugins>
</build>
14

15. Unit Testing
XML/TXT Reports
The Surefire plugin generates reports in two different file formats:
● Plain text files (*.txt)
● XML files (*.xml)
By default, these files are generated in ${basedir}/target/surefire-reports/TEST-*.xml.
To generate .txt/.xml reports:
mvn site
15

16. Unit Testing
HTML Reports with Surefire Report Plugin
The Surefire Report Plugin
• parses the generated TEST-*.xml files under ${basedir}/target/surefire-reports
• creates the web interface version of the test results in ${basedir}/target/site/surefire-report.html folder
Goals:
● surefire-report:report Generates the test results report into HTML format.
● surefire-report:report-only This goal does not run the tests, it only builds the reports.
16

17. Unit Testing
Surefire Report
Dependency:
<reporting>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-report-plugin</artifactId>
<version>2.21.0</version>
</plugin>
</plugins>
</reporting>
17

18. Unit Testing
HTML Report
Sample of a report:
18

19. Unit Testing
HTML Report with Jenkins
The Jenkins JUnit plugin:
• provides a publisher that consumes XML test reports generated during the builds
• provides some graphical visualization of the historical test results
• provides a web UI for viewing test reports, tracking failures.
• JUnit publisher is configured at the job level by adding a Publish JUnit test result report post build action.
19

20. Integration Testing
20
• A level of the software testing process where individual units are combined and tested as a group.
• The purpose of this level of testing is to expose faults in the interaction between integrated units.

21. Integration Testing
Maven
• Default Lifecycle Phases for running integration test:
• pre-integration-test: sets up the integration test environment.
• integration-test: runs the integration tests.
• post-integration-test: tears down the integration test environment.
• Verify: checks the results of the integration tests.
Failsafe Plugin is designed to run integration tests, while the Surefire Plugin is designed to run unit tests.
The Failsafe Plugin has 2 goals:
● failsafe:integration-test:
• runs the integration tests of an application
● failsafe:verify
• verifies that the integration tests of an application passed
21

22. Integration Testing
Maven Failsafe Plugin
• Use Maven Failsafe Plugin by calling the verify phase of the build lifecycle:
22

23. Integration Testing
Maven structure:
In the source folder, we have an extra directory called "it", with its usual java and resources folder.
23
Failsafe Plugin

24. Integration Testing
By default, the Failsafe Plugin will automatically include all test classes with the following wildcard patterns:
• "**/IT*.java" - includes all of its subdirectories and all Java filenames that start with "IT".
• "**/*IT.java" - includes all of its subdirectories and all Java filenames that end with "IT".
• "**/*ITCase.java" - includes all of its subdirectories and all Java filenames that end with "ITCase".
To specify test sources folder other than the default:
• Configure Failsafe plugin with includes, excludes attribute
• Configure build-helper-maven-plugin with sources attribute
24
Failsafe Plugin

25. Integration Testing
Failsafe Plugin uses the exact same format as the Surefire Plugin.
To generate a report, just add a second Surefire Report Plugin report set using the Failsafe reports directory, and run:
mvn site
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-report-plugin</artifactId>
<version>2.21.0</version>
<reportSets>
<reportSet>
<id>integration-tests</id>
<reports>
<report>failsafe-report-only</report>
</reports>
</reportSet>
</reportSets>
</plugin>
25
Reports

26. Integration Testing
Jenkins
stage('Test'){
steps{
bat 'mvn test'
}
}
26
New maven job: mvn verify or mvn integration-test
Artifact: Jenkinsfile
stage('Verify'){
steps{
bat 'mvn verify'
}
}

27. Functional Testing
27
SYSTEM TESTING is a level of software testing where a
complete and integrated software is tested.
The purpose of this test is to evaluate the system’s
compliance with the specified requirements.
ACCEPTANCE TESTING is a level of software testing
where a system is tested for acceptability.
The purpose of this test is to evaluate the system’s
compliance with the business requirements and assess
whether it is acceptable for delivery.

28. Functional Testing
28
List of Checks (Test Cases)
Test Framework (TestNG vs JUnit)
Test Scope (Chrome, Internet
Explorer, FireFox etc.)
Selenium driver (Selenuim)
Alithya Digital Solutions Center : using JUnit(4 or 5) + Selenium

29. Functional Testing
Selenium
• Selenium is a portable software-testing framework for web applications
29

30. Functional Testing
Maven - Dependencies for Selenium
<dependency>
<groupId>org.seleniumhq.selenium</groupId>
<artifactId>selenium-java</artifactId>
<version>3.11.0</version>
</dependency>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.12</version>
</dependency>
30

31. Functional Testing
Web Driver Manager
The Web Driver Manager is a library that is aimed to automate the Selenium WebDriver binaries management in runtime for Java.
System.setProperty("webdriver.chrome.driver", "/absolute/path/to/binary/chromedriver");
System.setProperty("webdriver.gecko.driver", "/absolute/path/to/binary/geckodriver");
System.setProperty("webdriver.opera.driver", "/absolute/path/to/binary/operadriver");
System.setProperty("phantomjs.binary.path", "/absolute/path/to/binary/phantomjs");
System.setProperty("webdriver.edge.driver", "C:/absolute/path/to/binary/MicrosoftWebDriver.exe");
System.setProperty("webdriver.ie.driver", "C:/absolute/path/to/binary/IEDriverServer.exe");
WebDriverManager comes to the rescue, performing in an automated way all this dirty job for you.
● it downloads the necessary binary
● it checks for the latest version each time
● it is fully configurable
31

32. Maven - Dependencies for Webdriver Manager
<dependency>
<groupId>io.github.bonigarcia</groupId>
<artifactId>webdrivermanager</artifactId>
<version>2.2.1</version>
<scope>test</scope>
</dependency>
If using JUnit 5, add:
<dependency>
<groupId>io.github.bonigarcia</groupId>
<artifactId>selenium-jupiter</artifactId>
<version>2.1.1</version>
</dependency>
32

33. Functional Testing
Implementation
Page object model: every page -> separate java class
Access page elements (buttons, checkboxes, fields etc.) with css and xpath selectors.
Application Oracle Selenium Demo
33

34. Functional Testing
Automation Test Approach Data Driven (DDD)
Test Scenario:
1. Open Login Page
2. Login
3. Open Saisie Adhérents page
4. Select random employee group
5. Create a new employee line
6. Fill in a new employee
7. Check created line for importing
8. Click addEmployeeButton
9. Verify that the employee was added to the system.
34

35. Functional Testing
Reporting
• <dependency>
• <groupId>com.aventstack</groupId>
• <artifactId>extentreports</artifactId>
• <version>3.1.5</version>
• <scope>test</scope>
• </dependency>
35
Extent Report Demo

36. Acceptance Testing
• Cucumber (Java) supports Behavior Driven Development (BDD) – Specflow (C#)
• It explains the behavior of the application in a simple English text using Gherkin language.
• Scenarios are part of tests
36
Automation Test Approach Behavior-Driven

37. Acceptance Testing
Cucumber
37

38. Acceptance Testing
In the test runner, the @CucumberOptions offers plugin option to specify different formatting options for the output reports.
The following example will generate a Json report at the mentioned location:
@RunWith(Cucumber.class)
@CucumberOptions(features = {"src/test/java/features/Feature01.feature"},
format = {"json:target/cucumber.json"},
glue = "tests.employees")
public class CucumberRunner {
//empty class for linking Feature class and java class
Json report -> HTML report with Maven-Cucumber-Reporting plugin
38
Reports

39. Acceptance Testing
Maven Dependencies
<cucumber-version>2.4.0</cucumber.version>
<dependency>
<groupId>io.cucumber</groupId>
<artifactId>cucumber-java</artifactId>
<version>${cucumber-version}</version>
</dependency>
<dependency>
<groupId>io.cucumber</groupId>
<artifactId>cucumber-java8</artifactId>
<version>${cucumber-version}</version>
</dependency>
<dependency>
<groupId>io.cucumber</groupId>
<artifactId>cucumber-java8</artifactId>
<version>${cucumber-version}</version>
</dependency>
39
<plugin>
<groupId>net.masterthought</groupId>
<artifactId>maven-cucumber-reporting</artifactId>
<version>3.16.0</version>
<executions>
<execution>
<id>execution</id>
<phase>post-integration-test</phase>
<goals>
<goal>generate</goal>
</goals>
<configuration>
<projectName>${project.name}</projectName>
<outputDirectory>${project.build.directory}</outputDirectory>
</configuration>
</execution>
</executions>
</plugin>

40. Acceptance Testing
40
Jenkinsfile
• The simplest way to get Cucumber report in Jenkins is to Install cucumber-reports-plugin for Jenkins which takes json
report file of cucumber report and returns pretty html report.
• After installation of the plugin a method cucumber(params…) will be available for pipeline usage.
• cucumber '**/*.json‘
Jenkins file:
post{
always{
cucumber '**/*.json'
}
}
Cucumber automation example

41. References
• Software Testing Fundamentals (http://softwaretestingfundamentals.com)
• Cucumber Home (https://cucumber.io)
• Using a Jenkinsfile (https://jenkins.io/doc/book/pipeline/jenkinsfile)
• Declarative Pipeline with Jenkins (https://dzone.com/storage/assets/6132886-dzone-rc241-declarativepipelinewithjenkins-
2.pdf)
41