High level tutorial on automated testing foundations, practices, and patterns.

1. John Liebenau
Enterprise Architecture / Software Engineering © 2017 John Liebenau 1
Automated Testing Tutorial
Foundations, Patterns, and Best Practices

2. John Liebenau
Enterprise Architecture / Software Engineering
2
1/27/2021
Objectives
Present spectrum of automated testing and establish context for discussing automated
testing’s place in application development
Describe various aspects of automated testing including
• different kinds of automated testing,
• motivations for using automated testing, and
• benefits of automated testing
Discuss techniques, patterns, and best practices for automated testing

3. John Liebenau
Enterprise Architecture / Software Engineering
3
1/27/2021
Overview
Organization
1. Context for Automated Testing
2. Definitions and Motivations
3. Patterns, Best Practices, and Examples
Assumptions
• Familiarity with object-oriented design
─ UML class diagrams
─ design patterns
• Some exposure to xUnit style testing frameworks
─ JUnit
─ NUnit
─ MSTest

4. John Liebenau
Enterprise Architecture / Software Engineering
Context for Automated Testing
4

5. John Liebenau
Enterprise Architecture / Software Engineering
5
1/27/2021
Automated
Testing Levels
Unit Testing
Integration Testing
Acceptance Testing
Manual Testing
Tests individual unit of functionality
Does not interact with environment (databases, networks, filesystems, …)
Unit Tests should execute extremely fast
Tests collaborations between components
May interact with environment (databases, networks, filesystems, …)
Integration Tests typically execute slower than Unit Tests
Tests functional scenarios for regression and user story acceptance
Attempts to test end-to-end in production-like environment
(however automated tests typically bypass GUI)
Acceptance Tests may execute slower than Integration Tests
Explores seldom exercised paths through system functionality
End-to-end testing in production-like environment
Manual Tests are most expensive to execute

6. John Liebenau
Enterprise Architecture / Software Engineering
Unit Testing
Manual Testing
Acceptance Testing
Integration Testing
Unit Testing
Automated Testing supports Delivery Pipeline
Version
Control
Artifact
Repository
Commit Stage
build system
run commit test suite
store build artifacts
report results
Integration Test Stage
run integration test suite
report results
Manual Test Stage
install system/prepare environment
do exploratory testing
report results
Developer Tester
build system
run commit test suite
commit changes
store build artifacts retrieve build artifacts retrieve build artifacts
select build/install system
perform testing
report results
Acceptance Test Stage
install system/prepare environment
run acceptance test suite
report results
retrieve build artifacts
BuildServer
trigger trigger
trigger
provide feedback
provide feedback

7. John Liebenau
Enterprise Architecture / Software Engineering 7
1/27/2021
Release Management in Delivery Pipeline
Development Pipeline
Commit Stage Integration Test Stage Acceptance Test Stage Manual Test Stage
Release Pipeline Initiator
Create Release Branch Create Release Pipeline Activate Release Pipeline
Release Pipeline
Commit Stage Integration Test Stage Acceptance Test Stage Manual Test Stage

8. John Liebenau
Enterprise Architecture / Software Engineering
Definitions and Motivations
8

9. John Liebenau
Enterprise Architecture / Software Engineering
What is Automated Testing
Writing software that tests an application's functionality at multiple levels
• Unit Testing focuses on individual classes as the test subjects
• Integration Testing focuses on collaborations of classes or components as the test
subjects
• Acceptance Testing focuses on application components that implement specific
scenarios or user stories as the test subjects
Automated tests follow a common pattern
• Specify preconditions and postconditions for each test
─ A precondition is the legal and required state of test subject instance prior to executing a test
─ A postcondition is the legal and required state of a test subject instance after executing a test
• Tests are conducted by
─ Setting up a test subject with the appropriate preconditions
─ Exercising a specific functionality of the test subject
─ Checking for the expected postconditions
• If the postconditions are correct the test subject has passed the test

10. John Liebenau
Enterprise Architecture / Software Engineering
Why is Automated Testing Necessary
Long Term
• Enables longer application lifetime and greater ROI
• Applications continue delivering business value for longer period of time
─ Increased ability to adapt to changes in user requirements and environment
─ Increased system quality as more tests are written
Short Term
• Enables development team to code with more confidence, increasing productivity
─ Reduction in the time spent debugging and integrating
─ More errors are discovered and fixed in development preventing them from being released into production
Automated Testing also:
• Verifies test subject semantics. Automated tests specify the exact behavior of your test subjects
and test them to ensure they conform to that specification
• Enables refactoring. Automated tests are the main tool needed for rapid and successful refactoring
• Improves code quality. Well written automated tests thoroughly exercise your code, catching errors
early in development
• Eliminates manual repetition of test plans. Automated tests can be run efficiently and faithfully
each time your code has changed freeing QA personnel to concentrate on some of the more
difficult and creative aspects of testing

11. John Liebenau
Enterprise Architecture / Software Engineering
Goals supported by Automated Testing
Goal Definition Support
Maximize System Quality • Establish an application architecture and
development method that increases
application quality in a cost effective manner
• Deliver applications that do not generate
production support issues due to application
logic errors or coding errors
• Automated tests repeatedly validate application
code as new functionality is added and existing
functionality is changed
• Automated tests enable development teams to
deliver more value by eliminating production
errors, freeing developers to focus on
application development and not application
support
Maximize System Adaptability • Establish an application architecture and
development method that can effectively and
efficiently adapt to changes in: user
requirements, business environment, and
technology
• Automated tests are the foundation that enables
effective application adaptation and refactoring
Minimize Development and
Operation Cost
• Establish practices and make decisions that
reduce the total cost of developing and
operating applications without
compromising quality
• Automated tests can be economical to develop
by organizing tests into levels and allocating
effort according to the cost of developing tests
at each level
• Automated tests generate significant cost
reductions over expensive manual testing
• Comprehensive automated tests prevent costly
errors from being released into production
Minimize Delivery Time • Establish practices that reduce the amount of
time to deliver applications without
compromising quality
• Automated tests reduce overall application
delivery time by dramatically reducing the time
needed for integration and debugging

12. John Liebenau
Enterprise Architecture / Software Engineering
Patterns, Best Practices, and Examples
12

13. John Liebenau
Enterprise Architecture / Software Engineering
Four Phase Test Pattern
Context
• We are designing automated tests to
test a system
Problem
• How do we structure test logic to make
what we are testing obvious
Solution
• Structure each test with four distinct
parts that are executed in sequence
─ Set Up
─ Exercise
─ Verify
─ Tear Down
Related Patterns
• Dependency Injection
─ Four Phase Test often uses Dependency
Injection during Set Up
TestCase
- subject: TestSubject
«creator»
+ setUp() : void
«destroyer»
+ tearDown() : void
«test»
+ testMethod() : void
«action»
+ runTest() : void
TestSubject
+ method(Input) : Output
Assert
«verifier»
+ assertEquals(Object, Object) : void
void
runTest()
{
setUp(); // Set Up Phase
testMethod(); // Exercise Phase (in-lines Verify Phase)
tearDown(); // Tear Down Phase
}
void
testMethod()
{
Input input = new Input( ... );
Output expected = new Output( ... );
Output actual = null;
assert...( ... ); // Verify Phase (Precondition)
actual = subject.method( input ); // Exercise Phase
assertEquals( expected,actual); // Verify Phase (Postcondition)
}
Inteface
Test Subject
Test Case
Helper
Legend
tests

14. John Liebenau
Enterprise Architecture / Software Engineering
Test Double Pattern
Context
• A class being tested—the test subject—depends on the interfaces of other components
Problem
• How do we isolate the test subject from the components it depends on in order to do unit testing
Solution
• Replace the components the test subject depends on with test specific equivalents—test
doubles—that mimic the components’ behavior in a way that is more suitable for testing
Variations
• Mock Object
─ Accepts and verifies inputs from test subject
─ Returns outputs to test subject
• Fake Object
─ Implements a component’s functionality in a simpler way suitable for testing
Related Patterns
• Dependency Injection
─ Often used to configure test subjects with Test Doubles

15. John Liebenau
Enterprise Architecture / Software Engineering
Dependency Injection Pattern
Context
• A class being tested—the test subject—depends on the interfaces of other components
Problem
• How do we design the test subject so we can replace its dependencies at runtime
Solution
• Provide mechanisms in the test subject’s interface (such as constructors or setters) that enable clients to configure
the test subject with appropriate dependent components
Variations
• Constructor Injection
─ Configure the test subject with the desired dependencies during its construction
• Setter Injection
─ Configure the test subject with the desired dependencies through setter methods
• Parameter Injection
─ Pass the desired dependency to the test subject through a parameter when a test subject method is invoked
Related Patterns
• Four Phase Test
─ Dependency Injection is used by Four Phase Test to Set Up the state of Test Subjects
• Test Double
─ Dependency Injection is often used to configure Test Subjects with Test Doubles

16. John Liebenau
Enterprise Architecture / Software Engineering
Best Practices
Depend on Interfaces (not Implementations)
• Types of variables and parameters should be interfaces
• Clients only know about the interfaces they expect
• Clients remain unaware of the classes that implement the objects they use
• Greatly reduces implementation dependencies between objects
Apply Single Responsibility Principle
• Design classes that each have only one overarching responsibility
• All methods of such a class are aligned to supporting its responsibility
• Complex behavior is formed by interacting networks of objects each responsible for elements of
the behavior
Favor Object Composition over Class Inheritance
• Helps keep each class encapsulated and focused on one responsibility
• Classes and class hierarchies will remain small and will be less likely to grow to be unmanageable
Isolate Unit Test Subjects with Test Doubles
• Set up test subjects with test doubles in order to isolate test subjects for unit testing
Optimize tests to run as fast as possible
• Once tests are in place, look for opportunities to improve test performance
• Faster tests mean faster feedback

17. John Liebenau
Enterprise Architecture / Software Engineering
17
1/27/2021
Structure of a Unit Test
OrderServiceImpl
- notifier: TraderNotifier
- repository: OrderRepository
«action»
+ processOrder(OrderRequest) : OrderResponse
«modifier»
+ setRepository(OrderRepository) : void
+ setTraderNotifier(TraderNotifier) : void
OrderServiceTest
«creator»
+ setUp() : void
«destroyer»
+ tearDown() : void
«test»
+ testProcessOrder() : void
«interface»
OrderRepository
«modifier»
+ insertOrder(Order) : void
+ removeOrder(Order) : void
«selector»
+ getOrder(OrderId) : Order
+ getOrders(OrderCriteria) : Collection<Order>
«query»
+ hasOrder(OrderId) : boolean
«interface»
TraderNotifier
«notifier»
+ notifyCreatedOrder(CreatedOrderDto) : void
+ notifyCancelledOrder(CancelledOrderDto) : void
«interface»
OrderService
«action»
+ processOrder(OrderRequest) : OrderResponse
MockTraderNotifier
InMemoryOrderRepository
Inteface
Class Being Tested
Test Case
Mock (Test Double)
Fake (Test Double)
Legend
configures
configures
uses
uses
tests

18. John Liebenau
Enterprise Architecture / Software Engineering
18
1/27/2021
Structure of an Integration Test
OrderServiceImpl
- notifier: TraderNotifier
- repository: OrderRepository
«action»
+ processOrder(OrderRequest) : OrderResponse
«modifier»
+ setRepository(OrderRepository) : void
+ setTraderNotifier(TraderNotifier) : void
OrderServiceTest
«creator»
+ setUp() : void
«destroyer»
+ tearDown() : void
«test»
+ testProcessOrder() : void
«interface»
OrderRepository
«modifier»
+ insertOrder(Order) : void
+ removeOrder(Order) : void
«selector»
+ getOrder(OrderId) : Order
+ getOrders(OrderCriteria) : Collection<Order>
«query»
+ hasOrder(OrderId) : boolean
«interface»
TraderNotifier
«notifier»
+ notifyCreatedOrder(CreatedOrderDto) : void
+ notifyCancelledOrder(CancelledOrderDto) : void
«interface»
OrderService
«action»
+ processOrder(OrderRequest) : OrderResponse
InMemoryOrderRepository
Inteface
Test Subject
Test Case
Mock (Test Double)
Fake (Test Double)
Legend
ActualTraderNotifier
DbOrderRepository
Can use fake or real depending on the integration test.
tests
uses
uses
configures
tests
tests

19. John Liebenau
Enterprise Architecture / Software Engineering
19
1/27/2021
Developing a New Class with Unit Testing
1. Declare the class interface
2. Specify each method's preconditions and postconditions (where necessary)
3. Implement each method as a stub
4. Write or generate the unit test skeleton
5. For each public method in the class, implement the test based on the preconditions
and postconditions
6. Implement a method in the class
7. Run the tests
8. If the test passes repeat with another method, else fix the method or the test (which
ever is incorrect) and rerun the test
9. Repeat this process until all methods are fully implemented and pass all of the tests

20. John Liebenau
Enterprise Architecture / Software Engineering
20
1/27/2021
Changing Released Code with Automated Testing
1. Identify how or where bug or enhancement affects code
2. Add test(s) that exercises bug or enhancement
• Could involve unit test, integration test, and acceptance test
3. Run test(s)
• The test (or tests) must fail
• Failure is the indication that the test exercises the bug or enhancement
4. Fix affected method(s) for a bug fix or add the necessary functionality for an
enhancement
5. Run all tests
• This includes all unit tests, integration tests, and acceptance tests
6. If all tests pass, then issue new release, else continue fixing and rerun tests until all
tests pass

21. John Liebenau
Enterprise Architecture / Software Engineering
Appendix
21

22. John Liebenau
Enterprise Architecture / Software Engineering
22
1/27/2021
Testing Tools and Frameworks
Test Level Tools Information
Acceptance Testing Fit http://fit.c2.com/
Fitnesse http://fitnesse.org/
Custom developed tools
Integration Testing JUnit http://www.junit.org/
NUnit http://www.nunit.org/
MS Testing Framework http://msdn.microsoft.com/en-us/library/ms243147.aspx
Unit Testing JUnit http://www.junit.org/
NUnit http://www.nunit.org/
MS Testing Framework http://msdn.microsoft.com/en-us/library/ms243147.aspx
HtmlUnit http://htmlunit.sourceforge.net/
JMock http://www.jmock.org/
Moq http://code.google.com/p/moq/

23. John Liebenau
Enterprise Architecture / Software Engineering
23
1/27/2021
Additional Information
Books
• xUnit Test Patterns
─ http://www.amazon.com/xUnit-Test-Patterns-Refactoring-Code/dp/0131495054
• Growing Object-Oriented Software Guided By Tests
─ http://www.amazon.com/Growing-Object-Oriented-Software-Guided-Tests/dp/0321503627
• Continuous Delivery
─ http://www.amazon.com/Continuous-Delivery-Deployment-Automation-Addison-Wesley/dp/0321601912
Web Sites
• http://xunitpatterns.com/
• http://www.javaworld.com/javaworld/jw-05-2004/jw-0510-tdd.html
• http://www.junit.org/
• http://www.nunit.org/
• http://continuousdelivery.com