3. GFT Group 27/06/2016 3
Executive Summary
GFT GROUP
This presentation discusses the important role test automation plays in shaping Quality Assurance in Agile delivery
methodology and process. The topics include:
Test Automation
Test automation approach and benefits
Test automation framework
Industrialised test automation
Agile Testing
Agile delivery approach overview, activities, roles and responsibilities
Agile testing scope, approach, and activities
Agile testing phases and process
Role of test automation in shaping Agile testing
4. GFT Group 27/06/2016 4
Test Automation Overview– Approach
GFT GROUP
Approach
Test automation solutions can be implemented through either custom-built solution or 3rd party products
Test automation framework should be portable, reusable, easily maintained and designed to be transitioned to client’s team for
both GUI and server side automations
Test automation solutions allow financial institutions to significantly increase test effectiveness and test coverage, therefore
ensuring a considerable reduction in “time to test” and decreased cost of testing over time
It should be recognised that test automation comes at a cost and the predicted return on that investment should be achievable.
For that reason, the early estimation of effort and expected benefits is critical as is the ongoing collection of metrics to validate
that ROI
The most practical automation solution should be pursued – whether using a known testing tool (e.g. QTP or Selenium, etc) or
through the development of own ‘fit for purpose’ test harness
The approach is to provide deliverables (automated test suites) early and then on a regular basis such that the overall test
automation framework provides visible and increasing benefit – end-to-end
Test automated solutions will, wherever feasible, be integrated into the build process to deliver an effective continuous
integration solution
5. GFT Group 27/06/2016 5
Test Automation Overview - Benefits
GFT GROUP
Benefits
Increased efficiency and coverage of tests through effective test automation
Reduced overhead of regression testing
Reduced cost of testing over time
Reliability and predictability of tests by eliminating risk of human error
Greater number of test runs possible in regression testing but also as smoke tests or within a continuous build /
integration environment
Enables the realisation of Agile development methodology and delivers shippable product after each sprint
6. GFT Group 27/06/2016 6
What is a “Test Automation Framework”?
The key to successful test automation is to build a
solution that is usable by all test analysts, is robust
(can handle unexpected events and system errors),
reuses common test steps and is easily maintainable.
Using a test automation tool ‘out of the box’ will only
cover these criteria to a small extent as the principle
behind most tools is to record the steps taken –
including all keystrokes, mouse operations, screen
navigation – that a manual tester performs to execute
a test such that the test can be played back multiple
times. This has numerous drawbacks: e.g. every
single test has to be recorded and so no reuse; as
soon as the system changes so all the tests have to
be re-recorded.
The answer therefore is to build libraries of common
functions; parameterise the test scripts and externally
hold as much as possible in relation to the test
actions (using keywords) and test data. In addition
build error handling, smart reporting, test
dependencies and importantly an interface to the
underlying test automation tool that can be used by
all test analysts. These are all components of what is
referred to as the Test Automation Framework.
The full framework encapsulates the automated regression test capability at
both the front (GUI) end and the server side (database/interfaces/messages);
the ability to manage the testing and the use of the framework; the tracking of
progress through dashboards and reports
Test Automation
Framework
Automated
RegressionScripts
Automated
Smoke Test
Scripts
Standards
Function
Libraries
Environment
Variables
Common
Libraries
Object
Repository
Error Handling
Tools and
Utilities
TestAutomationTool
(e.g.QTP,Selenium)
Data
Test ManagementArchive
Risk
Management
Trade
Processing
Finance &
Accounting
Pricing &
Analytics
Reporting &
Audit
Settlement
UpstreamSystems
DownstreamSystems
Application Under Test
Auto Test
GUI
Usability
Portability Maintainability
Flexibility
Robustness
Reuse
GFT GROUP
The GFT Test Automation Framework
7. GFT Group 27/06/2016 7
Description
Industrialised test automation approach harmonises different
testing activities applied to the different subsystems in a large
scale systems integration project. This harmonization enables
reuse of testware, test environments, test simulations, test
automation and testing concepts for the systems integration
project.
Approach
Reach a testing maturity level where the test process is well
established and the corresponding working products are
defined and controlled, at which point reuse can take place
systematically.
For synchronisation of the test processes between different
parties, an independent third party can be utilised which
guarantees that each involved party is working on reaching the
testing maturity level in sync with all other parties.
Objective
Overall goal is to increase test efficiency by a high degree of
reuse and to increase effectiveness by combining different test
environments to have more intermediate test stages on the
integration path.
Benefits
Many synergies by reusing testware that can be leveraged,
reducing test efforts significantly for all involved parties.
Improve the overall quality of the system, since each involved
party can focus its test activities on those aspects that are core
for their deliverables
Industrialised test automation enables layered automated
integration testing.
Industrialised Test Automation
GFT GROUP
Industrialised test automation approach applies to a large scale systems integration project:
8. GFT Group 27/06/2016 8
Agile Delivery Approach - Overview
Project Kick Off
Ensure that there are clearly defined
sprint and release goals (i.e. through to
delivery milestone).
Agree and document the definition of
(‘Done’) at Sprint level and Release level
Agree and publish the team’s sprint cycle
showing the length and schedule for
each sprint iteration.
Define and document the triage
processes for story readiness and
defects
Plan for automated testing and
continuous integration
Ongoing
Keep the daily stand up brief: what I did
yesterday, what I will be doing today, any
blockers; take issues offline.
The agreed Definition of ‘Done’ must be
adhered to
Do not change the agreed sprint items unless
directed to by the Product Owner.
Deploy each sprint to UAT or production
(depending on definition of ‘Done’)
2 31 4 5 6 7 8 9 10
ACTIVITIES
Deliver & Test Stories
Business & Regression Test
Code Freeze
Sprint Planning
Show & Tell
Retrospective
Daily Stand Up
Defect Triage
REGULAR
MEETINGS
Project
Working Group
ARTE-
FACTS
Delta Report
Commitment
List Project Status
Report
Project Status
Report
Backlog Backlog Refinement
Estimate &
Prioritise Stories
DAY
Deliver & Test Stories
Code Freeze & Deployment
Check
Point
Project
Working Group
Scrum of
Scrums
Show & Tell Rehearsal
GFT GROUP
9. GFT Group 27/06/2016 9
The role of testing within an Agile development environment is of paramount importance with the testers fully integrated within the team so that testing
requirements can drive development
Agile Testing Approach
The GFT ‘sprint testing’ approach includes:
Test Driven Development (TDD)
Behaviour Driven Development (BDD)
Exploratory Testing
Incremental Testing
Test Driven Development
Prioritised
requirements
backlog
Prioritised Manual and
Automated Test
Scripts
Daily Meetings
Daily Reports
(Metrics)
Continuous Automated
Regression Test
Release ready to
go live
Go-Live
Readiness
Weekly Test Reports
GFT Sprint Testing Approach
GFT GROUP
Automation
The implementation of a test
automation solution is essential and
will bring significant benefits in an
Agile environment.
Automating tests as early as possible
and building a regression pack to run
on a daily basis
10. GFT Group 27/06/2016 10
GFT implementation teams focus on the following areas in the context of Agile testing.
GFT GROUP
Agile Testing Scope and Activities
Designing a testing framework and infrastructure can be as complex as designing and implementing
the system being tested. Test analysts focus on doing the technical design of testing frameworks that
are able to meet requirements for testing individual releases and ongoing system test and certification
Functional test design requires tight alignment between the business analysts that are defining the
functional capability of the system and the test analysts that are defining the test cases. Test analysts
and engineers should be deployed to work as part of an integrated design and development team and
ensure the specification of comprehensive test coverage
Test automation experts engage with clients to design and implement test automation frameworks that
are portable, reusable, maintainable, and able to achieve increased test effectiveness, reduced testing
time and reduced overall cost. Test automation tools are analysed and selected for both functional and
non-functional test.
One of the most difficult aspects of testing is creating and managing the data that drives the testing
framework. Test teams deploy for clients a project specific approach to the management and masking
of real production data and the generation of comprehensive test data where the use of live data is not
possible
The test team is able to provide strategic direction and hands-on implementations around non-
functional testing and in particular, performance tuning, performance testing, system availability testing
and security
Test Design
Functional Testing
Test Automation
Test Data Management
Non-Functional
Testing
11. GFT Group 27/06/2016 11
Testing for individual user stories are done
within the sprint (i.e. the definition of Done
includes the completion of testing)
A continuous integration environment is in
place to enable code deployed daily and
also on-demand.
Automated test suites kicked off after code
deployment
SIT and UAT are run after each sprint in
an iterative and progressive manner
Each sprint aims to deliver a shippable
product to production environment
Test automation and Continuous
Integration plays vital role in the realisation
of such approach
GFT GROUP
Agile Testing Phases and Process
SprintNstart
Sprint N (Sprint test)
SprintNend
Sprint N deployed
to SIT env
UAT execution
ProdDeployable
Continuous Integration Test
CIDeployment
SprintN+1start
SprintN+1end
Product Backlog Refinement
Sprint N deployed
to UAT env
Activity
Test Phase
12. GFT Group 27/06/2016 12
Test Automation in Agile Testing
GFT GROUP
Levels and areas
Unit tests
Backend (DB, services)
UI (validating behaviour or UX design)
Continuous Integration Testing (daily
build and automated test)
Full E2E solutions (with all system
components and dependency systems)
Non-functional testing (Performance
validation, soak test)
Implementation
Test framework based on open source, custom solution or mix
Building re-usable solutions (reducing next project kick-off cost)
Detailed test cases prepared by QA
Tests can be written by QA or Dev (fast test coverage increase if
needed)
Coverage can be measured by traceability matrix
All tests integrated with CI environment to produce daily reports
Test automation built for both functional and non-functional test