Alexey Zverev, CEO of Exactpro, presented on testing complex systems within agile development frameworks. The presentation outlined fundamental principles of QA management, including continuously learning about the system and improving test design through iterations. It then discussed common "anti-patterns" such as over-relying on requirements traceability matrices or test plans. Finally, it summarized Exactpro's principles, such as developing team knowledge, versus these anti-patterns.

1. Alexey Zverev
CEO
Exactpro
Testing Complex
Systems within
Agile Development
Framework

2. Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49
www.exactpro.com2
QA management at Exactpro
Open Access Quality Assurance & Related Software Development for Financial Markets
Tel: +7 495 640 2460, +1 415 830 38 49
www.exactpro.com
Alexey Zverev, Co-CEO, Exactpro

3. Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49
www.exactpro.com3
Introduction

4. Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49
www.exactpro.com4
Agenda
1. Fundamental principles of QA Management
2. The anti-patterns
3. Summary of our principles vs. the anti-patterns
4. Questions & Answers

5. Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49
www.exactpro.com5
Fundamental principles of
QA Management
QA is a continuous learning and development process
1. Learn how the system should operate in production
2. Learn about the actual behaviour
3. Make sure all discrepancies are either fixed or documented
4. Develop Test Design, Test Tools and an Automated Test Library through a series of
small iterations
This is a permanent process throughout the delivery cycle and
after the actual Go-Live, whilst building the test tools adds more efficiency,
and therefore, we find as many issues as possible.

6. Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49
www.exactpro.com6
The anti-patterns
1. “The Requirement Traceability Matrix”
2. “Test Automation”
3. “The Test Library”
4. “The Test Plan”
5. “Non-Functional Testing”
6. “The Test Tool”
7. “The Scrum Team”

7. Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49
www.exactpro.com7
“The Requirements Traceability Matrix”
“We need to write Test Cases at the
beginning of the project”
“All Test Cases must contain direct
links to Requirements”
“Stakeholders must see test coverage and correspondence between tests
cases and software requirements”

8. Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49
www.exactpro.com8
“Test Automation”
“We must automate 100% of our
Test Library.
Let’s start now”
“Complete regression must
happen overnight as part of the
build process”
“As a result of test automation we should reduce the QA team to a few
people”

9. Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49
www.exactpro.com9
“The Test Library”
“We must document all our test scenarios and
put them into Test Management Software”
“Test Library consists of Test Suites which
consist of Test Scenarios which consist of
Test Cases”
“Once we accomplish this we will start test automation”
“Exploratory Testing? What is it? No we don’t do it”
“All test cases contain detailed test steps and
Testers must put Pass/Fail as they execute them”

10. Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49
www.exactpro.com10
“The Test Plan”
“Let’s estimate QA effort required to
deliver this project”
“Lets plan QA tasks and start
implementing them in waterfall model”
“Let’s monitor process closely and put
our effort into achieving targets and
milestones”

11. Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49
www.exactpro.com11
“Non-Functional Testing”
“We need to test that the system
supports 10,000 orders per second”
“We need to make sure that latency is
within expected limits”
“We will allocate production-like
hardware to focus on capacity and
latency measurements”

12. Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49
www.exactpro.com12
“The Test Tool”
“We have purchased a licence
for a magical test automation
tool”
“We will use it for all of our
test automation purposes”
“It allows interacting with GUI / via FIX / via SWIFT and has the capacity to
record/replay test scenarios”
“It is expensive”

13. Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49
www.exactpro.com13
“The Scrum Team”
“We are using scrum and will test
everything as a part of the sprint”
“Testing is done and automated by
Developers who can temporarily
allocate their time within scrum”
“We will put some QA engineers into Scrum team and let them write unit
tests”
“We are using DevOps we don’t need QA, QA is dead”

14. Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49
www.exactpro.com14
Summary of our principles vs. the anti-patterns
1. Developing team knowledge vs. “The Requirement Traceability Matrix”
2. Building software to test software vs. “Test Automation”
3. Test Design vs. “The Test Library”
4. Agile iterative approach vs. “The Test Plan”
5. Testing at the confluence of Functional and Non-Functional Testing vs. “Non-Functional
Testing”
6. Pragmatic in-house development of test tools vs. “The The Test Tool”
7. The QA team is a pacemaker vs. “The Scrum Team”

15. Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49
www.exactpro.com15
Questions & Answers
Thank you!