Complex Software Systems Testing Strategies

On Complex Software
Systems Testing
Alexey Zverev,
co-CEO & co-founder, Exactpro

Distributed exchange trading system and data
flow

Value Meaning
0 New
1 Partially Filled
C Expired
4 Cancelled
2 Filled
8 Rejected
9 Suspended
Orders transitions after library execution
New (100)
New (279)
New (4784)
END
NEW
Rejected
Filled
Expired
Cancelled
START
New (38)
Reject (3222)
(3222)
(100)
(38)
(279)
(4784)
Cancel (100)
Cancel (38)
Expired (279)
Trade (4784)
Trade (38)
TradeCancel (38)

Production order life cycle diagram
Amend (94)
START
END
Expired
Cancelled
Filled
Rejected
PartiallyFilledNEW
New (1166642)
Trade (23816)
Trade (29361)
Reject (4224)
Amend (937420)
Amend (2)
Trade
(30933)
Expire (270741)
Cancel (144)
Expire (38987)
Cancel (1062624)
Trade (2944)
Amend (28530)
Trade
(14489)
Expire (16400)
Cancel (23647)
Trade
(14520)
(19701)
(4224)
(184)
(58370)
(326128)
(1086415)
Suspended
(44)

Workflows comparison:
- Gap analysis/coveragein the test library/run;
- Efficiency in test execution;
- Edge cases/raceconditions;
- Are all the use cases from Prod/UAT covered?
Specifications
Test runs (Logs,msgs)
FT / NFT
UAT/Production
(Logs, msgs)
Single database
creation
Process mining
model
Process mining workflows

Update testing library coverage

Mistakes make us better
SUCCESS
Failure
Failure
Failure
Failure
Failure
Failure
Failure

What is Sailfish?
Web
Binary
FIX
Market Data
Database

What is Sailfish?
#step #description #user_name #action #message_type SecurityID OrdType Side OrderQty Price Status
Test Case 1: Trade test case start
1 Send BUY Order with Qty = 1000, Price = 10 (O1) User_1 send NewOrderSingle 12345 LIMIT BUY 1000 10
2 Receive Execution Report with Status = NEW (O1) User_1 receive ExecutionReport 12345 LIMIT BUY 1000 10 NEW
3 Send SELL Order with Qty = 1000, Price = 10 (O2) User_2 send NewOrderSingle 12345 LIMIT SELL 1000 10
4 Receive Execution Report with Status = FILLED (O2) User_2 receive ExecutionReport 12345 LIMIT SELL 1000 10 FILLED
5 Receive Execution Report with Status = FILLED (O1) User_1 receive ExecutionReport 12345 LIMIT BUY 1000 10 FILLED
test case end

The main actions of Sailfish
#step #description #user_name #action #message_type SecurityID OrdType Side OrderQty Price Status
Test Case 1: Trade test case start
1 Send BUY Order with Qty = 1000, Price = 10 (O1) User_1 send NewOrderSingle 12345 LIMIT BUY 1000 10
2 Receive Execution Report with Status = NEW (O1) User_1 receive ExecutionReport 12345 LIMIT BUY 1000 10 NEW
3 Send SELL Order with Qty = 1000, Price = 10 (O2) User_2 send NewOrderSingle 12345 LIMIT SELL 1000 10
4 Receive Execution Report with Status = FILLED (O2) User_2 receive ExecutionReport 12345 LIMIT SELL 1000 10 FILLED
5 Receive Execution Report with Status = FILLED (O1) User_1 receive ExecutionReport 12345 LIMIT BUY 1000 10 FILLED
test case end

Passed and Failed test steps
send NewOrderSingle (PASSED) [0.001s]
- receive ExecutionReport (PASSED) [0.001s]:
+ Input Parameters
- Verification: Message (PASSED):
Field Expected Result Actual Result Status
... ... ... ...
OrdType LIMIT LIMIT PASSED
Status NEW NEW PASSED
... ... ... ...

Passed and Failed test steps
- receive ExecutionReport (FAILED) [0.421s]:
+ Input Parameters
+ Verification: Similar message [1]. Failed/Passed/Conditionally Passed/NA: 4/26/0/6 (FAILED)
- Verification: Similar message [4]. Failed/Passed/Conditionally Passed/NA: 1/29/0/6 (FAILED):
... ... ... ...
Status NEW REJECTED FAILED
... ... ... ...

Supervised Machine Learning through
unpleasant experience
+ Input Parameters
- Verification: Similar message [4]. Failed/Passed/Conditionally Passed/NA: 1/29/0/6 (FAILED):
... ... ... ...
... ... ... ...

Supervised Machine Learning through
unpleasant experience
+ Input Parameters
- Verification: Similar message [1]. Failed/Passed/Conditionally Passed/NA: 1/29/0/6 (FAILED): 89%
+ Verification: Similar message [2]. Failed/Passed/Conditionally Passed/NA: 4/26/0/6 (FAILED) 81%
... ... ... ...
... ... ... ...

Supervised Machine Learning: going forward
tens of thousands of tests
thousands of raised bugs

Supervised Machine Learning: going forward
This test step failed
with the following
results...
This test step failed
with the following
results…
And the reason is...

Dancing with Whales:
Adaptive Log Classification System
Elena Treshcheva,
Business Development Manager,
Researcher, Exactpro

What turns a problem into a disaster?

Mission-Critical Software - Lots of log messages
daily

Huge volumes of data vs. error class clusters

WhaleShark
WhaleShark
The First IEEE International Conference on Artificial Intelligence Testing
(IEEE AITest 2019), April 4-9 2019, San Francisco East Bay, CA, USA
User-Assisted Log Analysis for Quality Control of Distributed Fintech
Systems
Iosif Itkin, Anna Gromova, Anton Sitnikov, Rostislav Yavorskiy, Evgenii Tsymbalov,
Andrey Novikov and Kirill Rudakov.

One error class - different text strings
text text text text
0100101010001010000
ERROR
text text text text
text text text text
text text text text
text text text text
FATAL……………………………………
text text text text
text text text text
It’s so cold and
dark... I’m scared...

Step 1: Normalization
ERROR 07 Aug 2018 13:04:39 - Unable to load messages from file
'/UAT/collectors/mq1'
ERROR 25 Sep 2018 18:24:55 - Unable to load messages from file
'/prod/collectors/mq3'
ERROR 16 Oct 2018 15:15:10 - Unable to load messages from file

Step 1: Normalization
ERROR 07 Aug 2018 13:04:39 - Unable to load messages from file
'/UAT/collectors/mq1'
ERROR 25 Sep 2018 18:24:55 - Unable to load messages from file
ERROR ** *** **** **:**:** - Unable to load messages from file
'/*/*/*'
ERROR 16 Oct 2018 15:15:10 - Unable to load messages from file

Result: System state visualization

More Control over system outputs

Kirill Anisimov,
Senior QA Analyst, Exactpro
On Traceability and the Illusion
of Control

When I was 5 years old, my mother always
told me that happiness was the key to life.
When I went to school, they asked me
what I wanted to be when I grew up.
I wrote down ‘happy’. They told me I didn’t
understand the assignment, and I told
them they didn’t understand life.
John Lennon

Hey Stress, don't make it bad
Sherman, G. D., Lee, J. J., Cuddy, A. J. C., Renshon, J., Oveis, C., Gross, J. J., &
Lerner, J. S. (2012). Leadership is associated with lower levels of stress.
Proceedings of the National Academy of Sciences, 109(44), 17903–17907.

Fenton-O’Creevy, M., Nicholson, N., Soane, E.,
& Willman, P. (2003). “Trading on illusions:
Unrealistic perceptions of control and trading
performance”. Journal of Occupational and
Organizational Psychology, 76(1), 53–68.
Imagine all the brokers trading on...

We all live in an illusion of control

Software testing is relentless
learning, continuous
improvement and keeping
abreast of cutting-edge
technologies.
The research and development
team at Exactpro uses machine
learning methods of intellectual
data analysis to create next-
generation program analysis
tools.
We work in an agile environment
collaborating with a multinational
team of software developers.
R&D Case Study: Cradle Test Database
for Machine Learning

They told me I didn’t understand the
assignment and the test report does not
make anyone happy. I told them they
didn’t understand software testing.
Software testing is our life.

Demystifying DLT Testing One Network at a Time
Ekaterina Volobueva,
QA Analyst, Exactpro

The complexity of post-trade
- There is a multitude of components in modern
complex post-trade infrastructures;
- Upstream and downstream system dependency;
- The participant structure is very complex;
- Trade/Position/Account life cycle;
- The number of Asset Classes may vary;
- The complexity of the Risk calculation process;
- Access via a set of API endpoints.

Maybe blockchain?
J.Gardner decision model https://medium.com/@sbmeunier/when-do-you-need-
blockchain-decision-models-a5c40e7c9ba1

Morgen E. Peck decision model https://medium.com/@sbmeunier/when-do-you-
need-blockchain-decision-models-a5c40e7c9ba1

D. Birch decision model https://medium.com/@sbmeunier/when-do-you-need-
blockchain-decision-models-a5c40e7c9ba1
Do I need a blockchain?
No.

Is it OK to release my blockchain into live-
service?

ClearTH for DLT-Based Post-Trade Systems research based on R3 Corda
technology has won the Best Tool Demo Award in Xi’an, China

Testing Distributed Applications

Functional testing
Non-functional
testing

Exactpro Test Framework and Test Approach for
DLT-based system
3 steps on the ledger side:
1st - Initialisation of positions
2nd - Trade creation and participant
notification
3rd - Positions updates
Across each and every step the test
framework provides different methods
to validate the endpoints, inputs and
outputs.
The actual business flow steps and
validation check items are transformed
into ClearTH matrix after execution of
which the deviations from the expected
behaviour can be analysed.

ClearTH and Woodpecker for DLT
is an Exactpro test harness able to simultaneously
execute multiple end-to-end test scenarios in
batches.
● Supports multiple protocols (SWIFT, FIX,
Market Data, etc.)
● Automatically runs test scripts
● Executes concurrent automated tests
● Allows to create multiple-day test scenarios
● Has integrated scheduler and simulators
● Verifies each stage of the system daily life cycle

Meet Exactpro at CordaCon in October

Get the MOST from FIX
Marina Soloveva,
QA Project Manager, Exactpro

Open source strategy
Exactpro has donated 4 tools to Open Source:
2018 2019

71 Build Software to Test Software
exactpro.com

It is not called Georgia
• Territory: 69,700 sq km
• Population: ~4 million
• Capital: Tbilisi (1.5 million)
Five Cross Flag
ძალა ერთობაშია
Strength is in Unity
Sakartvelo საქართველო

Why Sakartvelo
To provide additional level of security to our Clients and Exactpro business, we
have decided to diversify our delivery locations by country.
Georgia is very well ranked in terms of economic freedom, ease of doing business
and corruption.

A Warm Location
19 September 2018
As part of executing our
strategy of global growth as
a company focused on
software testing and building
software to test our clients’
software, we launched
Exactpro Systems Georgia
and our new project delivery
center in Tbilisi.

Software Testing is Relentless Learning -
QA Community Tbilisi

GeoSTQB
• Increase the awareness about ISTQB among
testers within different companies and universities
• Promote professionalism in Software Testing in
Georgia by implementing an internationally
recognized certified tester scheme
• Establish certification of IT professionals in Georgia
according to ISTQB standards
• Support talented and promising testers
22 February 2019

Partnerships
Exactpro has recently joined the ISTQB® Partner Program as a platinum partner
alongside 74 other companies.
Joining the ISTQB® Partner Program allows us to play a bigger role in the
software testing community and to highlight our goal of providing accessible,
internationally relevant high-quality education to our staff
July 2019

Software Testing is Relentless Learning
The International Conference on Software
Testing, Machine Learning and Complex
Process Analysis (TMPA-2019) –
Tbilisi, Georgia on 7-9 November 2019.
Focus on the application of modern methods of
data science to the analysis of software quality.
www.tsu.ge

Welcome!
მოგესალმებით

exactpro.com

Exactpro in 2019
• Capital Markets

Exactpro in 2019
• Capital Markets
• Risk Management & Post Trade

Exactpro in 2019
• Capital Markets
• Risk Management & Post Trade
• Information Services

Software Testing is Relentless Learning

Information Services – Automated Testing

Changes in Software
• Extend regression libraries
• Reduce the gaps in the test coverage
• Build faster software to execute more tests
• Run tests concurrently and under load
• Repeat the same tests multiple times
• Introduce more diversity in tools and data sets
• Intensify Use of Chaos Engineering

Changes in Hardware
A) Work Collaboratively with Our Clients
B)

exactpro.com

Exactpro Test Cloud
• Cradle for the next generation of test tools
• Built-in test coverage and compliance reporting
• Pricing tailored to massive defect mining processes
• Located in UK, EU, US to satisfy data regulation

Exactpro Plans
• Cradle for the next generation of test tools
• Built-in test coverage and compliance reporting
• Pricing tailored to massive defect mining processes
• Located in UK, EU, US to satisfy data regulation
• Consider raising capital for it in 2021

Complex Software Systems Testing Strategies

Recommended

Recommended

More Related Content

Similar to Complex Software Systems Testing Strategies

Similar to Complex Software Systems Testing Strategies (20)

More from Iosif Itkin

More from Iosif Itkin (20)

Recently uploaded

Recently uploaded (20)

Complex Software Systems Testing Strategies