Avik and Gautam delivered a Lightning Talk on Non-Functional Testing of Chatbots at #ATAGTR2020 ATAGTR2020 was the 5th Edition of Global Testing Retreat. Avik has 10 years of experience as a Non-Functional Testing (NFT) enthusiast and is associated with Cognizant’s NFT CoE. Gautam has 10+ years in performance testing & engineering. He is an agile enthusiast & passionate learner who is always on the lookout to learn & implement new technologies & tools. The video recording of the session is now uploaded on the following link: https://youtu.be/yHJ53hTP1Qo To know more about #ATAGTR2020, please visit: https://gtr.agiletestingalliance.org/

1. #ATAGTR2020
As a author of this presentation I/we own the copyright and confirm the originality of the content. I/we allow Agile testing alliance to use the content for social media marketing, publishing it on ATA Blog or ATA social medial
channels(Provided due credit is given to me/us)
#ATAGTR2020
As a author of this presentation I/we own the copyright and confirm the originality of the content. I/we allow Agile testing alliance to use the content for social media marketing, publishing it on ATA Blog or ATA social medial
channels(Provided due credit is given to me/us)
Non Functional Testing of
Chatbots
Avik Chatterji
&
Gautam Kar
Cognizant Worldwide Limited

2. #ATAGTR2020
As a author of this presentation I/we own the copyright and confirm the originality of the content. I/we allow Agile testing alliance to use the content for social media marketing, publishing it on ATA Blog or ATA social medial
channels(Provided due credit is given to me/us)
#ATAGTR2020
As a author of this presentation I/we own the copyright and confirm the originality of the content. I/we allow Agile testing alliance to use the content for social media marketing, publishing it on ATA Blog or ATA social medial
channels(Provided due credit is given to me/us)
Agenda
Conversational AI (Chatbots) Overview
Risk Areas in Chabot world
Performance Evaluation Area
Performance Testing Approach
Probable use of Chatbot for NFT
Challenges
Conclusion

3. #ATAGTR2020
As a author of this presentation I/we own the copyright and confirm the originality of the content. I/we allow Agile testing alliance to use the content for social media marketing, publishing it on ATA Blog or ATA social medial
channels(Provided due credit is given to me/us)
#ATAGTR2020
As a author of this presentation I/we own the copyright and confirm the originality of the content. I/we allow Agile testing alliance to use the content for social media marketing, publishing it on ATA Blog or ATA social medial
channels(Provided due credit is given to me/us)
Conversational AI (Chatbots) Overview
What is Conversational AI?
Conversational AI is a smart technology
that people can interact with simply by
having a conversation, our most natural
form of interaction
To automate repetitive tasks and reduce
operational cost, increase user
experience & increase efficiency
Intelligent
Advanced software learns from past
interactions, improving responses
over time
Engaging
Enables human-like interaction
delivered through a channel that
is easily scalable
Effective
Let users perform tasks efficiently
and accurately through
conversational self service
Security & Privacy
Complies with security and privacy
requirements.
Faceless
Less importance on user interface
Cost Effective
Low maintenance cost
Why?

4. #ATAGTR2020
As a author of this presentation I/we own the copyright and confirm the originality of the content. I/we allow Agile testing alliance to use the content for social media marketing, publishing it on ATA Blog or ATA social medial
channels(Provided due credit is given to me/us)
#ATAGTR2020
As a author of this presentation I/we own the copyright and confirm the originality of the content. I/we allow Agile testing alliance to use the content for social media marketing, publishing it on ATA Blog or ATA social medial
channels(Provided due credit is given to me/us)
Risk Areas in Chabot world
User Experience
Concurrency
Network speed
Geographically
distributed Users
Client-side
performance
Chatbot space is expected for a whooping growth(85%) by 2025, an early hedge position is a
must for business in order to mitigate potential risks. NFT plays a crucial role in this spec
Quick Information
Application
configurations
Poorly
performing code
Database
Infrastructure
Cloud
performance
Backend servers
health
Scalability/ Auto
scaling
Better Availability
24/7 availability
Resource
leakage

5. #ATAGTR2020
As a author of this presentation I/we own the copyright and confirm the originality of the content. I/we allow Agile testing alliance to use the content for social media marketing, publishing it on ATA Blog or ATA social medial
channels(Provided due credit is given to me/us)
#ATAGTR2020
As a author of this presentation I/we own the copyright and confirm the originality of the content. I/we allow Agile testing alliance to use the content for social media marketing, publishing it on ATA Blog or ATA social medial
channels(Provided due credit is given to me/us)
Performance Evaluation Area
ClientSideCloud/OnPremiseBackend
Client Layer
UI Layer
Web API Layer
Cognitive Layer
Data Layer
Integration Layer
External System
Python
IE , Chrome etc.
HTML 5,CSS (Web Bots)
VoiceXML (VPAs)
IIS
No SQL
Web API / REST
External Systems
Web Browser (only for Web based Chatbots)
Web User Interface
Authentication / Authorization Configuration
NLP / ML Business Logic Success and Failure Logs
Postgre SQL database
API Handler
SSO Application API’s E-Mail
SMTP
DeviceLevel
Analysis
EndtoEndResponseTime
Server
Performance
APITesting
Technology (e.g.)Layers High Level Architecture NFT Analysis
Voice User Interface

6. #ATAGTR2020
As a author of this presentation I/we own the copyright and confirm the originality of the content. I/we allow Agile testing alliance to use the content for social media marketing, publishing it on ATA Blog or ATA social medial
channels(Provided due credit is given to me/us)
#ATAGTR2020
As a author of this presentation I/we own the copyright and confirm the originality of the content. I/we allow Agile testing alliance to use the content for social media marketing, publishing it on ATA Blog or ATA social medial
channels(Provided due credit is given to me/us)
Performance Testing Approach
Device level
Analysis
& E2E Response
Time
Types of Testing
API Testing &
Server Side
Performance
Analysis
User Experience
 Capture metrics such as Accuracy, Precision and Recall
Bot responsiveness Test
 Measure single user end to end response time from
device/UI using Echo Simulator Script/ LR Scripts
 Measure responsiveness under varying network b/w
and latency
Load and Stress testing
 Validate server side responsiveness and system
resources consumption
 E2E testing for web based Chatbots
 Load testing through API calls
Endurance testing
 Check system reliability when subjected to significant
user load for prolonged duration.
Autoscale testing
 Benchmarking of Autoscale conditions(CPU limit)
 Verifying fault tolerance during scale in/out situation
Single User
Device based
Concurrent Users
via Load Generators
Application Layer
Micro services Database Server
Analysis Type
Single User
Concurrent
Users
 Device level Analysis
 E2E Response Time for VPAs
 API testing for VPAs
 E2E performance testing for web-based
Chatbots

7. #ATAGTR2020
As a author of this presentation I/we own the copyright and confirm the originality of the content. I/we allow Agile testing alliance to use the content for social media marketing, publishing it on ATA Blog or ATA social medial
channels(Provided due credit is given to me/us)
#ATAGTR2020
As a author of this presentation I/we own the copyright and confirm the originality of the content. I/we allow Agile testing alliance to use the content for social media marketing, publishing it on ATA Blog or ATA social medial
channels(Provided due credit is given to me/us)
Probable use of Chatbot for NFT
 Automated Root Cause Analysis - We can use AI enabled Cognitive agents to automate root cause analysis by
feeding historic data such as response time, CPU utilization, memory utilization from server logs over time and for
different user load. If trained and fed with data for a considerable amount of time, Cognitive Agents might be
quite helpful in bottleneck analysis.
 Workload Modelling - Cognitive agents can be fed and trained for workload modelling by feeding data from
source systems such as server logs or Google Analytics. It can also be trained to predict system behaviour under
the anticipated user growth so that remedial steps can be taken to evade any non-functional issues in future.
 Security Analysis - Cognitive agents can be used in penetration testing and predicting any security breach.
 Emails and alerts - Cognitive agents can also be trained to send automated personalized emails and alerts, once a
performance test is complete with high level observations or in case of performance metrics crossing their
optimal level.
 Converting server logs into readable format with personalized touch - Cognitive Agents can be used to create
readable data with proper graphs and texts with data feed from the server logs. These graphs can be modified
and presented as per user choice and requirement as Chatbots have sentiment analysis feature.
 NFR Analysis - Cognitive Agents can be trained and used to validate NFRs based on architectural and other details
 Tool identification – We can design and train Chatbots to identify the right tool for a customer, based on input
data such as budget, application architecture, number of concurrent user, volume and other NFRs

8. #ATAGTR2020
As a author of this presentation I/we own the copyright and confirm the originality of the content. I/we allow Agile testing alliance to use the content for social media marketing, publishing it on ATA Blog or ATA social medial
channels(Provided due credit is given to me/us)
#ATAGTR2020
As a author of this presentation I/we own the copyright and confirm the originality of the content. I/we allow Agile testing alliance to use the content for social media marketing, publishing it on ATA Blog or ATA social medial
channels(Provided due credit is given to me/us)
Challenges
Most platforms are proprietary with limited access
• NLP/NLU services like Diagflow, Wit.ai Or Luis are subject to constant
training & improvement. Having such non– deterministic component
i.e. for the same data we might be getting different output, makes it
extremely difficult for testing as we cant predict the exact reason for
the failures.
Design scripts for unpredictable user input
• When testing a Chatbot, either with voice or with text interface, there
are no interaction barriers for users. Websites and smartphone apps
allow predefined means of interaction with common user interface
components (clickable hyperlinks, buttons, text entry boxes, …).
Chatbots have to cover all kind of unexpected user input in a decent
way. Designing scripts to emulate this behaviour is difficult.
User accent and voice
• Non Linear output is a major concern for Chatbots with voice
interface as the Chatbots many a times have problems with the
accent and voice of the users.
Most platforms are proprietary with limited access
• API’s of NLP layer are proprietary and not all are available for testing.
Some are available but it involves significant cost as the platform
provider charge according to the number of text records that are sent
for analysis. End to end load testing for VPAs is a challenge as
cloud based platforms might not allow to generate load on
their systems
Gradual learning process
• Chatbots keeps learning from each iteration of testing and evolves
over a period of time. Therefore in the beginning , the results &
predictions will never be 100% accurate.
AI/ML needs specialization
• Chatbots combined with AI & ML can help predict by quickly going
over results, logs & application code to predict application
bottlenecks & deviation of load test results etc. However in order to
inspect through voluminous data, appropriate data models need to
be developed, for which one will require performance engineers with
considerable data science experience.

9. #ATAGTR2020
As a author of this presentation I/we own the copyright and confirm the originality of the content. I/we allow Agile testing alliance to use the content for social media marketing, publishing it on ATA Blog or ATA social medial
channels(Provided due credit is given to me/us)
#ATAGTR2020
As a author of this presentation I/we own the copyright and confirm the originality of the content. I/we allow Agile testing alliance to use the content for social media marketing, publishing it on ATA Blog or ATA social medial
channels(Provided due credit is given to me/us)
Conclusion
 Chatbot as a technology is growing faster
than expected and will impact our
everyday life in the very near future
 Chatbots will be a critical part of
business strategy and digital road
map for all business across different
sectors.
 To stay ahead in the race, business
need to be ready for this
technology shift and provide a
better customer experience
 Non-Functional testing of Chatbots are clearly a
mandate in today’s world
 But now it is time to pause and
think differently. Instead of
working to develop only
solutions, why not we embrace
this technology to develop
leading edge Non Functional
Testing Services.
 It would be a wise and prudent
step to develop new non-
functional testing approach by
leveraging Chatbots , which will
replace the traditional testing
approach.

10. #ATAGTR2020
As a author of this presentation I/we own the copyright and confirm the originality of the content. I/we allow Agile testing alliance to use the content for social media marketing, publishing it on ATA Blog or ATA social medial
channels(Provided due credit is given to me/us)
#ATAGTR2020
As a author of this presentation I/we own the copyright and confirm the originality of the content. I/we allow Agile testing alliance to use the content for social media marketing, publishing it on ATA Blog or ATA social medial
channels(Provided due credit is given to me/us)
Thanks