Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

#ATAGTR2020 Presentation - Non-Functional Testing of Chatbots

8 views

Published on

Avik and Gautam delivered a Lightning Talk on Non-Functional Testing of Chatbots at #ATAGTR2020

ATAGTR2020 was the 5th Edition of Global Testing Retreat.

Avik has 10 years of experience as a Non-Functional Testing (NFT) enthusiast and is associated with Cognizant’s NFT CoE.

Gautam has 10+ years in performance testing & engineering. He is an agile enthusiast & passionate learner who is always on the lookout to learn & implement new technologies & tools.

The video recording of the session is now uploaded on the following link: https://youtu.be/yHJ53hTP1Qo

To know more about #ATAGTR2020, please visit: https://gtr.agiletestingalliance.org/

Published in: Technology
  • Be the first to comment

  • Be the first to like this

#ATAGTR2020 Presentation - Non-Functional Testing of Chatbots

  1. 1. #ATAGTR2020 As a author of this presentation I/we own the copyright and confirm the originality of the content. I/we allow Agile testing alliance to use the content for social media marketing, publishing it on ATA Blog or ATA social medial channels(Provided due credit is given to me/us) #ATAGTR2020 As a author of this presentation I/we own the copyright and confirm the originality of the content. I/we allow Agile testing alliance to use the content for social media marketing, publishing it on ATA Blog or ATA social medial channels(Provided due credit is given to me/us) Non Functional Testing of Chatbots Avik Chatterji & Gautam Kar Cognizant Worldwide Limited
  2. 2. #ATAGTR2020 As a author of this presentation I/we own the copyright and confirm the originality of the content. I/we allow Agile testing alliance to use the content for social media marketing, publishing it on ATA Blog or ATA social medial channels(Provided due credit is given to me/us) #ATAGTR2020 As a author of this presentation I/we own the copyright and confirm the originality of the content. I/we allow Agile testing alliance to use the content for social media marketing, publishing it on ATA Blog or ATA social medial channels(Provided due credit is given to me/us) Agenda Conversational AI (Chatbots) Overview Risk Areas in Chabot world Performance Evaluation Area Performance Testing Approach Probable use of Chatbot for NFT Challenges Conclusion
  3. 3. #ATAGTR2020 As a author of this presentation I/we own the copyright and confirm the originality of the content. I/we allow Agile testing alliance to use the content for social media marketing, publishing it on ATA Blog or ATA social medial channels(Provided due credit is given to me/us) #ATAGTR2020 As a author of this presentation I/we own the copyright and confirm the originality of the content. I/we allow Agile testing alliance to use the content for social media marketing, publishing it on ATA Blog or ATA social medial channels(Provided due credit is given to me/us) Conversational AI (Chatbots) Overview What is Conversational AI? Conversational AI is a smart technology that people can interact with simply by having a conversation, our most natural form of interaction To automate repetitive tasks and reduce operational cost, increase user experience & increase efficiency Intelligent Advanced software learns from past interactions, improving responses over time Engaging Enables human-like interaction delivered through a channel that is easily scalable Effective Let users perform tasks efficiently and accurately through conversational self service Security & Privacy Complies with security and privacy requirements. Faceless Less importance on user interface Cost Effective Low maintenance cost Why?
  4. 4. #ATAGTR2020 As a author of this presentation I/we own the copyright and confirm the originality of the content. I/we allow Agile testing alliance to use the content for social media marketing, publishing it on ATA Blog or ATA social medial channels(Provided due credit is given to me/us) #ATAGTR2020 As a author of this presentation I/we own the copyright and confirm the originality of the content. I/we allow Agile testing alliance to use the content for social media marketing, publishing it on ATA Blog or ATA social medial channels(Provided due credit is given to me/us) Risk Areas in Chabot world User Experience Concurrency Network speed Geographically distributed Users Client-side performance Chatbot space is expected for a whooping growth(85%) by 2025, an early hedge position is a must for business in order to mitigate potential risks. NFT plays a crucial role in this spec Quick Information Application configurations Poorly performing code Database Infrastructure Cloud performance Backend servers health Scalability/ Auto scaling Better Availability 24/7 availability Resource leakage
  5. 5. #ATAGTR2020 As a author of this presentation I/we own the copyright and confirm the originality of the content. I/we allow Agile testing alliance to use the content for social media marketing, publishing it on ATA Blog or ATA social medial channels(Provided due credit is given to me/us) #ATAGTR2020 As a author of this presentation I/we own the copyright and confirm the originality of the content. I/we allow Agile testing alliance to use the content for social media marketing, publishing it on ATA Blog or ATA social medial channels(Provided due credit is given to me/us) Performance Evaluation Area ClientSideCloud/OnPremiseBackend Client Layer UI Layer Web API Layer Cognitive Layer Data Layer Integration Layer External System Python IE , Chrome etc. HTML 5,CSS (Web Bots) VoiceXML (VPAs) IIS No SQL Web API / REST External Systems Web Browser (only for Web based Chatbots) Web User Interface Authentication / Authorization Configuration NLP / ML Business Logic Success and Failure Logs Postgre SQL database API Handler SSO Application API’s E-Mail SMTP DeviceLevel Analysis EndtoEndResponseTime Server Performance APITesting Technology (e.g.)Layers High Level Architecture NFT Analysis Voice User Interface
  6. 6. #ATAGTR2020 As a author of this presentation I/we own the copyright and confirm the originality of the content. I/we allow Agile testing alliance to use the content for social media marketing, publishing it on ATA Blog or ATA social medial channels(Provided due credit is given to me/us) #ATAGTR2020 As a author of this presentation I/we own the copyright and confirm the originality of the content. I/we allow Agile testing alliance to use the content for social media marketing, publishing it on ATA Blog or ATA social medial channels(Provided due credit is given to me/us) Performance Testing Approach Device level Analysis & E2E Response Time Types of Testing API Testing & Server Side Performance Analysis User Experience  Capture metrics such as Accuracy, Precision and Recall Bot responsiveness Test  Measure single user end to end response time from device/UI using Echo Simulator Script/ LR Scripts  Measure responsiveness under varying network b/w and latency Load and Stress testing  Validate server side responsiveness and system resources consumption  E2E testing for web based Chatbots  Load testing through API calls Endurance testing  Check system reliability when subjected to significant user load for prolonged duration. Autoscale testing  Benchmarking of Autoscale conditions(CPU limit)  Verifying fault tolerance during scale in/out situation Single User Device based Concurrent Users via Load Generators Application Layer Micro services Database Server Analysis Type Single User Concurrent Users  Device level Analysis  E2E Response Time for VPAs  API testing for VPAs  E2E performance testing for web-based Chatbots
  7. 7. #ATAGTR2020 As a author of this presentation I/we own the copyright and confirm the originality of the content. I/we allow Agile testing alliance to use the content for social media marketing, publishing it on ATA Blog or ATA social medial channels(Provided due credit is given to me/us) #ATAGTR2020 As a author of this presentation I/we own the copyright and confirm the originality of the content. I/we allow Agile testing alliance to use the content for social media marketing, publishing it on ATA Blog or ATA social medial channels(Provided due credit is given to me/us) Probable use of Chatbot for NFT  Automated Root Cause Analysis - We can use AI enabled Cognitive agents to automate root cause analysis by feeding historic data such as response time, CPU utilization, memory utilization from server logs over time and for different user load. If trained and fed with data for a considerable amount of time, Cognitive Agents might be quite helpful in bottleneck analysis.  Workload Modelling - Cognitive agents can be fed and trained for workload modelling by feeding data from source systems such as server logs or Google Analytics. It can also be trained to predict system behaviour under the anticipated user growth so that remedial steps can be taken to evade any non-functional issues in future.  Security Analysis - Cognitive agents can be used in penetration testing and predicting any security breach.  Emails and alerts - Cognitive agents can also be trained to send automated personalized emails and alerts, once a performance test is complete with high level observations or in case of performance metrics crossing their optimal level.  Converting server logs into readable format with personalized touch - Cognitive Agents can be used to create readable data with proper graphs and texts with data feed from the server logs. These graphs can be modified and presented as per user choice and requirement as Chatbots have sentiment analysis feature.  NFR Analysis - Cognitive Agents can be trained and used to validate NFRs based on architectural and other details  Tool identification – We can design and train Chatbots to identify the right tool for a customer, based on input data such as budget, application architecture, number of concurrent user, volume and other NFRs
  8. 8. #ATAGTR2020 As a author of this presentation I/we own the copyright and confirm the originality of the content. I/we allow Agile testing alliance to use the content for social media marketing, publishing it on ATA Blog or ATA social medial channels(Provided due credit is given to me/us) #ATAGTR2020 As a author of this presentation I/we own the copyright and confirm the originality of the content. I/we allow Agile testing alliance to use the content for social media marketing, publishing it on ATA Blog or ATA social medial channels(Provided due credit is given to me/us) Challenges Most platforms are proprietary with limited access • NLP/NLU services like Diagflow, Wit.ai Or Luis are subject to constant training & improvement. Having such non– deterministic component i.e. for the same data we might be getting different output, makes it extremely difficult for testing as we cant predict the exact reason for the failures. Design scripts for unpredictable user input • When testing a Chatbot, either with voice or with text interface, there are no interaction barriers for users. Websites and smartphone apps allow predefined means of interaction with common user interface components (clickable hyperlinks, buttons, text entry boxes, …). Chatbots have to cover all kind of unexpected user input in a decent way. Designing scripts to emulate this behaviour is difficult. User accent and voice • Non Linear output is a major concern for Chatbots with voice interface as the Chatbots many a times have problems with the accent and voice of the users. Most platforms are proprietary with limited access • API’s of NLP layer are proprietary and not all are available for testing. Some are available but it involves significant cost as the platform provider charge according to the number of text records that are sent for analysis. End to end load testing for VPAs is a challenge as cloud based platforms might not allow to generate load on their systems Gradual learning process • Chatbots keeps learning from each iteration of testing and evolves over a period of time. Therefore in the beginning , the results & predictions will never be 100% accurate. AI/ML needs specialization • Chatbots combined with AI & ML can help predict by quickly going over results, logs & application code to predict application bottlenecks & deviation of load test results etc. However in order to inspect through voluminous data, appropriate data models need to be developed, for which one will require performance engineers with considerable data science experience.
  9. 9. #ATAGTR2020 As a author of this presentation I/we own the copyright and confirm the originality of the content. I/we allow Agile testing alliance to use the content for social media marketing, publishing it on ATA Blog or ATA social medial channels(Provided due credit is given to me/us) #ATAGTR2020 As a author of this presentation I/we own the copyright and confirm the originality of the content. I/we allow Agile testing alliance to use the content for social media marketing, publishing it on ATA Blog or ATA social medial channels(Provided due credit is given to me/us) Conclusion  Chatbot as a technology is growing faster than expected and will impact our everyday life in the very near future  Chatbots will be a critical part of business strategy and digital road map for all business across different sectors.  To stay ahead in the race, business need to be ready for this technology shift and provide a better customer experience  Non-Functional testing of Chatbots are clearly a mandate in today’s world  But now it is time to pause and think differently. Instead of working to develop only solutions, why not we embrace this technology to develop leading edge Non Functional Testing Services.  It would be a wise and prudent step to develop new non- functional testing approach by leveraging Chatbots , which will replace the traditional testing approach.
  10. 10. #ATAGTR2020 As a author of this presentation I/we own the copyright and confirm the originality of the content. I/we allow Agile testing alliance to use the content for social media marketing, publishing it on ATA Blog or ATA social medial channels(Provided due credit is given to me/us) #ATAGTR2020 As a author of this presentation I/we own the copyright and confirm the originality of the content. I/we allow Agile testing alliance to use the content for social media marketing, publishing it on ATA Blog or ATA social medial channels(Provided due credit is given to me/us) Thanks

×