The early 2024 version of my How AI improve testing presentation that I delivery since 5 years ago in different conference, webinars and customer events abroad and in Finland.
It gives some background on AI and talks about AI opportunities: in short test automation getting more fault tolerant, generative AI bringing efficiency to test creation, and advanced test selection with AI.
3. A Nordic powerhouse
for digital solutions
4 400+
Experts
6 countries
SE, NO, FI, DK, DE, PL
613 MEUR
Combined turnover 2022
4 business areas
Solutions, Experience, Connectivity and Insight
Nordic ESG-ambassador
Clear vision to accelerate the UN sustainability agenda
4. The book project ”Dragons Out!”
• Mission
• “Software testing brought to children”
• Book
• Author Kari Kakkonen
• Illustrator Adrienn Széll
• Text and illustration rights Dragons Out Oy
• In Finnish, English, Polish, French and growing
• For ages of 10-99
• Free “Dragon lesson in software testing”
presentation under Creative Commons –license
• Translated to 20 languages!
• More info: www.dragonsout.com
• Also other books coming out
4
5. ISTQB GLOBAL PRESENCE
• Number of exams
administered: over 1,2 million
• Number of certifications
issued: 914,000
• In 130 countries
6. TMMi for test improvement in all kinds of testing,
including agile and DevOps
7. • Artificial Intelligence (AI) in short
• Case of NASA code quality metrics
using AI
• Other AI and testing developments
Agenda
Copyright Knowit Solutions Oy 2021 7
9. Quality management as top use-case for AI
Source: AI software platform adoption survey, IDC, February 2019 [Percent of respondents: N=505]
Copyright Knowit Solutions Oy 2021 9
10. • When talking about AI, it is
important to ask if we are
talking about modelling
humans or how to work in
ideal way.
• Machine learning is a sub-
category of AI, which
considers algorithms that
enable AI to learn ideal way to
work.
Artificial intelligence == machine learning?
Figure: Example of concepts related to AI.
Copyright Knowit Solutions Oy 2021 10
11. Three methods of machine learning
Machine learning methods
Supervised Learning
• The aim is to learn
mapping from input to
output.
• Output is known by
supervisor.
Unsupervised Learning
• The aim is to find
”something interesting”.
• Output is not known
(known also as density
estimation in statistics).
Reinforcement Learning
• Seemingly good
solutions are
encouraged to
develop further.
AI literature review[1]
1 Reagan, A., et al., The emotional arcs of stories are dominated by six basic shapes, EPJ Data Science (2016).
Copyright Knowit Solutions Oy 2021 11
12. No one tool for all
Copyright Knowit Solutions Oy 2021
Performance
Amount of data, time, and computing power
• Linear discriminant
analysis
• Logistic regression
• Medium-sized ANNs
• Genetic algorithms
• Extreme learning
machines
• Convolution neural
networks
• Generative neural
networks
• Deep belief networks
• Recurrent neural
networks
• Simple neural networks
12
13. Planning
descriptions
Code Unit testing
Integration testing
System testing
Acceptance
testing
Definition
descriptions
Requirements
• Static methods
• Reviews
• Dynamic methods
• Test runs
Remember there are different kinds of testing
Copyright Knowit Solutions Oy 2021 13
14. AI in software testing
Copyright Knowit Solutions Oy 2021
• In static testing, several machine learning
algorithms can be used to measure code
quality.[1] NASA’s code review.
• In dynamic testing, model-based regression
tests can be replaced by algorithms that
generate test flows on-the-fly.[2] Chaos
monkey of Netflix.
• In maintenance testing, machine learning can
be used to plan more realistic statistical
models of how application is actually used.
Figure: Margaret Hamilton standing next
to source code of guidance system of
Apollo spacecraft.
www.nasa.gov/
14
17. • Detecting:
• abnormal behaviour
• new kind of customers
• etc.
Anomaly detection
Copyright Knowit Solutions Oy 2021
Figure: Example of time series
17
20. • Goal is to understand technical quality and test coverage of modules.
Also, goal is to guide and focus testing:
• Focusing technical debt reduction
• Found defects per module
• Guide for test coverage in automating system level testing.
AI code quality metrics
Copyright Knowit Solutions Oy 2021
Requires that there is link from components (version control) and tests (test management) to feature descriptions and from reported defects to modules (observation control)
Module Developer Benchmark
(order.numb.)
Relative
risk
Number of
reported
bugs
Module_1 Jaakko 50 58 4
Module_2 Jaakko 4 5 1
Module_3 Mikko 180 95 7
SCRUM TEAM
20
21. • Goal is to provide overview:
• How fast the development is advancing
• Maturity for release
• Quality and technical debt of the software
• Defect probabilities
• Risks related to modifications of the
software.
AI code quality metrics
Copyright Knowit Solutions Oy 2021
Interpretation:
• Module 5 has high defect probability and is
related to five most important features of
the application. This results into high risk in
release at this moment.
• Defect probability of module 10 has fallen
to acceptable level.
Start-up
phase
Development
phase
Maturing
phase
MANAGEMENT
21
22. Stages of execution
4. DEPLOYMENT AND UTILIZATION OF THE ALGORITHM
▪ Monitoring and analysing results
2. DEFINING QUALITY METRICS
▪ Defining quality and defect metrics
▪ Building the metrics
3. DEFINING HOW RESULTS ARE PRESENTED
▪ Defining who will use results and how
▪ Building data visualization
1. DEFINING DATA SOURCES
▪ From where data is collected
▪ How will the data be collected
Copyright Knowit Solutions Oy 2021 22
Most
effort
24. • Several code quality metrics are currently
being used. The most popular ones have
been introduced in 1970.[1,2]
1 T.J. McCabe, A Complexity Measure, IEEE Transactions on Software Engineering (1976) SE-2, 4, 308.
2 M.H. Halstead, Elements of Software Science, Elsevier (1977).
3 N.E. Fenton, S.L. Pfleeger, Software Metrics: A Rigorous & Practical Approach, International Thompson Press (1997).
4 S.J. Sayyad, T.J. Menzies, The PROMISE Repository of Software Engineering Databases (2005).
• Goal is to provide feedback:
1. for developers about code quality.
2. for testers about needed test coverage.
3. for management about the development
project
• Although traditional metrics are extensively
used, benefit of using them has been
questioned.[3]
0
50
100
150
200
250
300
350
400
450
0 20 40 60 80 100 120
Number
of
lines.
Cyclomatic complexity
Scattering of defects in NASA CM1 project
OK Defected
Figure: NASA Metrics Data Program, CM1 Project[4]
In real applications defected modules do not separate from
non-defected by standard metrics.
AI code quality metrics
Copyright Knowit Solutions Oy 2021 24
25. 1
2
𝑤 2
+ 𝐶
𝑖=1
𝑁
ξ𝑖
1-𝑦𝑖 𝑤, 𝑓(𝑥𝑖) + 𝑏 ≤ ξ𝑖
Minimize
Subject to with all values of i.
Quality metric 1
Quality
metric
2
• Machine learning approach was tested for NASA’s CM1
software repository[1].
• Support vector type of machine learning algorithm was
developed for this demonstration.
• This task can be solved by supervised machine learning
methods because both input (quality metrics) and output
(defects) is known.
• Goal is to find such hyperplane in multidimensional data
space that separate defected modules from non-
defected.
• Mathematically, problem can be described with algorithm:
AI-guided software testing with NASA data
Copyright Knowit Solutions Oy 2021 25
26. 0,0
0,2
0,4
0,6
0,8
Tested on real data against NASA CM1 project[1]
Performance
(F)
Training set: 1595
Test set: 400
𝐹 =
2 ∗ 𝑎𝑐𝑐𝑢𝑟𝑎𝑐𝑦 ∗ 𝑟𝑒𝑐𝑎𝑙𝑙
𝑎𝑐𝑐𝑢𝑟𝑎𝑐𝑦 + 𝑟𝑒𝑐𝑎𝑙𝑙
AI-guided software testing with NASA data
Copyright Knowit Solutions Oy 2021 26
28. • Enhanced automation
• Test automation
• Robotic Process Automation -> Intelligent automation
• AI-powered test automation
• Efficiency by Generative AI
• Test script creation
• Manual test case creation
• Test date creation
• Enhanced test selection
• Risk analysis
• Test case selection
• Code analysis
General direction at businesses
on AI and testing
Copyright Knowit Solutions Oy 2021 28
29. • Start from screenshots
• Detect layout issues
• Cross-browser testing for
differences
• Ignore invisible rendering, size,
position differences
• Case: Applitools
• https://applitools.com / Visual AI
• Also works as an AI library for test
automation frameworks like Robot
Framework
AI-powered comparison of visual elements
Copyright Knowit Solutions Oy 2021 29
30. • AI-based test generation
• Checking functional differences
• Checking visual differences
• Automatically learning the test
automation
• For: regression testing after changes
• Case: retest
• https://retest.de/ai-based-test-
generation/
AI with regression testing
Copyright Knowit Solutions Oy 2021 30
31. • “World’s first”
• Go automatically through mobile apps
• Use general test cases
• Reinforcement learning
• Case: test.ai
• https://www.test.ai/
AI-powered mobile test automation platform
Copyright Knowit Solutions Oy 2021 31
32. • Default tests to web sites
• Teach your own tests
• Running on containers in virtual
machines
• Use for regression testing, cross-
browser testing
• Case: Mabl
• https://www.mabl.com/
Auto-healing automated testing of web
Copyright Knowit Solutions Oy 2021 32
33. • Build test through interface backed by ML
• Plain English
• New life for “record-playback”
• ML algorithms maintain the tests
• Case: Functionize
• https://www.functionize.com/
Intelligent testing
Copyright Knowit Solutions Oy 2021 33
34. • Automatically scope and generate optimized test cases
• Generate test scripts for automated execution from models and
manual tests
• Display the test coverage requirements
• Integrate with existing tools
• Case: Conformiq
• https://www.conformiq.com/
AI-enabled test creation from models
Copyright Knowit Solutions Oy 2021 34
35. • Enhance core test automation
• Develop automated test scripts
• Regression testing
• Build test environments rapidly
• Service focus
• Case: Mammoth
• https://www.mammoth-ai.com/
• https://www.mammoth-ai.com/artificial-intelligence/
Enhancing test automation with AI
Copyright Knowit Solutions Oy 2021 35
36. • Support test automation engine data selection with AI
• Can be used for testing with large quantities of data
• Can be used automating production as Robotic Process Automation
(RPA)
• Cases:
• Robot Framework test automation framework
• Aito AI tool
• https://robotframework.org/
• https://aito.ai/
• https://aito.ai/blog/machine-learning-in-robot-framework/
Automating tests or production with lots of
repetitive data
Copyright Knowit Solutions Oy 2021 36
37. • Supports test automation object recognition
• Helps with typical GUI recognition problems for test automation
• Can be used as a support library for existing test automation
• Cases:
• ImageHorizonLibrary is a cross-platform library for Robot Framework.
• Eficode, Oulu University and Business Finland
• https://www.eficode.com/projects/testomat
Image recognition with AI
for open source test automation
Copyright Knowit Solutions Oy 2021 37
38. • Python-editor PyCharm has
Refact.ai library that uses ChatGPT
• Understands Python and Robot
Framework test automation setup
• Refact.ai is labelled as an AI coding
assistant software
• https://refact.ai/
Python enhanced with ChatGPT
Copyright Knowit Solutions Oy 2021 38
39. • Assist test automation script
creation
• Suggest what the user might want
• What kind of code
• What kind of test script
• User reviews and approves
suggestions
• Based on Generative AI
• https://github.com/features/copilot
Github Copilot
Copyright Knowit Solutions Oy 2021 39
40. • Python-editor PyCharm can use
Dev Tools AI library
• Asks which element to click on the
screen and then keeps it running
• Suggests how to fix
semiautomatically a changed test
script that doesn’t pass any more
• Reduces the need for test script
maintenance
• Supports e.g. Cypress, Selenium,
Appium, Playwright
• https://www.dev-tools.ai/
AI library helps pass Python test scripts
Copyright Knowit Solutions Oy 2021 40
41. • Assist extending existing test
automation sets
• With e.g. Robot Framework,
Playwright, Selenium, Cypress
• Use tests as self-driving test-
bots
• Exploring web apps
• Regression testing
• Autonomous testing
• https://wopee.io/
Test automation assisstants and bots
Copyright Knowit Solutions Oy 2021 41
42. • Derive tests from specifications
• Create tests based on user input
• Enabled by good prompt
engineering
• Challenge: repeated teaching
• User reviews AI proposals and uses
in their tests
• https://chat.openai.com/ (ChatGPT)
• https://bard.google.com/
• https://copilot.microsoft.com/ (Bing)
Generative AI helps in test creation
Copyright Knowit Solutions Oy 2021 42
43. • Go through functionalities
• Identify risky areas
• Focus testing
• Case: Eggplant
• https://www.eggplantsoftwar
e.com/
Predict quality issues
Copyright Knowit Solutions Oy 2021 43
44. • Analyze risk in commits to the code
• Select the tests that test the risky areas
• Reduce number of test cases needed to run
• Case: Appsurify
• https://appsurify.com/
Select most suitable test cases
Copyright Knowit Solutions Oy 2021 44
45. • Power of open source
developers
• Machine learning
algorithms learn from the
community automatically
• Analyse code and
propose improvements
• Case: DeepCode
• https://www.deepcode.ai/
Code analysis using AI
Copyright Knowit Solutions Oy 2021 45
46. • Trainings and certifications
• https://www.istqb.org/certification-path-root/ai-testing.html
• Standards
• ISO
• AI/ML standard https://www.iso.org/standard/74438.html
• Testing of AI-based systems https://www.iso.org/standard/79016.html
• IEEE
• What is AI Software Testing https://ieeexplore.ieee.org/document/8705808
• AI testing perspectives https://ieeexplore.ieee.org/document/9514942
• Application of AI in Testing https://ieeexplore.ieee.org/document/9676244
Promoting and teaching AI and testing
Copyright Knowit Solutions Oy 2021 46
47. • AI in Testing: Impact, Problems, Challenges and Prospect
https://www.researchgate.net/publication/357876318_Artificial_Inte
lligence_in_Software_Testing_Impact_Problems_Challenges_and_Pro
spect
• Utilizing AI in Software Testing
https://www.theseus.fi/handle/10024/263992
• AI Applied to Software Testing
https://dl.acm.org/doi/10.1145/3616372
• AI Applied to Testing: A Literature Review
https://ieeexplore.ieee.org/abstract/document/9141124
• ChatGPT helps testing https://www.linkedin.com/pulse/gpt-4-sdlcs-
secret-weapon-reinventing-testing-phase-andy-abbott/
Some more research notes
Copyright Knowit Solutions Oy 2021 47
48. • AI-enabled testing is already a reality
• Many companies are enhancing their solutions with AI
• New companies are set up around AI
• AI can provide simplicity to complex software development projects.
• With GenerativeAI, manual and automated test script creation is
more productive
• With AI, testing activities can be focused on high risk areas.
• With AI, test automation becomes more autonomous
• Testers are freed to created new tests
Conclusion
Copyright Knowit Solutions Oy 2021 48