Slides used in Agile Testing Conference hosted by KnowledgeHut in Pune, India in March 2017. The slides talk about the Testing Challenge posed by Machine Learning applications and some suggested approaches to point us in the right direction

1. @vinaya1980 @hrishikarekar
TESTING IN THE AGE OF MACHINE LEARNING
March 2016

2. https://www.youtube.com/watch?v=DCgHsxISE0Q
W H E R E A R E
W E H E A D E D
httpsweforum.org/agenda/2017/01/worried-
about-ai-taking-your-job-its-already-happening-in-
japan?utm_content=buffera14af&utm_medium=s
ocial&utm_source=twitter.com&utm_campaign=b
uffer

3. THE WORLD OF ARTIFICIAL INTELLIGENCE
NLP
Neural Networks
MACHINE LEARNING
DEEP LEARNING
…..

4. THE FUTURE IS ALREADY HERE
Google RankBrain
Assistants

5. HOW DOES IT WORK
Supervised
• Labelled data
• Given new
data, predict
outcome
• Classification
Unsupervised
• No labels
• Find hidden
structures
• Clustering
Reinforcement
• Decision
process
• Actions are
rewarded or
punished
• Learns to
optimize
rewards

6. http://scikit-learn.org/
LEARNING FROM DATA

7. SUPERVISED LEARNING - CLASSIFCATION
http://www.mikedeff.in/MLIntro.PNG

8. INTO THE REALM OF PROBABILITIES
Y = f ( x ) Y ≈ f ( x )
What is scrum ?
{
"Prediction": {
"details": {
"Algorithm": "SGD",
"PredictiveModelType": "MULTICLASS"
},
"predictedLabel": "definition",
"predictedScores": {
"advantages": 0.0001860455668065697,
"characteristics": 0.00006915141420904547,
"compare": 0.00017757616296876222,
"definition": 0.9970965385437012,
"disadvantages": 0.0000534967657586094,
}
}
}
Can you tell me about scrum ?
{
"Prediction": {
"details": {
"Algorithm": "SGD",
"PredictiveModelType": "MULTICLASS"
},
"predictedLabel": "definition",
"predictedScores": {
"advantages": 0.01977257989346981,
"characteristics": 0.022757112979888916,
"compare": 0.008386141620576382,
"definition": 0.21092116832733154,
"disadvantages": 0.04002799838781357
}
}
}

9. TOLERANCE LEVELS
Y ≈ f ( x )
Know the probability that is within acceptable limits

10. EVALUATE WITH DIFFERENT MODELS
Evaluate against a set of
algorithms to iterate towards a
model that’s closest
representation and for further
tuning
https://s3.amazonaws.com/MLMastery/MachineLearningAlgorithms.png?__s=h4reg8jqwyg4sz3bzdqf

11. EVALUATION – DATA SET APPROACHES
Random split
• 70% train, 30% test
K-fold cross validation
Split into 3 datasets
• #1 Train on 1 and 2, test on 3
• #2 Train on 2 and 3, test on 1
• #3 Train on 1 and 3, test on 2
Never use the same dataset for training and evaluating

12. ITERATIVE – LEARNING PROCESS
Be prepared to throw the model and start again 

13. MODEL ACCURACY - CONFUSION MATRIX
Strive for better models, not 100% accurate.

14. OVERFITTING

15. MODEL IS AS GOOD AS THE TRAINING DATA
If all of the algorithms perform poorly,
• it maybe worth considering if there is a lack of learning
structure in the data set
• some transformation needed to make the structure more
learnable
• remove unnecessary noise - stop words are typically
removed because they cause unnecessary noise)

16. SUMMARY
Machine Learning applications demand a shift in testing
approach
• Use objective acceptance levels to evaluate the application
• Express test outcomes in statistical terms
• Have a high level understanding of the underlying working of
the application

17. @vinaya1980 @hrishikarekar
THANK YOU