3. • AI is a form of advanced computer science, that learns from data in order to expand
its generalization abilities on narrow tasks, as opposed to regular software
hardcoded instructions
• AI can be subdivided into supervised learning - the bulk of modern applications,
unsupervised learning - grouping for visualization and exploration purpose mainly,
and reinforcement learning - difficult to implement but powerful in some optimization
with actions cases
• The list of tasks AI can solve can broadly be divided into : classification, prediction,
clustering, outlier detection, recommandation, data generation
• The different subdomain of applications can be determined by the data input/output
types : vision, NLP (text&speech), structured classic, robotics
What have we seen last time?
3
5. if color == "green":
return "apple"
elif color == "orange":
return "orange"
else return "banana"
“apple”
“orange”
“banana”
“apple”
5
Can you write a computer program that does that ?
12. • AI is a form of advanced computer science, that learns from data in order to expand
its generalisation abilities on narrow tasks, as opposed to regular software
hardcoded instructions
• AI can be subdivided into supervised learning - the bulk of modern applications,
unsupervised learning - grouping for visualisation and exploration purpose mainly,
and reinforcement learning - difficult to implement but powerful in some optimisation
with actions cases
• The list of tasks AI can solve can broadly be divided into : classification, prediction,
clustering, outlier detection, recommandation, data generation
• The different subdomain of applications can be determined by the data input/output
types : vision, NLP (text&speech), structured classic, robotics
What have we seen last time?
12
13. •Understand the limits of AI and the main biases when it comes to
create intelligent machines in real life
•Lifecycle of an AI application, and how it differs from regular
workflows
•How to detect opportunities / use cases, and evaluate their impact on
the revenue of the company. Cost per task, revenue per task
•Team management, project management (create and deploy) and data
management
Our plan for today - the real world
13
17. Key steps of a machine learning project
Echo / Alexa
1. Collect data
2. Train model
Iterate many times until good enough
3. Deploy model
Get data back
Maintain / update the model
01
03
02
06
04
05
MaintenanceIdentify
DeployData
EvaluateModel
Source : deeplearning.ai
17
18. Key steps of a machine learning project
Self-driving car
1. Collect data image position of other cars
2. Train model
3. Deploy model
Get data back
Maintain model
01
03
02
06
04
05
MaintenanceIdentify
DeployData
EvaluateModel
Source : deeplearning.ai
18
21. Anything you can do with 1 second of
thought, can probably be automated today
21
22. “The toy arrived two days late, so I wasn’t able to give it to
my nephew for his birthday.
Can I return it ?”
“Refund request”
Refund/Shipping/OrderInput text
“Oh sorry to hear that!
I hope you nephew had a good birthday.
Yes, we can help with ...
Complex personalised
empathetic response
Input text
“Yes you can. The refund procedure is ...”
Simple responseInput text
22
23. Diagnose pneumonia on ~ 10.000
images
Diagnose pneumonia from 10
images of a medical textbook
Ask to perform on new type of
data
23
24. Take a (deep) look at your work
Break down your workflow and your business unit
24
26. Baby food ingredient: safe or spoiled?
Patient: ideal medication dosage?
Email: spam or ham?
Recorded phone call to call center: issue topic?
Bottle of wine: will I like it or not?
Steering wheel: left or right?
Photo: which animal?
Game piece: which location on the board?
Start of a sentence: end of that sentence?
Stock: tomorrow’s price?
Transaction: legitimate or fraudulent?
Data center cooling system: warmer or cooler?
Machine: when will it need maintenance?
Inventory: when to restock?
Scene description: pixels in a visual rendering?
Today’s temperature: tomorrow’s temperature?
Auction: how much to bid?
Movie: will you like it or not?
Live lecture: text captions?
Poem: what does it sound like out loud?
Image of an invoice: total amount in dollars?
Service request: waiting time?
Expense report: budget category?
Sound recording: correct text captions?
Song lyrics: language?
Sentence in English: same meaning in Chinese?
Form incorrectly filled out: correct fields?
Clothing item: skirt or blouse or …?
Video: which actors?
Video game: joystick motion?
Toilet user: did they wash their hands?
Idea 1
Ask simple guesswork labelling question
26
28. Idea 2
find the ROI of (cheap) prediction
Level 1: as an optimisation tool
Level 2: as an improvement / help / recommandation
Level 3: as a new feature / product
28
30. Discover Opportunities-
Brainstorming framework
1. Think about automating tasks rather
than jobs!
2. What are the main drivers of
business value?
3. What are the main pain points in
your business ?
4. How much data is needed ? Is my
data clean ? Are we mature in terms
of data ?
30
31. What AI can do
Valuable cases
for your business
AI experts Domain experts
Cross-functional team 31
35. Real life case studies
Fromcorebusinesstolow-hangingfruits
35
35
36. Recommandations
“35 percent of what consumers purchase on Amazon come from product recommendations”
https://www.mckinsey.com/industries/retail/our-insights/how-retailers-can-keep-up-with-consumers
36
36
37. Amazon Go
“Amazon.com Inc. may open up to 3,000 Amazon Go outlets by 2021”
https://www.bloomberg.com/news/articles/2018-09-19/amazon-is-said-to-plan-up-to-3-000-cashierless-stores-by-2021
37
37
44. Optimizing vs satisficing metric
Possible metrics
Cost = accuracy – 0.5 x running time
Or
Maximize accuracy
Subject to running time <= 100 ms
Use case 1 : Cat classifier
Use case 2 :
Detect trigger words for Amazon Alexa Device
Possible metrics
Accuracy
Or
Maximize accuracy
Subject to <= 1 FP for every 24 hours
44
56. Where will you get it?
Then prioritise by availability, accessibility & cost
- existing data sources
- data enrichment (feature engineering)
- data augmentation
- data generation
- manual data labeling
- create new data sources (e.g. sensors)
- Public data, scraping, etc
56
58. If the benefit of the performance
increase outweighs the cost of
acquiring more data, get more
data!
Diminishing returns mean that
more data won’t help
What amount of data ?
58
75. Design with Fairness in mind
! Consider the problem
! Ask experts
! Train the models to account for bias
! Interpret outcomes
! Publish with context
75
76. ⊙ Big data is always better, but not
necessary
⊙ Clean data better than a lot of messy
data
⊙ Small data is almost always enough to
make progress (activate feedback loop)
⊙ If no data, don’t give up, if can be
generated or augmented!
⊙ Design ML model with fairness in mind
Go talk to a ML Engineer to figure it out
Data
Consideratio
ns
76
79. Main steps for model training
1. Select your model family (and your performance metric)
2. Split you dataset into Train/Dev/Test set
3. Train model on training set
4. Take care of overfitting vs underfitting
5. Tuning hyper parameters
6. Select best model
79
87. 6) Selecting best model
For each algorithm (i.e. regularized regression, random forest, etc.):
For each set of hyperparameter values to try:
Perform cross-validation using the training set.
Calculate cross-validated score.
87
88. Checkpoint quizz
! Pick one: better data or fancier algorithms ?
! When should you split your dataset into training and test sets, and why?
! What's the key difference between model parameters and hyperparameters?
! Explain how cross-validation helps you "tune" your models?
88
94. Explainability of the model
● Depending on the machine learning model used, the results could be :
● Very simple to interpret: Like decision trees
● Very difficult to interpret: Like deep-learning neural networks
94
96. Explainability of the model
On a deep-learning neural network, interpretability of weights is difficult.
96
97. Explainability of the model
We could still use more sophisticated technique to partially understand
their predictions. This is an example on logo detection algorithms
Image Grad-cam Image Grad-cam
97
98. Performance of the model
TP
TNFP
FN
YES NO
YES
NO
Predicted
Actual
Confusion Matrix
98
99. Performance of the model
1.
4.3.
2.
YES NO
YES
NO
Predicted
Actual
How confusion matrix can help understand the model performance
Imagine you have a medical problem, do you go see your doctor?
1. If you should and you did, the fee is 25€
2. If you should and you didn’t, it gets worse and you will see a specialist, the fee is
70 €
3. If you shouldn't and you did, you still pay 25€
4. If you shouldn't and you didn’t, you do not pay anything
OK
OK
Loose 25 €
Loose 45 €
99
100. Performance of the model
200
10020
40
YES NO
YES
NO
Predicted
Actual
How confusion matrix can help understand the model performance
Which ML model is better, according to confusion matrices ?
Loose 25 €
Loose 45 €
210
8535
30
YES NO
YES
NO
Predicted
Actual
Loose 25 €
Loose 45 €
Loose 20 * 25 € + 40 * 45 € = 2 300 € Loose 35 * 25 € + 30 * 45 € = 2 225 € 100
101. Accuracy metric
Let us speak in terms of seeing your doctor:
● Accuracy: Over all the choices (see or not your doctor)
you make, how many of them were correct?
!""#$%"& =
() + (+
() + ,+ + ,) + (+
TP
TNFP
FN
YES NO
YES
NO
Predicted
Actual
101
102. Precision & Recall metrics
Let us speak in terms of seeing your doctor:
● Recall: Over all the times you should go see your doctor, how
many times you really went?
!"#$%% =
'(
'( + *+
● Precision: Over all the times you did go see your doctor, how
many of times you really needed to see him?
(,"#-.-/0 =
'(
'( + *(
TP
TNFP
FN
YES NO
YES
NO
Predicted
Actual
102
103. Accuracy VS Precision & Recall
● The accuracy is not used when the problem is not balanced.
● If 99% of your data are just one class
● An accuracy of 99% is just a majority vote
● Precision and recall are more useful in this case since you can focus on each class
individually
103
104. Which ML method is preferred ?
Use Case : Customer Churn
Target action A : phone call to potential churning customer
Target action B : send generous discount to potential churners
Which method is preferred for each target ?
104
113. ・Identifying fraudulent claims so that they can select claims for
deeper manual investigation; they have a business goal of
reducing fraud by 5% this year.
・Predicting weather patterns so that they can advise
customers to protect their vehicles by bringing them inside when
there’s a high chance of storms — thereby reducing vehicle
damage claims by 2%.
・Upselling other insurance products to the customer based on
the products they already have. The goal is to increase the
conversion rate for online upselling by 3%.
Still think vertical – 3 use cases
113