AI in SE: A 25-year Journey

AI in SE: A 25-year Journey
Lionel Briand
SEI @ ASE 2019

1991
Predicting faults in flight dynamics
software
2
Colleagues and friends:
“Machine learning?
Why would you want to apply this?
This is not serious.”

Acknowledgments
• Shiva Nejati
• Fabrizio Pastore
• Mehrdad Sabetzadeh
• Chetan Arora
• Sallam Abualhaija
• Raja Ben Abdessalem
• Chunhui Wang
• Seung Shin
3
• Donghwan Shin
• Reza Matinnejad
• Annibale Panichella
• Hazem Fahmy
• Erik Arisholm
• Stefano Di Alesio
• Tao Yue
• Shaukat Ali
• Ghanem Soltana
• Nadia Alshahwan
• Khouloud Gaaloul
• Yvan Labiche
• Marwa Shousha
• Amin Sleimi
• Nicolas Sannier
• Marcello Cecci
• …

Objectives
• Report on many years of experience about leveraging
AI on industrial research SE projects.
• Personal experience, not a survey.
• Partial presentation (very much so).
• Focus on real problems, real solutions, in real
contexts.
• Example projects and lessons learned.
4

5
Machine
Learning
Natural
Language
Processing
Logic-based
Reasoning
AI in SE
Search
ExhaustiveMetaheuristic
CP, SAT,
SMT
Evolutionary
Computing

< ~2000
Making software
development predictable
6

Context
• Software development data repositories were few.
• Data available to researchers was scarce and hard to use.
• Focus on resource and defect prediction.
• Hundreds of research papers.
7

Evolving Telecom System
8
Class complexity
- Class size
- Coupling
- Cohesion
- …
For each type of CR involving this class
- Number of CRs
- Lines of code added and deleted in this class
- Number of CPs involving this class
- Total number of files changed in CRs
- Total number of tests failed in CRs
- Total number of developers involved in CRs
- Total number of past CRs of the developers
1.0
0.0Not fault-prone
Highly fault-prone
Neural Network
C4.5
Logistic regression, etc
Class change and fault history
For each type of CR involving this class
for the past three releases:
- number of CRs (n-1, n-2, n-3)
Arisholm et al. Journal of Systems and Software, 2010

Cost-Effectiveness
10
Process
Change, fault, human,
product
Data
Learning
fault patterns
Fault
Patterns
Focus V&VRanking,
classification
Additional V&V

Results
11
• Assumption: the number of lines of
code is proportional to the
verification effort of the predicted
fault-prone classes
• Baseline: a random selection of classes to
test would require the testing of 60% of
the code to detect a maximum of 60% of
the faults.
• Model: Using our model we bring this
percentage down to less than 20% of the
code, thus potentially requiring only 20/60
= 1/3 of the verification effort.
• Optimal: If we somehow knew where the
faults were located, we could potentially
test only 5% of the code to detect a
maximum of 100% of the faults.

Actionable?
12
Process
Change, fault, human,
product
Data
Learning
fault patterns
Fault
Patterns
Focus V&VRanking,
classification
Additional V&V
Map predictions to V&V practices is not easy

ML for Prediction: Benefits
• Some machine learning techniques, such as random forests,
tend to be more accurate than classical techniques, e.g.,
based on regression.
• More flexible and robust (less assumptions), less prone to
overfitting, etc.
• As larger amounts of data became increasingly available, so
was their application.
• Mining Software Repositories
13
T. Menzies, “Machine Learning: A Tutorial”

ML for Prediction: Challenges
• Lots of papers on how to build prediction models.
• Very few papers on how to effectively use such prediction
models, their benefits, etc.
• How to use them in a cost-effective way is far from obvious.
• Cost-benefit of building and maintaining a prediction system.
• We are predicting moving targets as development practices
evolve quickly.
14

> ~2000
The rise of Search-Based SE
(SBSE)
15

Why SBSE?
• After decades of research, no scalable, practical solutions for
many automation problems.
• Many automation problems can be re-expressed as search
problems.
• Stochastic optimization, Meta-heuristic search.
• Increasing realization that Search-Based Software
Engineering has a much wider potential and is a research
topic in itself.
16
Harman and Jones, “Search-Based Software Engineering”, 2001

~1995
The first time I read about genetic
algorithms, meta-heuristic search …
17
My first reaction:
“You must be kidding me”

SBSE Applications
• Many applications: Requirements prioritization, refactoring,
test automation, program repair, etc.
• My first SBSE paper: “Using genetic algorithms and coupling
measures to devise optimal integration test orders.” SEKE’02
• Many articles since then …
18

Advanced Driver Assistance
Systems (ADAS)
19
Automated Emergency Braking (AEB)
Pedestrian Protection (PP)
Lane Departure Warning (LDW)
Traffic Sign Recognition (TSR)

Advanced Driver Assistance
Systems (ADAS)
Decisions are made over time based on sensor data
20
Sensors
Controller
Actuators Decision
Sensors
/Camera
Environment
ADAS

Automotive Environment
• Highly varied environments, e.g., road topology, weather, building and
pedestrians …
• Huge number of possible scenarios, e.g., determined by trajectories of
pedestrians and cars
• ADAS play an increasingly critical role in modern vehicles
• Systems must comply with functional safety standards, e.g., ISO 26262
• A challenge for testing
21

Project
• Developing an automated testing technique
for ADAS
22
• To help engineers efficiently and
effectively explore the complex scenario
space of ADAS
• To identify critical, failure-revealing test
scenarios
• Characterization of input conditions that
lead to most critical scenarios, e.g.,
safety violations Abdessalem et al., ASE 2016, ASE 2018

23
Automated Emergency Braking
System (AEB)
23
“Brake-request”
when braking is needed
to avoid collisions
Decision making
Vision
(Camera)
Sensor
Brake
Controller
Objects’
position/speed

Example Critical Situation
• “AEB properly detects a pedestrian in front of the car with a
high degree of certainty and applies braking, but an accident
still happens where the car hits the pedestrian with a
relatively high speed”
24

Testing ADAS
25
A simulator based on
physical/mathematical models
Time-consuming
Expensive
On-road testing
Simulation-based (model) testing
Unsafe

Testing via Physics-based
Simulation
26
ADAS
(SUT)
Simulator (Matlab/Simulink)
Model
(Matlab/Simulink)
▪ Physical plant (vehicle / sensors / actuators)
▪ Other cars
▪ Pedestrians
▪ Environment (weather / roads / traﬃc signs)
Test input
Test output
time-stamped output

AEB Domain Model
27
- intensity: Real
SceneLight
Dynamic
Object
1
- weatherType:
Condition
Weather
- fog
- rain
- snow
- normal
«enumeration»
Condition
- ﬁeld of view:
Real
Camera
Sensor
RoadSide
Object
- roadType: RT
Road
1 - curved
- straight
- ramped
«enumeration»
RT
- v0: Real
Vehicle
- x0: Real
- y0: Real
- θ: Real
- v0: Real
Pedestrian
- x: Real
- y: Real
Position
1
*
1
*
1
1
- state: Boolean
Collision
Parked
Cars
Trees
- simulationTime:
Real
- timeStep: Real
Test Scenario
AEB
- certainty: Real
Detection
1
1
11
1
1
1
1
«positioned»
«uses»
1 1
- AWA
Output
Trajectory
Environment inputs
Mobile object inputs
Outputs

ADAS Testing Challenges
• Test input space is multidimensional, large, and complex.
• Explaining failures and fault localization are difficult.
• Execution of physics-based simulation models is
computationally expensive.
28

Our Approach
• We use decision tree classification models.
• We use multi-objective search algorithm (NSGAII).
• Objective Functions:
• Each search iteration calls simulation to compute objective
functions.
29
1. Minimum distance between the pedestrian and the
field of view
2. The car speed at the time of collision
3. The probability that the object detected is a pedestrian

Multiple Objectives: Pareto Front
30
Individual A Pareto
dominates individual B if
A is at least as good as B
in every objective
and better than B in at
least one objective.
Dominated by x
F1
F2
Pareto front
x
• A multi-objective optimization algorithm (e.g., NSGA II)
must:
• Guide the search towards the global Pareto-Optimal
front.
• Maintain solution diversity in the Pareto-Optimal front.

Decision Trees
31
Partition the input space into homogeneous regions
All points
Count 1200
“non-critical” 79%
“critical” 21%
“critical” 41%
Count 564 Count 636
“critical” 2%
Count 412
“critical” 51%
Count 152
“critical” 16%
Count 230 Count 182
vp
0 >= 7.2km/h vp
0 < 7.2km/h
✓p
0 < 218.6 ✓p
0 >= 218.6
RoadTopology(CR = 5,
Straight, RH = [4 12](m))
RoadTopology
(CR = [10 40](m))
“critical” 69%
“critical” 28%

Genetic Evolution Guided by
Classification
32
Initial input
Fitness
computation
Classification
Selection
Breeding

Search Guided by Classification
33
Test input generation (NSGA II)
Evaluating test inputs
Build a classification tree
Select/generate tests in the fittest regions
Apply genetic operators
Input data ranges/dependencies + Simulator + Fitness functions
(candidate)
test inputs
Simulate every (candidate) test
Compute fitness functions
Fitness
values
Test cases revealing worst case system behaviors +
A characterization of critical input regions

Generated Decision Trees
34
GoodnessOfFit
RegionSize
1 5 642 3
0.40
0.50
0.60
0.70
tree generations
(b)
0.80
71 5 642 3
0.00
0.05
0.10
0.15
tree generations
(a)
0.20
7
GoodnessOfFit-crt
1 5 642 3
0.30
0.50
0.70
tree generations
(c)
0.90
7
The generated critical regions consistently become smaller, more
homogeneous and more precise over successive tree generations of
NSGAII-DT

Usefulness
• The characterizations (decision trees) of the different
critical regions can help with:
(1) Debugging the system model
(2) Identifying possible hardware changes to increase
ADAS safety
(3) Providing proper warnings to drivers
35

Integration of Autonomous
Features
36
Automated Emergency Braking (AEB) Traffic Sign Recognition (TSR)
Sensor/
Camera
Data
Autonomous
Feature
Actuator
Command
- Steering
- Acceleration
- Braking

37
…
…
Actuator Commands:
- Steering
- Acceleration
- BrakingConflicts
Undesired Feature Interactions
Sensor/
Camera
Data
Autonomous
Feature
Actuator
Command
Sensor/
Camera
Data
Autonomous
Feature
Actuator
Command
Sensor/
Camera
Data
Autonomous
Feature
Actuator
Command

Problem Statement
38
How to automatically detect undesired feature
interactions in self-driving systems at early
stages of development?

Sensor/
Camera
Data
Autonomous
Feature
Actuator
Command
Integration Component
39
…
…
Integration
Component
Actuator
Command
Software Under Test (SUT)
Executable
Function Models
(Matlab/Simulink)
Sensor/
Camera
Data
Autonomous
Feature
Actuator
Command
Sensor/
Camera
Data
Autonomous
Feature
Actuator
Command

Testing Function Models
40
SUT
Physics-based
Simulators
sensors/
cameras
(ego) car
other carsactuators
pedestrians
environment (road, weather, etc)
sensor/
camera data
Actuator
commands
Time Stamped Vectors
…
…
Integration "
Component
Sensor/
Camera
Data
Autonomous
Feature
Actuator
Command
Sensor/
Camera
Data
Autonomous
Feature
Actuator
Command
Sensor/
Camera
Data
Autonomous
Feature
Actuator
Command

Search-Based Solution
• We do not know a priori which safety requirements may be violated.
Neither do we know in which branches of IntC the violations may be
detected.
• Therefore, we search for any violation of system safety requirements that
may arise when exercising any branch of IntC.
• A combination of three test objectives:
• Coverage-based
• Failure-based
• Unsafe Overriding
41
Abdessalem et al., ICSE 2018

Search Algorithm
• Optimal test suite covers all search objectives, i.e., for all IntC branches
and all safety requirements
• The number of test objectives is large:
# of requirements × # of branches
• Not a Pareto front optimization problem
• Objectives compete with each others, e.g., cannot have the car
violating the speed limit after hitting the leading car in one test case
42

MOSA: Many-Objective Search-
based Test Generation
43
Objective 1
Objective 2
Not all (non-dominated) solutions
are optimal for the purpose of testing
These points are
better than others
Panichella et. al.
ICST 2015

SBSE: Benefits
• Effective automation targeting a wide set of problems.
• Potentially high scalability: No exhaustive search.
• Can be effective under certain conditions: Search landscape,
fitness computation, etc.
• Can be effectively parallelized.
• More versatile than mathematical optimization techniques.
44

SBSE: Challenges
• Validation is experimental and computationally expensive.
• Many problems require dedicated, tailored search algorithms, e.g.,
many-objective search in testing.
• Devising the right search algorithm for a given problem requires
expertise and experiments.
• Devising the right fitness functions is often a trade-off and is a trial
and error process.
• Computationally expensive fitness functions, e.g., simulations.
45

> ~2004
Increasingly Powerful
Natural Language
Processing
46

NLP
• Process and analyze large amounts of natural language data.
• Rule-based versus statistical NLP (based on machine
learning).
• Preprocessing: Tokenizer, sentence splitter, POS tagger.
• Parsing: Constituency, dependency, semantic.
• NLP has made huge leaps forward.
47
Arnaoudova et al., “The Use of Text Retrieval and Natural Language Processing in Software Engineering”, ICSE’15

Why NLP?
• Significant documentation of many kinds in natural
language …
• Example: Requirements
• are prominent throughout industry sectors, even safety-
critical ones,
• are not fading away any time soon.
48

Why NLP?
• Check well-formedness of NL artifacts
• Extract useful information from NL artifacts
• Check consistency and completeness of NL artifacts.
• Understand relationship and dependencies between NL
artifacts
49

Experiences in Requirements
Engineering
• Conformance of requirements with templates. (Arora et al.)
• Extraction of domain knowledge/model from requirements. (Arora et al.)
• Impact analysis of requirements changes (Arora et al., Nejati et al.)
• Identification and demarcation of requirements in large documents.
(Abualhaija et al.)
• Extraction of information from legal texts for devising legal requirements.
(Sleimi et al.)
• Requirements-driven system testing. (Wang et al.)
50

• In many sectors, traceability between requirements and test
cases is required by standards, customers, certifiers ...
• Requirements change, and therefore test cases as well.
• Huge traceability matrices are built and maintained manually.
• Academic work on automatically matching requirements and
test cases is not close to be sufficiently accurate or practical.
Traceability
51

Context
Automotive Embedded Systems
52
• Small but safety critical systems
• Traceability from requirements to system test cases
(ISO 26262)
• Requirements act as a contract
• Many requirements changes, leading to negotiations

Problem
Automatically verify the compliance of
software systems with their functional
requirements in a cost-effective way
53

Objective
Support the Generation of
System Test Cases from
Requirements in Natural Language
Traceability is a by-product
54

Textual descriptions are often ambiguous,
Incomplete, and not analyzable automatically
Problem
55

Stick to natural language but …
Compromise?
Restrict its usage so as to make it amenable
to NLP for system testing purposes
Find the right balance
56

• Restricted Use Case Modeling (RUCM)
• Experiments shows it yields better use cases
Yue et al. ACM TOSEM, 2013
Restricted Use Case
Specifications
• More analyzable with NLP
• Use Case Modeling is widely used
• Compliance is tool-supported (NLP)
57

RUCM Specifications Example
Precondition: The system has been initialized
Basic Flow
1. The SeatSensor SENDS the weight TO the system.
2. INCLUDE USE CASE Self Diagnosis.
3. The system VALIDATES THAT no error has been detected.
4. The system VALIDATES THAT the weight is above 20 Kg.
5. The system sets the occupancy status to adult.
6. The system SENDS the occupancy status TO AirbagControlUnit.
INPUT STEP
INCLUDE STEP
CONDITIONAL STEP
INTERNAL STEP
OUTPUT STEP
58

RUCM Specifications Example
Basic Flow
INPUT STEP
INCLUDE STEP
CONDITIONAL STEP
INTERNAL STEP
OUTPUT STEP
Alternative Flow
RFS 4.
1. IF the weight is above 1 Kg THEN
2. The system sets the occupancy status to child.
3. …
4. RESUME STEP 6. 59

Basic Flow
Alternative Flow
RFS 4.
3. …
4. RESUME STEP 6.
NLP for information extraction
DOMAIN ENTITY
60

Basic Flow
Alternative Flow
RFS 4.
3. …
4. RESUME STEP 6.
CONSTRAINT
NLP for information extraction
61

UseCaseStart
Input
Condition
Condition
Output
Exit
Condition
Internal
Internal
Include
"no error has been detected"
"the weight is above 20 Kg"
"the weight is above 1 Kg"
Weight
DomainEntity
Test Model

Use Case
Specifications
Executable Test Cases
Constraints capturing
the meaning of
conditions
Domain
Model
UMTG
Error.allInstances()
->forAll( i | i.isDetected = false)
Automated Generation of System Test Cases
for Embedded Systems from Requirements in NL
https://sntsvv.github.io/UMTG/
(Wang et al.)

Domain Model:
Formalizing Conditions
Manually written OCL constraint:
“The system VALIDATES THAT no error has been detected.”
Error.allInstances()->forAll( i | i.isDetected = false)

UseCaseStart
Input
Condition
Condition
Output
Exit
Condition
Internal
Internal
Include INCLUDE USE CASE Self Diagnosis.
IF the weight is above 1 Kg THEN
The SeatSensor SENDS the weight TO the system.
The system sets the occupancy status to adult.
The system SENDS the occupant class TO AirbagControlUnit.
The system VALIDATES THAT no error has been detected.
The system sets the occupancy status to child.
The system VALIDATES THAT the weight is above 20 Kg.
Precondition: The system has been initialized.
OCL
OCL
OCL
OCL
System.allInstances()->forAll( s | s.initialized = true )
AND System.allInstances()->forAll( s | s.initialized = true )
AND Error.allInstances()->forAll( e | e.isDetected = false)
AND System.allInstances()
->forAll( s | s.occupancyStatus = Occupancy::Adult )
Path condition:
65

UseCaseStart
Input
Condition
Condition
Output
Exit
Condition
Internal
Internal
Include INCLUDE USE CASE Self Diagnosis.
IF the weight is above 1 Kg THEN
The SeatSensor SENDS the weight TO the system.
The system sets the occupancy status to adult.
The system SENDS the occupant class TO AirbagControlUnit.
The system VALIDATES THAT no error has been detected.
The system sets the occupancy status to child.
The system VALIDATES THAT the weight is above 20 Kg.
Precondition: The system has been initialized.
OCL
OCL
OCL
OCL
System.allInstances()->forAll( s | s.initialized = true )
AND System.allInstances()->forAll( s | s.initialized = true )
AND Error.allInstances()->forAll( e | e.isDetected = false)
AND System.allInstances()
->forAll( s | s.occupancyStatus = Occupancy::Adult )
Path condition:
Constraint
Solving
(PLEDGE)
Test inputs:
https://sites.google.com/view/hybridoclsolver/
Soltana et al.

Challenge
Typically dozens of constraints
Engineers need help in defining constraints
67

Automatically Derive Formal Constraints
“The system VALIDATES THAT
no error has been detected.”
Error.allInstances()->forAll( i | i.isDetected = false)
OCLgen Based on NLP
68
(Wang et al.)

OCLgen Solution
2. match words in the sentence with concepts in the domain model
BodySense.allInstances()
->forAll( i | i.occupancyStatus = Occupancy::Adult)
3. generate the OCL constraint using a verb-specific transformation rule
“The system sets the occupancy status to adult.”
actor affected by the verb final state
1. determine the role of words in a sentence

Semantic Role Labeling (SRL)
A1: actor that is affected
by the activity described
in a sentence
A0: actor that performs
an activity
“The system sets the occupancy status to adult.”
A0 A1 A2
A2: final state
verb
70

Semantic Role Labeling (SRL)
“The system resets the counter.”
A0 A1verb
“The counter was reset by the system.”
“The counter has been reset.”
A1 A0verb
Determine the role of words independently from passive voices,
transitive/intransitive forms
A1 verb
Enable pattern-based text transformations:
System.allInstances()->forAll( i | i.counter = 0 )
<A1> = 0

NLP: Challenges
• Variation across domain practices and documents.
• Inherent ambiguity and inconsistency of natural language.
• Relevant data is usually spread across artifacts.
• Designing the right NLP pipeline.
• NLP components are not fully accurate.
• Human in the loop.
72

Multidisciplinary Approaches
• Single-technology approaches rarely work in practice
• Meta-heuristic search
• Machine learning
• Solvers, e.g., CP, SMT
• Statistical approaches, e.g., sensitivity analysis
• System and environment modeling and simulation
74

Search+CP Example
75
Drivers
(Software-Hardware Interface)
Control Modules
Alarm Devices
(Hardware)
Multicore Architecture
Real-Time Operating System
System monitors gas leaks and fire in
oil extraction platforms

RTES: Concurrent Tasks
76
Tasks can trigger other tasks, and
can share computational resources
with other tasks
Each task has a deadline (i.e., latest
finishing time) w.r.t. its arrival time
Some task properties depend on
the environment, some are design
choices
!"
#$"%
&'"%
!(
#$(%
&'(%
)
!%
#$%%
&'%%
*("
#$"(
#$%(
&'"(
#$((
)

Problem Definition
• Schedulability analysis encompasses techniques that try to
determine whether (critical) tasks are schedulable based on
design assumptions, i.e., meet their deadlines.
• Stress testing runs carefully selected test cases that have a
high probability of leading to deadline misses.
• Stress testing is complementary to schedulability analysis.
• Testing is typically expensive, e.g., hardware in the loop.
• Finding stress test cases is difficult (with complex scheduling
algorithms).
77

Finding Stress Test Cases is Hard
78
0
1
2
3
4
5
6
7
8
9
!", !#, !$ arrive at %&", %&#, %&$ and
must finish before '(", '(#, '($
!# can miss its deadline '(#
depending on when %&$ occurs!
0
1
2
3
4
5
6
7
8
9
!$
%&$
'($
!#
%&#
'(#
!"
%&"
'("
) = #
&+,--.+
!$
%&$
'($
!#
%&#
'(#
!"
%&"
'("
) = #
&+,--.+
A sequence of arrival times which is likely to
violate a task deadline characterizes a stress
test case
periodic triggered aperiodic periodic triggered aperiodic

Stress Testing
79
IEC 61508 deems stress testing as
highly recommended for SIL 3-4
Stress Testing: “Testing in which a system is subjected to […]
harsh inputs […] with the intention of breaking it”
— Boris Beizer
Arrival times for aperiodic tasks Worst-case scenarios
wrt. missing deadlines

Challenges and Solutions
• Ranges for arrival times form a very large input space
• Task interdependencies and properties constrain what
parts of the space are feasible
• Solution: We re-expressed the problem as a constraint
optimization problem and used a combination of constraint
programming (CP, IBM CPLEX) and meta-heuristic search
(GA)
• GA is scalable and CP offers guarantees
80

Combining CP and GA
81
Search Space
!"
!#
!$
!%
!&
!'
("
(#
($
(%
(&
('
)"
)#
)$
)%
)&
)'(∗
!∗
GA-step: !", (", )" evolve into !', (', )'
CP-step: !', (', )' are optimized into !∗
, (∗
, )∗
= )∗
The key idea behind GA+CP is to run complete searches
with CP in the neighbourhood of solutions found by GA
2-steps strategy
Di Alesio et al., ACM TOSEM 2015

Solution Overview
82
UML Modeling
(e.g., MARTE)
Constraint Optimization
Optimization Problem
(Find arrival times that maximize the
chance of deadline misses)
System Platform
Solutions
(Task arrival times likely to lead to
deadline misses)
Deadline Misses Analysis
System Design
Design Model (Time and
Concurrency Information)
INPUT
OUTPUT
Stress Test Cases
Constraint
Prog. and Genetic
Algorithms

• Quality analysis for best practice compliance
• Terminology and unit consistency analysis
• Requirements similarity analysis
• Universal quantifier warnings
• Passive voice detection
• Incomplete sentence detection
• Easy Approach to Requirements Syntax (EARS) templating
NLP+ML Example
83
• QRA Corp: <qracorp.com/qvscribe>

Ease of use and control
for all functions.
3.1.1.1 The rover
navigation controller
shall provide interfaces
for on-board control,
and remote control by
the operation team.
3.1.1 Controller
Requirements
Requirements Demarcation
84
Requirements
Specification
(RS)
Demarcated
RS
Requirements vs. information about project
scope, comments, rationale …

Why is demarcating
requirements important?
Only requirements may be
• considered as contractually binding
• subject to automated quality assurance tasks like ambiguity
detection, change impact analysis, etc.
85

Requirements Demarcation
in Practice
• RS are large with 100s or 1000s of
statements
• Requirements are not clearly
separated from non-requirements
• Manual requirements demarcation
is time- and effort-consuming
86

Overview
Natural language processing (NLP) and machine learning (ML)
87
Frequency
Metadata
RS with
(NLP-generated)
Annotations
Demarcated
RS
Parse
Document
1
Build ML
Feature
Matrix
2
Classify
Requirement
Candidates
3
RS
Semantic
Features
Token-
based
Features
Syntactic
Features
Requirement
Candidates
Frequency-
based
Features
· · ·<latexit sha1_base64="m94MPbhQ0jNffvSn8H3/t4KQWm4=">AAACYXicZZDNSgMxFIVvx//xr+rSTbEbcVFmBHElFqWgOwVbhaZIJnOnRmeSIUn9CwM+glt9IZ/BtS9iphVh9GxyObnn43CjPOXaBMFnzZuanpmdm1/wF5eWV1bra+s9LUeKYZfJVKqriGpMucCu4SbFq1whzaIUL6O74/L/8h6V5lJcmKccBxkdCp5wRo2zeoTF0ujrejNoBWM1/g/hz9A8/PAPXgDg7Hqt1iGxZKMMhWEp1brPpEhQoWA4sKedTscoKgqfjDTmlN3RIdpx14rVz+PE4OPADhXNbzh7rAZGKq0aVCn6VPg+EfjAZJZREVuCKZYtin44sJYkUhohDWr+jMSxjU5sMyyKohrKuOBl8DclTJmwe4UNCr/RIO6gyEzpVylVjHY7MudsjCH32hXFHdvadWCSU0WE5CJ25ewYwt0bJRoVR92Y4Nzhw79n/j/0dlth0ArPw2b7CCaah03Ygm0IYR/acAJn0AUGt/AKb/Be+/IWvLq3Pln1aj+ZDajI2/wGA1K96Q==</latexit><latexit sha1_base64="sROku4tHxBM7SCz7EWzFzPdrdr8=">AAACYXicZZDNSgMxFIXT8X/8q7rsptiNuCgzgrgSRSnoTsG2QlNKJnOnRmeSkKT+hXkHt7r1YXwEce2LmGlFGHs2uZzc83G4kUyZNkHwVfFmZufmFxaX/OWV1bX16sZmR4uRotCmIhXqOiIaUsahbZhJ4VoqIFmUQje6Oy3+u/egNBP8yjxJ6GdkyFnCKDHO6mAaC6MH1UbQDMaqTw/h79A4+vAP5funfzHYqLRwLOgoA25oSrTuUcETUMAp9O15q9UyivDcxyMNktA7MgQ77lqyejJODDz27VARecPoYzkwUmnZIEqRp9z3MYcHKrKM8NhiSKFokffCvrU4EcJwYUCzZ8CObXRiG2Ge5+VQxjgrgn8pboqE3c9tkPv1OnYHBWoKv0wpY7TbEZLRMQbfa1cUdm1zz4GxJApzwXjsytkxhLk3SjQoBro+wbnDh//PPD109pph0Awvw8bxCZpoEdXQNtpBITpAx+gMXaA2ougWvaBX9Fb59pa8qrc5WfUqv5ktVJJX+wGdcb9d</latexit><latexit sha1_base64="sROku4tHxBM7SCz7EWzFzPdrdr8=">AAACYXicZZDNSgMxFIXT8X/8q7rsptiNuCgzgrgSRSnoTsG2QlNKJnOnRmeSkKT+hXkHt7r1YXwEce2LmGlFGHs2uZzc83G4kUyZNkHwVfFmZufmFxaX/OWV1bX16sZmR4uRotCmIhXqOiIaUsahbZhJ4VoqIFmUQje6Oy3+u/egNBP8yjxJ6GdkyFnCKDHO6mAaC6MH1UbQDMaqTw/h79A4+vAP5funfzHYqLRwLOgoA25oSrTuUcETUMAp9O15q9UyivDcxyMNktA7MgQ77lqyejJODDz27VARecPoYzkwUmnZIEqRp9z3MYcHKrKM8NhiSKFokffCvrU4EcJwYUCzZ8CObXRiG2Ge5+VQxjgrgn8pboqE3c9tkPv1OnYHBWoKv0wpY7TbEZLRMQbfa1cUdm1zz4GxJApzwXjsytkxhLk3SjQoBro+wbnDh//PPD109pph0Awvw8bxCZpoEdXQNtpBITpAx+gMXaA2ougWvaBX9Fb59pa8qrc5WfUqv5ktVJJX+wGdcb9d</latexit><latexit sha1_base64="FPXtabmdPbeQhp/pU1fr5K0aIz8=">AAACYXicZZHNSsNAFIWn8a+Nf21duilmIy5KIojbohR0V8FWoRNkMrmpo8lMmJlqdcg7uNU3c+2LOGmLEHs3czlzz8fh3ihPmdK+/11z1tY3NrfqDXd7Z3dvv9lqj5SYSgpDKlIh7yOiIGUchprpFO5zCSSLUriLni/L/7sXkIoJfqvfcggzMuEsYZRoK40wjYVWD03P7/rz6qw2wbLx0LIGD61aH8eCTjPgmqZEqTEVPAEJnEJorvv9vpaEFy6eKsgJfSYTMPOsFWmcx4mGWWgmkuSPjM6qhqlMqwKRkrwVros5vFKRZYTHBkMKZYpiHITG4EQIzYUGxd4BW7ZWifGCoiiqpoxxVhr/XFyXDnNWGL9wOx1sFwpUl3qVUsUoOyNyRucY/KJsUDgx3VMLxjmRmAvGYxvOzCHMvlGiQDJQnQXOLj74v+bVZnTaDfxucBN4vYvlCeroEB2hYxSgc9RDV2iAhoiiJ/SBPtFX7cdpOE2nvRh1akvPAaqUc/gLwxi8HA==</latexit>
cand1<latexit sha1_base64="vdVL+t1rA3jOBWduVds0zWFnHxU=">AAACbHicZVBdSxtBFL1ZbWu3Hya29EWE0EApfQg7QvE1KIH6ptCokAnp7ORuOrg7s8zManSYX+Jr+6N88Sf4GzqbWGH1vszhzD2Hc09a5sLYJLltRWvrL16+2ngdv3n77v1mu7N1YlSlOY64ypU+S5nBXEgcWWFzPCs1siLN8TQ9P6j/Ty9QG6HkT3tV4qRgcykywZkN1LS9SS0urMkcZ3Lmp2Ta7iX9ZDnd54A8gN7g0/HdLwA4mnZaQzpTvCpQWp4zY8ZcyQw1So4TdzgcDq1m0se0Mlgyfs7m6JaxG9S4nGUhyMTNNSt/C75oCiqdNwmmNbvycUwlXnJVFCG8o5hjncKPycQ5millpbJoxDX+P7JHvPdNUSGkqIWPKmlrhfvuXeLjbpeGbpHbmm+6NG1M2FGl4EsbemFCUPzm+rvBmJZMU6mEnIVwbmkiwptmBrVA013ZheLJ05qfg5PdPkn65Jj0Bvuwmg3Yhs/wFQjswQB+wBGMgEMFN/AH/rbuo4/RdrSzWo1aD5oP0Jjoyz9MKsIr</latexit><latexit sha1_base64="DUXu223g8BmE2+8xz7GsBmqt+90=">AAACbHicZVDBThsxFPQu0IYtLQlFvaBKKyKhqodojVT1GoEitbcgNYAUR5HXeZta7Nor20sDlr+k13LkL/oTvfQT+Aa8CSAtvItH4zejeZOWOdcmSf4F4dr6xqvXrc3ozdbbd9vtzs6plpViMGIyl+o8pRpyLmBkuMnhvFRAizSHs/TiuP4/uwSluRQ/zFUJk4LOBc84o8ZT0/Y2MbAwOrOMipmb4mm7m/SS5cQvAX4A3f6Hk//89ujvcNoJBmQmWVWAMCynWo+ZFBkoEAwm9vtgMDCKCheRSkNJ2QWdg13GblDjcpb5IBM7V7T8ydmiKahU3iSoUvTKRRER8IvJovDhLYEc6hRujCfWkkxKI6QBza/h8cguds41RQUXvBY+qYSpFfaLs4mL4pj4boGZmm+6NG2035ElZ0sbcql9UPhse4femJRUESG5mPlwdmnC/ZtmGhQHHa/sfPH4ec0vwelhDyc9fIK7/SO0mhbaQ/voE8LoK+qjb2iIRoihCv1Gf9BNcBfuhnvhx9VqGDxo3qPGhAf3ZqrD5w==</latexit><latexit sha1_base64="DUXu223g8BmE2+8xz7GsBmqt+90=">AAACbHicZVDBThsxFPQu0IYtLQlFvaBKKyKhqodojVT1GoEitbcgNYAUR5HXeZta7Nor20sDlr+k13LkL/oTvfQT+Aa8CSAtvItH4zejeZOWOdcmSf4F4dr6xqvXrc3ozdbbd9vtzs6plpViMGIyl+o8pRpyLmBkuMnhvFRAizSHs/TiuP4/uwSluRQ/zFUJk4LOBc84o8ZT0/Y2MbAwOrOMipmb4mm7m/SS5cQvAX4A3f6Hk//89ujvcNoJBmQmWVWAMCynWo+ZFBkoEAwm9vtgMDCKCheRSkNJ2QWdg13GblDjcpb5IBM7V7T8ydmiKahU3iSoUvTKRRER8IvJovDhLYEc6hRujCfWkkxKI6QBza/h8cguds41RQUXvBY+qYSpFfaLs4mL4pj4boGZmm+6NG2035ElZ0sbcql9UPhse4femJRUESG5mPlwdmnC/ZtmGhQHHa/sfPH4ec0vwelhDyc9fIK7/SO0mhbaQ/voE8LoK+qjb2iIRoihCv1Gf9BNcBfuhnvhx9VqGDxo3qPGhAf3ZqrD5w==</latexit><latexit sha1_base64="mwyGVp4WI1/sB/1rXgz/+RWCf2I=">AAACbHicZVHNSgMxGEzX//WvVW8iFAsiHspGEK+iFPSmYFVoSsmm39bgbrIkWbWGPIlXfShfwmcwW6uwOpd8TL4ZJpM4T7k2UfRRC2Zm5+YXFpfC5ZXVtfV6Y+NGy0Ix6DKZSnUXUw0pF9A13KRwlyugWZzCbfxwVt7fPoLSXIprM86hn9GR4Aln1HhqUF8nBp6NTiyjYugGeFBvRe1ogub/AU+HFprictCodchQsiIDYVhKte4xKRJQIBj07UWn0zGKCheSQkNO2QMdgZ3ErlC9fJj4IH07UjS/5+y5KihUWiWoUnTswpAIeGIyy3x4SyCFMoXr4b61JJHSCGlA8xf4eWQLO+eqoowLXgp/VcKUCnvkbOTCZpP4boGZkq+6VG2035E5ZxMb8qh9UDiw7UNvTHKqiJBcDH04OzHh/owTDYqDbn7b+eLx35r/DzeHbRy18RVunZxOv2ARbaNdtI8wOkYn6Bxdoi5iqECv6A291z6DrWA72PleDWpTzSaqINj7AuGwwEY=</latexit>
cand2<latexit sha1_base64="nnOJ6LbjwaaJnBc1UprVTrjKRIg=">AAACbHicZVBdSxtBFL1ZW6ur1WjFFykEA6X4EHYDxdegBOqbglEhE+Ls5G4c3J1ZZmb9GuaX+Nr+KF/8Cf4GZxMtbL0vczhzz+HckxQZ1yaKnhrBwqfPi1+WlsOV1a9r682NzTMtS8VwwGQm1UVCNWZc4MBwk+FFoZDmSYbnyfVh9X9+g0pzKU7NfYGjnE4FTzmjxlPj5joxeGd0ahkVEzfujpvtqBPNpvURxG+g3ds+eb4EgOPxRqNPJpKVOQrDMqr1kEmRokLBcGSP+v2+UVS4kJQaC8qu6RTtLHaNGhaT1AcZ2amixRVnd3VBqbI6QZWi9y4MicBbJvPch7cEM6xSuGE8spakUhohDWr+gO9HtmPnXF2Uc8Er4T+VMJXC/nI2cmGrRXy3yEzF113qNtrvyIKzmQ250T4o7tlO1xuTgioiJBcTH87OTLh/k1Sj4qhbcztffPx/zR/BWbcTR534JG73DmA+S7ADu/ATYtiHHvyGYxgAgxIe4Q/8bbwEW8FO8H2+GjTeNN+gNsGPV04iwiw=</latexit><latexit sha1_base64="TmV1aUG2URW8ONGbHJFGTzfHO2U=">AAACbHicZVDLSgMxFE3H9/iqD9yIUCyIuCgzBXFblILuFKwKTSmZ9E4NziRDkvEV8iVudelf+BNu/AS/wUyrwujd5HByz+HcE2UJUzoI3ivexOTU9MzsnD+/sLi0XF1ZvVAilxQ6VCRCXkVEQcI4dDTTCVxlEkgaJXAZ3RwV/5e3IBUT/Fw/ZNBLyZCzmFGiHdWvLmMN91rFhhI+sP1mv1oPGsFoav9B+A3qrY2zD/Z6+HbaX6m08UDQPAWuaUKU6lLBY5DAKfTMSbvd1pJw6+NcQUboDRmCGcUuUd1sELsgPTOUJLtm9L4syGVSJoiU5MH6PuZwR0WauvAGQwJFCtsNe8bgWAjNhQbFHuHnyHporS2LUsZZIfxVcV0ozL41gfVrNey6BaoLvuxStlFuR2SMjmzwrXJBYc80ms4YZ0RiLhgfuHBmZMLcG8UKJANVG9u54sO/Nf8HF81GGDTCs7DeOkTjmUWbaBvtohAdoBY6RqeogyjK0RN6Ri+VT2/d2/S2xqte5Vuzhkrj7XwBaKLD6A==</latexit><latexit sha1_base64="TmV1aUG2URW8ONGbHJFGTzfHO2U=">AAACbHicZVDLSgMxFE3H9/iqD9yIUCyIuCgzBXFblILuFKwKTSmZ9E4NziRDkvEV8iVudelf+BNu/AS/wUyrwujd5HByz+HcE2UJUzoI3ivexOTU9MzsnD+/sLi0XF1ZvVAilxQ6VCRCXkVEQcI4dDTTCVxlEkgaJXAZ3RwV/5e3IBUT/Fw/ZNBLyZCzmFGiHdWvLmMN91rFhhI+sP1mv1oPGsFoav9B+A3qrY2zD/Z6+HbaX6m08UDQPAWuaUKU6lLBY5DAKfTMSbvd1pJw6+NcQUboDRmCGcUuUd1sELsgPTOUJLtm9L4syGVSJoiU5MH6PuZwR0WauvAGQwJFCtsNe8bgWAjNhQbFHuHnyHporS2LUsZZIfxVcV0ozL41gfVrNey6BaoLvuxStlFuR2SMjmzwrXJBYc80ms4YZ0RiLhgfuHBmZMLcG8UKJANVG9u54sO/Nf8HF81GGDTCs7DeOkTjmUWbaBvtohAdoBY6RqeogyjK0RN6Ri+VT2/d2/S2xqte5Vuzhkrj7XwBaKLD6A==</latexit><latexit sha1_base64="gOE5Tsr0XG9wfp6PJaYZIvat/u0=">AAACbHicZVHLSgMxFE3H9/hofexEKBZEXJSZgrgVpaA7BfuAppRMeqcGZ5IhyfgK+RK3+lH+hN9gplZh7NnkcnLP4eQkyhKmdBB8VryFxaXlldU1f31jc6ta297pKpFLCh0qEiH7EVGQMA4dzXQC/UwCSaMEetHDZXHfewSpmOB3+iWDYUomnMWMEu2oUa2KNTxrFRtK+NiOWqNaI2gGU9Tnh3A2NNAMN6PtShuPBc1T4JomRKkBFTwGCZzC0Fy3220tCbc+zhVkhD6QCZhp7BI1yMaxCzI0E0mye0afy4JcJmWCSElerO9jDk9UpKkLbzAkUKSwg3BoDI6F0FxoUOwVfh/ZCK21ZVHKOCuEfyquC4U5tSawfr2OXbdAdcGXXco2yu2IjNGpDX5ULiicmGbLGeOMSMwF42MXzkxNmDujWIFkoOo/dq748H/N80O31QyDZngbNs4vZl+wivbRITpGITpD5+gK3aAOoihHb+gdfVS+vD1v3zv4WfUqM80uKsE7+gbjqMBH</latexit>
candn<latexit sha1_base64="1fNP/MX+F29viko764w5DE/BU+M=">AAACbHicZVBdSxtBFL3Z+tVt/YgVX6QQGhDxIewK0tegBNo3hUaFTEhnJ3fjkN2ZZWbWRof5JX3VH+WLP6G/wdlEha33ZQ5n7jmce5Ii49pE0WMj+LC0vLK69jH89Hl9Y3OruX2hZakY9pnMpLpKqMaMC+wbbjK8KhTSPMnwMpmeVv+XN6g0l+KXuS1wmNOJ4Cln1HhqtLVJDM6MTi2jYuxGnmlHnWg+rfcgfgHt7u75028AOBs1Gz0ylqzMURiWUa0HTIoUFQqGQ/uz1+sZRYULSamxoGxKJ2jnsWvUoBinPsjQThQtrjmb1QWlyuoEVYreujAkAv8wmec+vCWYYZXCDeKhtSSV0ghpUPM7fD2yHTvn6qKcC14J31TCVAp77GzkwlaL+G6RmYqvu9RttN+RBWdzG3KjfVA8tJ0jb0wKqoiQXIx9ODs34f5NUo2Ko24t7Hzx8f81vwcXR5046sTncbt7AotZgz34BgcQw3fowg84gz4wKOEv3MND41+wE+wFXxerQeNF8wVqE+w/A8RCwmg=</latexit><latexit sha1_base64="ncjRRTHvNSb0Cn5suQWb5+WFdF8=">AAACbHicZVDLSgMxFE3H9/isihsRigURF2VGELdFKehOwarQlJJJ79TgTDIkGa2GfIlbXfoX/oQbP8FvMNOqMHo3OZzcczj3RFnClA6C94o3MTk1PTM7588vLC4tr1RXL5XIJYU2FYmQ1xFRkDAObc10AteZBJJGCVxFt8fF/9UdSMUEv9APGXRTMuAsZpRoR/VWlrGGoVaxoYT3bc8x9aARjKb2H4TfoN7cOP9gr0dvZ71qpYX7guYpcE0TolSHCh6DBE6ha05brZaWhFsf5woyQm/JAMwodonqZP3YBemagSTZDaPDsiCXSZkgUpIH6/uYwz0VaerCGwwJFClsJ+wag2MhNBcaFHuEnyProbW2LEoZZ4XwV8V1oTAH1gTWr9Ww6xaoLviyS9lGuR2RMTqywXfKBYU909h3xjgjEnPBeN+FMyMT5t4oViAZqNrYzhUf/q35P7jcb4RBIzwP680jNJ5ZtIm20S4K0SFqohN0htqIohw9oWf0Uvn01r1Nb2u86lW+NWuoNN7OF97CxCQ=</latexit><latexit sha1_base64="ncjRRTHvNSb0Cn5suQWb5+WFdF8=">AAACbHicZVDLSgMxFE3H9/isihsRigURF2VGELdFKehOwarQlJJJ79TgTDIkGa2GfIlbXfoX/oQbP8FvMNOqMHo3OZzcczj3RFnClA6C94o3MTk1PTM7588vLC4tr1RXL5XIJYU2FYmQ1xFRkDAObc10AteZBJJGCVxFt8fF/9UdSMUEv9APGXRTMuAsZpRoR/VWlrGGoVaxoYT3bc8x9aARjKb2H4TfoN7cOP9gr0dvZ71qpYX7guYpcE0TolSHCh6DBE6ha05brZaWhFsf5woyQm/JAMwodonqZP3YBemagSTZDaPDsiCXSZkgUpIH6/uYwz0VaerCGwwJFClsJ+wag2MhNBcaFHuEnyProbW2LEoZZ4XwV8V1oTAH1gTWr9Ww6xaoLviyS9lGuR2RMTqywXfKBYU909h3xjgjEnPBeN+FMyMT5t4oViAZqNrYzhUf/q35P7jcb4RBIzwP680jNJ5ZtIm20S4K0SFqohN0htqIohw9oWf0Uvn01r1Nb2u86lW+NWuoNN7OF97CxCQ=</latexit><latexit sha1_base64="2hnVasoxEhPkAPOHTjTasjrspsg=">AAACbHicZVHNSgMxGEzXv7r+tFVvIhQLIh7KbkG8ilLQWwWrQlNKNv22BneTJcmqNeRJvOpD+RI+g9lahdW55GPyzTCZRFnClA6Cj4q3sLi0vFJd9dfWNzZr9cbWjRK5pNCnIhHyLiIKEsahr5lO4C6TQNIogdvo4by4v30EqZjg13qawTAlE85iRol21KhewxqetYoNJXxsR45pBe1ghub/IZwPLTRHb9SodPFY0DwFrmlClBpQwWOQwCkMzWW329WScOvjXEFG6AOZgJnFLlGDbBy7IEMzkSS7Z/S5LMhlUiaIlGRqfR9zeKIiTV14gyGBIoUdhENjcCyE5kKDYi/w88hWaK0ti1LGWSH8VXFdKMyxNYH1m03sugWqC77sUrZRbkdkjM5s8KNyQeHItDvOGGdEYi4YH7twZmbC3BnFCiQD1fy2c8WHf2v+P9x02mHQDq/C1unZ/AuqaBfto0MUohN0ii5QD/URRTl6RW/ovfLp7Xi73t73qleZa7ZRCd7BF1nXwIM=</latexit>
Abualhaija et al., RE 2019

Document Parsing
• Extracting sentences =
requirement candidates
• Getting NLP annotations
• Computing frequency related
metadata
88
NLP Pipeline
Constituency Parser
Semantic Parser
Dependency Parser
Parsing
Tokenizer
POS Tagger
Sentence Splitter
Preprocessing
Frequency
Metadata
RS with
(NLP-generated)
Annotations
Parse
Document
1
RS

Feature Matrix Building
89
20 features collectively help in distinguishing requirements
from non-requirements
6 Token-based, 8 Syntactic, 3 Semantic & 3 Frequency-based
Build ML
Feature
Matrix
2
Semantic
Features
Token-
based
Features
Syntactic
Features
Requirement
Candidates
Frequency-
based
Features
cand1<latexit sha1_base64="vdVL+t1rA3jOBWduVds0zWFnHxU=">AAACbHicZVBdSxtBFL1ZbWu3Hya29EWE0EApfQg7QvE1KIH6ptCokAnp7ORuOrg7s8zManSYX+Jr+6N88Sf4GzqbWGH1vszhzD2Hc09a5sLYJLltRWvrL16+2ngdv3n77v1mu7N1YlSlOY64ypU+S5nBXEgcWWFzPCs1siLN8TQ9P6j/Ty9QG6HkT3tV4qRgcykywZkN1LS9SS0urMkcZ3Lmp2Ta7iX9ZDnd54A8gN7g0/HdLwA4mnZaQzpTvCpQWp4zY8ZcyQw1So4TdzgcDq1m0se0Mlgyfs7m6JaxG9S4nGUhyMTNNSt/C75oCiqdNwmmNbvycUwlXnJVFCG8o5hjncKPycQ5millpbJoxDX+P7JHvPdNUSGkqIWPKmlrhfvuXeLjbpeGbpHbmm+6NG1M2FGl4EsbemFCUPzm+rvBmJZMU6mEnIVwbmkiwptmBrVA013ZheLJ05qfg5PdPkn65Jj0Bvuwmg3Yhs/wFQjswQB+wBGMgEMFN/AH/rbuo4/RdrSzWo1aD5oP0Jjoyz9MKsIr</latexit><latexit sha1_base64="DUXu223g8BmE2+8xz7GsBmqt+90=">AAACbHicZVDBThsxFPQu0IYtLQlFvaBKKyKhqodojVT1GoEitbcgNYAUR5HXeZta7Nor20sDlr+k13LkL/oTvfQT+Aa8CSAtvItH4zejeZOWOdcmSf4F4dr6xqvXrc3ozdbbd9vtzs6plpViMGIyl+o8pRpyLmBkuMnhvFRAizSHs/TiuP4/uwSluRQ/zFUJk4LOBc84o8ZT0/Y2MbAwOrOMipmb4mm7m/SS5cQvAX4A3f6Hk//89ujvcNoJBmQmWVWAMCynWo+ZFBkoEAwm9vtgMDCKCheRSkNJ2QWdg13GblDjcpb5IBM7V7T8ydmiKahU3iSoUvTKRRER8IvJovDhLYEc6hRujCfWkkxKI6QBza/h8cguds41RQUXvBY+qYSpFfaLs4mL4pj4boGZmm+6NG2035ElZ0sbcql9UPhse4femJRUESG5mPlwdmnC/ZtmGhQHHa/sfPH4ec0vwelhDyc9fIK7/SO0mhbaQ/voE8LoK+qjb2iIRoihCv1Gf9BNcBfuhnvhx9VqGDxo3qPGhAf3ZqrD5w==</latexit><latexit sha1_base64="DUXu223g8BmE2+8xz7GsBmqt+90=">AAACbHicZVDBThsxFPQu0IYtLQlFvaBKKyKhqodojVT1GoEitbcgNYAUR5HXeZta7Nor20sDlr+k13LkL/oTvfQT+Aa8CSAtvItH4zejeZOWOdcmSf4F4dr6xqvXrc3ozdbbd9vtzs6plpViMGIyl+o8pRpyLmBkuMnhvFRAizSHs/TiuP4/uwSluRQ/zFUJk4LOBc84o8ZT0/Y2MbAwOrOMipmb4mm7m/SS5cQvAX4A3f6Hk//89ujvcNoJBmQmWVWAMCynWo+ZFBkoEAwm9vtgMDCKCheRSkNJ2QWdg13GblDjcpb5IBM7V7T8ydmiKahU3iSoUvTKRRER8IvJovDhLYEc6hRujCfWkkxKI6QBza/h8cguds41RQUXvBY+qYSpFfaLs4mL4pj4boGZmm+6NG2035ElZ0sbcql9UPhse4femJRUESG5mPlwdmnC/ZtmGhQHHa/sfPH4ec0vwelhDyc9fIK7/SO0mhbaQ/voE8LoK+qjb2iIRoihCv1Gf9BNcBfuhnvhx9VqGDxo3qPGhAf3ZqrD5w==</latexit><latexit sha1_base64="mwyGVp4WI1/sB/1rXgz/+RWCf2I=">AAACbHicZVHNSgMxGEzX//WvVW8iFAsiHspGEK+iFPSmYFVoSsmm39bgbrIkWbWGPIlXfShfwmcwW6uwOpd8TL4ZJpM4T7k2UfRRC2Zm5+YXFpfC5ZXVtfV6Y+NGy0Ix6DKZSnUXUw0pF9A13KRwlyugWZzCbfxwVt7fPoLSXIprM86hn9GR4Aln1HhqUF8nBp6NTiyjYugGeFBvRe1ogub/AU+HFprictCodchQsiIDYVhKte4xKRJQIBj07UWn0zGKCheSQkNO2QMdgZ3ErlC9fJj4IH07UjS/5+y5KihUWiWoUnTswpAIeGIyy3x4SyCFMoXr4b61JJHSCGlA8xf4eWQLO+eqoowLXgp/VcKUCnvkbOTCZpP4boGZkq+6VG2035E5ZxMb8qh9UDiw7UNvTHKqiJBcDH04OzHh/owTDYqDbn7b+eLx35r/DzeHbRy18RVunZxOv2ARbaNdtI8wOkYn6Bxdoi5iqECv6A291z6DrWA72PleDWpTzSaqINj7AuGwwEY=</latexit>
cand2<latexit sha1_base64="nnOJ6LbjwaaJnBc1UprVTrjKRIg=">AAACbHicZVBdSxtBFL1ZW6ur1WjFFykEA6X4EHYDxdegBOqbglEhE+Ls5G4c3J1ZZmb9GuaX+Nr+KF/8Cf4GZxMtbL0vczhzz+HckxQZ1yaKnhrBwqfPi1+WlsOV1a9r682NzTMtS8VwwGQm1UVCNWZc4MBwk+FFoZDmSYbnyfVh9X9+g0pzKU7NfYGjnE4FTzmjxlPj5joxeGd0ahkVEzfujpvtqBPNpvURxG+g3ds+eb4EgOPxRqNPJpKVOQrDMqr1kEmRokLBcGSP+v2+UVS4kJQaC8qu6RTtLHaNGhaT1AcZ2amixRVnd3VBqbI6QZWi9y4MicBbJvPch7cEM6xSuGE8spakUhohDWr+gO9HtmPnXF2Uc8Er4T+VMJXC/nI2cmGrRXy3yEzF113qNtrvyIKzmQ250T4o7tlO1xuTgioiJBcTH87OTLh/k1Sj4qhbcztffPx/zR/BWbcTR534JG73DmA+S7ADu/ATYtiHHvyGYxgAgxIe4Q/8bbwEW8FO8H2+GjTeNN+gNsGPV04iwiw=</latexit><latexit sha1_base64="TmV1aUG2URW8ONGbHJFGTzfHO2U=">AAACbHicZVDLSgMxFE3H9/iqD9yIUCyIuCgzBXFblILuFKwKTSmZ9E4NziRDkvEV8iVudelf+BNu/AS/wUyrwujd5HByz+HcE2UJUzoI3ivexOTU9MzsnD+/sLi0XF1ZvVAilxQ6VCRCXkVEQcI4dDTTCVxlEkgaJXAZ3RwV/5e3IBUT/Fw/ZNBLyZCzmFGiHdWvLmMN91rFhhI+sP1mv1oPGsFoav9B+A3qrY2zD/Z6+HbaX6m08UDQPAWuaUKU6lLBY5DAKfTMSbvd1pJw6+NcQUboDRmCGcUuUd1sELsgPTOUJLtm9L4syGVSJoiU5MH6PuZwR0WauvAGQwJFCtsNe8bgWAjNhQbFHuHnyHporS2LUsZZIfxVcV0ozL41gfVrNey6BaoLvuxStlFuR2SMjmzwrXJBYc80ms4YZ0RiLhgfuHBmZMLcG8UKJANVG9u54sO/Nf8HF81GGDTCs7DeOkTjmUWbaBvtohAdoBY6RqeogyjK0RN6Ri+VT2/d2/S2xqte5Vuzhkrj7XwBaKLD6A==</latexit><latexit sha1_base64="TmV1aUG2URW8ONGbHJFGTzfHO2U=">AAACbHicZVDLSgMxFE3H9/iqD9yIUCyIuCgzBXFblILuFKwKTSmZ9E4NziRDkvEV8iVudelf+BNu/AS/wUyrwujd5HByz+HcE2UJUzoI3ivexOTU9MzsnD+/sLi0XF1ZvVAilxQ6VCRCXkVEQcI4dDTTCVxlEkgaJXAZ3RwV/5e3IBUT/Fw/ZNBLyZCzmFGiHdWvLmMN91rFhhI+sP1mv1oPGsFoav9B+A3qrY2zD/Z6+HbaX6m08UDQPAWuaUKU6lLBY5DAKfTMSbvd1pJw6+NcQUboDRmCGcUuUd1sELsgPTOUJLtm9L4syGVSJoiU5MH6PuZwR0WauvAGQwJFCtsNe8bgWAjNhQbFHuHnyHporS2LUsZZIfxVcV0ozL41gfVrNey6BaoLvuxStlFuR2SMjmzwrXJBYc80ms4YZ0RiLhgfuHBmZMLcG8UKJANVG9u54sO/Nf8HF81GGDTCs7DeOkTjmUWbaBvtohAdoBY6RqeogyjK0RN6Ri+VT2/d2/S2xqte5Vuzhkrj7XwBaKLD6A==</latexit><latexit sha1_base64="gOE5Tsr0XG9wfp6PJaYZIvat/u0=">AAACbHicZVHLSgMxFE3H9/hofexEKBZEXJSZgrgVpaA7BfuAppRMeqcGZ5IhyfgK+RK3+lH+hN9gplZh7NnkcnLP4eQkyhKmdBB8VryFxaXlldU1f31jc6ta297pKpFLCh0qEiH7EVGQMA4dzXQC/UwCSaMEetHDZXHfewSpmOB3+iWDYUomnMWMEu2oUa2KNTxrFRtK+NiOWqNaI2gGU9Tnh3A2NNAMN6PtShuPBc1T4JomRKkBFTwGCZzC0Fy3220tCbc+zhVkhD6QCZhp7BI1yMaxCzI0E0mye0afy4JcJmWCSElerO9jDk9UpKkLbzAkUKSwg3BoDI6F0FxoUOwVfh/ZCK21ZVHKOCuEfyquC4U5tSawfr2OXbdAdcGXXco2yu2IjNGpDX5ULiicmGbLGeOMSMwF42MXzkxNmDujWIFkoOo/dq748H/N80O31QyDZngbNs4vZl+wivbRITpGITpD5+gK3aAOoihHb+gdfVS+vD1v3zv4WfUqM80uKsE7+gbjqMBH</latexit>
candn<latexit sha1_base64="1fNP/MX+F29viko764w5DE/BU+M=">AAACbHicZVBdSxtBFL3Z+tVt/YgVX6QQGhDxIewK0tegBNo3hUaFTEhnJ3fjkN2ZZWbWRof5JX3VH+WLP6G/wdlEha33ZQ5n7jmce5Ii49pE0WMj+LC0vLK69jH89Hl9Y3OruX2hZakY9pnMpLpKqMaMC+wbbjK8KhTSPMnwMpmeVv+XN6g0l+KXuS1wmNOJ4Cln1HhqtLVJDM6MTi2jYuxGnmlHnWg+rfcgfgHt7u75028AOBs1Gz0ylqzMURiWUa0HTIoUFQqGQ/uz1+sZRYULSamxoGxKJ2jnsWvUoBinPsjQThQtrjmb1QWlyuoEVYreujAkAv8wmec+vCWYYZXCDeKhtSSV0ghpUPM7fD2yHTvn6qKcC14J31TCVAp77GzkwlaL+G6RmYqvu9RttN+RBWdzG3KjfVA8tJ0jb0wKqoiQXIx9ODs34f5NUo2Ko24t7Hzx8f81vwcXR5046sTncbt7AotZgz34BgcQw3fowg84gz4wKOEv3MND41+wE+wFXxerQeNF8wVqE+w/A8RCwmg=</latexit><latexit sha1_base64="ncjRRTHvNSb0Cn5suQWb5+WFdF8=">AAACbHicZVDLSgMxFE3H9/isihsRigURF2VGELdFKehOwarQlJJJ79TgTDIkGa2GfIlbXfoX/oQbP8FvMNOqMHo3OZzcczj3RFnClA6C94o3MTk1PTM7588vLC4tr1RXL5XIJYU2FYmQ1xFRkDAObc10AteZBJJGCVxFt8fF/9UdSMUEv9APGXRTMuAsZpRoR/VWlrGGoVaxoYT3bc8x9aARjKb2H4TfoN7cOP9gr0dvZ71qpYX7guYpcE0TolSHCh6DBE6ha05brZaWhFsf5woyQm/JAMwodonqZP3YBemagSTZDaPDsiCXSZkgUpIH6/uYwz0VaerCGwwJFClsJ+wag2MhNBcaFHuEnyProbW2LEoZZ4XwV8V1oTAH1gTWr9Ww6xaoLviyS9lGuR2RMTqywXfKBYU909h3xjgjEnPBeN+FMyMT5t4oViAZqNrYzhUf/q35P7jcb4RBIzwP680jNJ5ZtIm20S4K0SFqohN0htqIohw9oWf0Uvn01r1Nb2u86lW+NWuoNN7OF97CxCQ=</latexit><latexit sha1_base64="ncjRRTHvNSb0Cn5suQWb5+WFdF8=">AAACbHicZVDLSgMxFE3H9/isihsRigURF2VGELdFKehOwarQlJJJ79TgTDIkGa2GfIlbXfoX/oQbP8FvMNOqMHo3OZzcczj3RFnClA6C94o3MTk1PTM7588vLC4tr1RXL5XIJYU2FYmQ1xFRkDAObc10AteZBJJGCVxFt8fF/9UdSMUEv9APGXRTMuAsZpRoR/VWlrGGoVaxoYT3bc8x9aARjKb2H4TfoN7cOP9gr0dvZ71qpYX7guYpcE0TolSHCh6DBE6ha05brZaWhFsf5woyQm/JAMwodonqZP3YBemagSTZDaPDsiCXSZkgUpIH6/uYwz0VaerCGwwJFClsJ+wag2MhNBcaFHuEnyProbW2LEoZZ4XwV8V1oTAH1gTWr9Ww6xaoLviyS9lGuR2RMTqywXfKBYU909h3xjgjEnPBeN+FMyMT5t4oViAZqNrYzhUf/q35P7jcb4RBIzwP680jNJ5ZtIm20S4K0SFqohN0htqIohw9oWf0Uvn01r1Nb2u86lW+NWuoNN7OF97CxCQ=</latexit><latexit sha1_base64="2hnVasoxEhPkAPOHTjTasjrspsg=">AAACbHicZVHNSgMxGEzXv7r+tFVvIhQLIh7KbkG8ilLQWwWrQlNKNv22BneTJcmqNeRJvOpD+RI+g9lahdW55GPyzTCZRFnClA6Cj4q3sLi0vFJd9dfWNzZr9cbWjRK5pNCnIhHyLiIKEsahr5lO4C6TQNIogdvo4by4v30EqZjg13qawTAlE85iRol21KhewxqetYoNJXxsR45pBe1ghub/IZwPLTRHb9SodPFY0DwFrmlClBpQwWOQwCkMzWW329WScOvjXEFG6AOZgJnFLlGDbBy7IEMzkSS7Z/S5LMhlUiaIlGRqfR9zeKIiTV14gyGBIoUdhENjcCyE5kKDYi/w88hWaK0ti1LGWSH8VXFdKMyxNYH1m03sugWqC77sUrZRbkdkjM5s8KNyQeHItDvOGGdEYi4YH7twZmbC3BnFCiQD1fy2c8WHf2v+P9x02mHQDq/C1unZ/AuqaBfto0MUohN0ii5QD/URRTl6RW/ovfLp7Xi73t73qleZa7ZRCd7BF1nXwIM=</latexit>

ML Algorithm Selection
90
90 91.7 91.9 90.4 91
94.3 95.4 96.9 95.7
92.4
0
10
20
30
40
50
60
70
80
90
100
Neural
Network
Decision
Tree
Random
Forest
Logistic SVM
Recall (%)
Recall Recall+CSL
84.3
91.1
94.3
92.1 90.1
76.5
86.9 88.9 87.3 87.5
0
10
20
30
40
50
60
70
80
90
100
Neural
Network
Decision
Tree
Random
Forest
Logistic SVM
Precision (%)
Precision Precision+CSL
= 96.6% = 88.9%

Other Projects
• Security testing of financial applications: Search + ML+solver (Appelt
et al., Jan et al.)
• Satellite in-orbit testing (HiL testing): Search+ simulation (Shin et al.)
• Solving complex system-level constraints for system testing:
Search+SMT (Soltana et al.)
• Recommender systems for security controls in banking applications:
NLP+ML (Bettaieb et al.)
• Testing and debugging Simulink models: Search+ML (Matinnejad et
al., Liu et al.)
91

Metaheuristic Search
• Most automation problems can be re-expressed into
search (stochastic optimization) problems.
• Versatile. Parallelizable. Many algorithms matching
various situations.
• Can help optimize ML hyper parameters.
• But not that easy to use and deploy for practitioners.
93

Machine Learning
• Helps guide search by learning interesting parts of the
search space on the go.
• Complements NLP when no deterministic rules are
possible to extract information and make decisions
from text.
• Helps understand actionable patterns in SE data, e.g.,
interpret test results and support debugging by
understanding failure conditions.
94

NLP
• NLP helps exploit information in natural language and
push automation further.
• Extract useful features for ML.
95

Newer Learning Paradigms
• Deep learning, Active learning, Reinforcement learning.
• We need to understand in which SE contexts and
applications they are truly our best options.
• But we should not yield to fashion and hype …
• For many SE problems, “classical” ML works well:
Feature engineering is most important, along with the
right combination of techniques.
96

SE for AI
• We barely start to understand the SE implications of
the increasing adoption of Deep Learning and ML in
general.
• Amershi et al., “Software Engineering for Machine
Learning: A Case Study”, ICSE SEIP 2019
• Very active research on testing and debugging of
Deep Neural Networks and the systems they are
embedded in.
97

The Road Ahead
• Real solutions strike a balance in terms of scalability,
practicality, applicability, and offering a maximum level of
dependability guarantees.
• Real solutions involves several techniques, combined to
achieve the best trade-offs.
• We need more multi-disciplinary research driven by (well-
defined) problems.
• Research in this field cannot be oblivious to context (e.g.,
domain): Working assumptions, desirable trade-offs …
98

Academia-Industry Gap
• Many academic papers address problems that are
unlikely to exist as defined, e.g., working assumptions.
• On the other hand, many industrial problems (often long
standing) are insufficiently addressed by research.
• Context factors and working assumptions have a huge
impact on software engineering solutions.
• Scalability and practicality aspects are largely ignored by
research – they are often considered as an afterthought
99

Selected References
• Matinnejad et al., “MiL Testing of Highly Configurable Continuous Controllers: Scalable Search Using Surrogate
Models”, ASE 2014
• Matinnejad et al., “Automated Test Suite Generation for Time-Continuous Simulink Models”, ICSE 2016.
• Matinnejad et al., “Test Generation and Test Prioritization for Simulink Models with Dynamic Behavior”, IEEE
Transactions on Software Engineering, 2018
• Liu et al., “Effective Fault Localization of Automotive Simulink Models: Achieving the Trade-Off between Test Oracle
Effort and Fault Localization Accuracy”, Empirical Software Engineering (Springer), 2019
• Liu et al., “Simulink Fault Localisation: An Iterative Statistical Debugging Approach”, Software Testing, Verification &
Reliability (Wiley), 2016
• Ben Abdessalem et al., "Testing Vision-Based Control Systems Using Learnable Evolutionary Algorithms”, ICSE 2018
• Ben Abdessalem et al., "Testing Autonomous Cars for Feature Interaction Failures using Many-Objective Search”, ASE
2018
• Nejati et al., “Evaluating Model Testing and Model Checking for Finding Requirements Violations in Simulink Models”,
ESEC/FSE 2019.
103

Selected References
• Shousha et al., "Using Genetic Algorithms for Early Schedulability Analysis and Stress Testing in Real-Time Systems",
Journal of Genetic Programming and Evolvable Machines (Springer), 2006.
• Shousha et al., “A UML/MARTE Model Analysis Method for Uncovering Scenarios Leading to Starvation and Deadlocks
in Concurrent Systems”, IEEE Transactions on Software Engineering, 2012.
• Nejati and Briand, “Identifying Optimal Trade-Offs between CPU Time Usage and Temporal Constraints Using Search”,
ISSTA 2014
• Di Alesio et al. “Combining genetic algorithms and constraint programming to support stress testing of task
deadlines”, ACM Transactions on Software Engineering and Methodology, 2015
• Arisholm et al., “A systematic and comprehensive investigation of methods to build and evaluate fault prediction
models”, Journal of Systems and Software, 2010
• Yue et al., “Facilitating the transition from use case models to analysis models: Approach and experiments.”, ACM
TOSEM, 2013
104

Selected References
• Wang et al., “Automatic generation of system test cases from use case specifications”, ISSTA 2015
• Wang et al., “System Testing of Timing Requirements Based on Use Cases and Timed Automata”, ICST 2017
• Wang et al., “Automated Generation of Constraints from Use Case Specifications to Support System Testing”, ICST
2018
• Wang et al., “Oracles for Testing Software Timeliness with Uncertainty”, ACM TOSEM, 2019
• Shin et al., “Test case prioritization for acceptance testing of cyber-physical systems”, ISSTA 2018
• Shin et al.,“HITECS: A UML Profile and Analysis Framework for Hardware-in-the-Loop Testing of Cyber Physical
Systems”, MODELS 2018
• Soltana et al., “Practical Model-driven Data Generation for System Testing”, CoRR abs/1902.00397 (2019)
• Shin et al., “Dynamic Adaptive Network Configuration for IoT Systems: A Search-based
Approach”. CoRRabs/1905.12763 (2019)
105

Selected References
• Jan et al., Automatic Generation of Tests to Exploit XML Injection Vulnerabilities in Web Applications. IEEE Transactions
on Software Engineering (TSE), 2019
• Appelt et al., A Machine Learning-Driven Evolutionary Approach for Testing Web Application Firewalls. IEEE
Transactions on Reliability (TR), 2018
• Appelt et al., Automatically Repairing Web Application Firewalls Based on Successful SQL Injection Attacks. IEEE
International Symposium on Software Reliability Engineering (ISSRE 2017)
• Jan et al., Search-based Testing Approach for XML Injection Vulnerabilities in Web Applications. IEEE International
Conference on Software Testing, Verification and validation (ICST 2017)
• Jan et al., Automated and Effective Testing of Web Services for XML Injection Attacks. International Symposium on
Software Testing and Analysis (ISSTA 2016)
• Ceccato et al., SOFIA: An Automated Security Oracle for Black-Box Testing of SQL-Injection Vulnerabilities, IEEE/ACM
International Conference on Automated Software Engineering (ASE 2016)
106

Selected References
• Arora et al. “Extracting Domain Models from Natural-Language Requirements: Approach and Industrial
Evaluation”, MODELS 2016
• Arora et al. “Automated Extraction and Clustering of Requirements Glossary Terms”, IEEE TSE, 2017
• Arora et al., “An Active Learning Approach for Improving the Accuracy of Automated Domain Model Extraction”, ACM
TOSEM, 2019
• Nejati et al., “Automated Change Impact Analysis between SysML Models of Requirements and Design”, FSE 2016
• Abualhaija et al., “A Machine-Learning Approach for Demarcating Requirements in Textual Specifications”, RE 2019
• Bettaieb et al., “Decision Support for Security-Control Identification Using Machine Learning”, REFSQ 2019
• Sleimi et al., “ A Query System for Extracting Requirements-related Information from Legal Texts”, RE 2019
• Sleimi et al., “Automated Extraction of Semantic Legal Metadata using Natural Language Processing”, RE 2018
107

AI in SE: A 25-year Journey

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to AI in SE: A 25-year Journey

Similar to AI in SE: A 25-year Journey (20)

More from Lionel Briand

More from Lionel Briand (20)

Recently uploaded

Recently uploaded (20)

AI in SE: A 25-year Journey