Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
RECOGNIZING PATTERNS
IN NOISY DATA USING
TRAINABLE ‘FUNCTIONAL’
STATE MACHINES
Faisal Waris
OUTLINE
BACKGROUND & MOTIVATION -
3 TRENDS
Android Auto
Cadillac Gesture Pad
BMW In-the-air Gestures
IDEA – WEARABLE DEVICE
GESTURE CONTROL OF HMI
Twist
Tap Swipe Left Right Escape
Twist gesture
recognized
Escape gesture recognized
IMPLEMENTATION
Simple Gesture
Vocabul...
GESTURE RECOGNITION ≈
PATTERN RECOGNITION IN
NOISY (SEQUENTIAL) DATA
Sensors Inherent Noise
Fusion: Accelerometer, Gyro, M...
SEQUENTIAL DATA, PATTERN
RECOGNITION TECHNIQUES
FINITE STATE MACHINES
(FSM)
OUTLINE
PATTERN RECOGNITION WITH
FSM
FSM to recognize “…aaab…” pattern
a1 a2 a3
a a a
not a
b
OUTLINE
‘FUNCTIONAL’ STATE
MACHINES (FNSM*)
FSM expressed using functional programming constructs
* FnSM is both singular and plur...
NATURAL TRANSFORMATION TO
F# (ML)
type M<'s> = M of ('s -> M<'s>)
let rec start = function
| 'a' -> M a1
| _ -> M start
an...
STATE-SPACE EXPLOSION
How about 3 ‘a’s and 3 ‘b’s in any order?
Are 6 states enough?
a1 a2 a3
b1 b2 b3
Need exponential # ...
FNSM FOR 3 ‘A’S AND 3 ‘B’S IN
ANY ORDERlet rec fStart = function
| 'a' -> fAB 1 0 |> M
| 'b' -> fAB 0 1 |> M
| _ -> fStart...
FNSM FOR WEARABLE DEVICE
SENSOR DATA
type SnsrEvent =
{
Snsr : SensorTypeEnum
Ticks : long
X : float; Y : float; Z : float...
F# FNSM MODULE IN
PRACTICE - ANDROID ‘function-state’
recursive type
Evaluation function takes a
function-state and an eve...
OUTLINE
HANDLING NOISY DATA
let rec start = function
| {Snsr=LinearAcceleration; Z=z} when abs z > 5.0 // m/sec2
 possible_swing ...
PARAMETERIZE FNSM FOR
TRAINABILITY
let rec start cfg = function
| {Snsr=LinearAcceleration; Z=z} when abs z > cfg.MIN_Z_TH...
FNSM – OTHER
FnSM as partially applied values in
other FnSM
 Hierarchical organization of state machines
 Call-return se...
FNSM CONCEPTS – INTERIM
SUMMARY
OUTLINE
FNSM TRAINING – PARAMETERS
z_accel_front_thrsld = 2.74647951f;
z_accel_back_thrsld = -4.25041485f;
z_accel_avg_front_thrsl...
FNSM TRAINING – PROCESS
2
Labelled training
data: Perform
each gesture X
times and record
sensor output
Analyze gesture da...
FNSM TRAINING – OPTIMIZATION
FORMULATION
p = parm array values from a population individual
g ∈ G ={Left, Right, Tap, Swip...
SUMMARY
Functional programming provides a powerful
means of modeling FSM
 State | Events | Transitions  function | patt...
OUTLINE
A TAXONOMY OF EVOLUTIONARY COMPUTING
ALGORITHMS
EvolutionaryComputing
Biological
Genetic
Algorithm
Genetic
Programming
Evo...
“NO FREE LUNCH” THEOREM*
Essentially states that if a search based algorithm is
better on some problems, it will be worse ...
CULTURAL ALGORITHM
FRAMEWORK*
INSPIRED BY THE PROBLEM SOLVING ABILITIES OF
CULTURE
Population Space
Belief Space
accept()
...
CULTURAL ALGORITHM TYPES–
F# (ALMOST)
type CA =
{
Population : Population
Network : Network
KnowlegeDistribution : Knowled...
THANK YOU contact: fwaris /
wayne.edu
Upcoming SlideShare
Loading in …5
×

Recognizing Patterns in Noisy Data using Trainable ‘Functional’ State Machines

808 views

Published on

Finite state machines (FSM) are a common technique for recognizing patterns over streams of data. However basic FSM cannot easily handle noisy data such as sensor data from a wearable device that is generated using human limb movement. This session presents a novel variation of FSM using techniques from functional programming to construct 'functional' state machines (FnSM) that are computationally as efficient as FSM but can handle probabilistic patterns almost as well as more elaborate techniques such as Hidden Markov Models.

A real-world application of FnSM is described; an Android smartwatch app that recognizes gestures performed using wrist and forearm movements. Also described is the use of an evolutionary computing algorithm to optimize the performance of the FnSM by selecting better parameter values for the various state-transition decisions.

We are increasingly living in a word where data processing needs to happen in real-time to enable quicker decision making. FSM are an essential technique for recognizing patterns over sequential data. This talk describes techniques that leverage FP to succinctly define complex FSM. Additionally, such constructed FSM can be trained with the help of a suitable evolutionary computing algorithm to also handle noisy data.

Published in: Software
  • Login to see the comments

  • Be the first to like this

Recognizing Patterns in Noisy Data using Trainable ‘Functional’ State Machines

  1. 1. RECOGNIZING PATTERNS IN NOISY DATA USING TRAINABLE ‘FUNCTIONAL’ STATE MACHINES Faisal Waris
  2. 2. OUTLINE
  3. 3. BACKGROUND & MOTIVATION - 3 TRENDS Android Auto Cadillac Gesture Pad BMW In-the-air Gestures
  4. 4. IDEA – WEARABLE DEVICE GESTURE CONTROL OF HMI
  5. 5. Twist Tap Swipe Left Right Escape Twist gesture recognized Escape gesture recognized IMPLEMENTATION Simple Gesture Vocabulary HMI Menu Hierarchy
  6. 6. GESTURE RECOGNITION ≈ PATTERN RECOGNITION IN NOISY (SEQUENTIAL) DATA Sensors Inherent Noise Fusion: Accelerometer, Gyro, Magnetometer Limb Movement Not precise Variations Vehicle Motion Acceleration Cornering Bumps
  7. 7. SEQUENTIAL DATA, PATTERN RECOGNITION TECHNIQUES
  8. 8. FINITE STATE MACHINES (FSM)
  9. 9. OUTLINE
  10. 10. PATTERN RECOGNITION WITH FSM FSM to recognize “…aaab…” pattern a1 a2 a3 a a a not a b
  11. 11. OUTLINE
  12. 12. ‘FUNCTIONAL’ STATE MACHINES (FNSM*) FSM expressed using functional programming constructs * FnSM is both singular and plural
  13. 13. NATURAL TRANSFORMATION TO F# (ML) type M<'s> = M of ('s -> M<'s>) let rec start = function | 'a' -> M a1 | _ -> M start and a1 = function | 'a' -> M a2 and a2 = function | 'a' -> M a3 and a3 = function | 'b' -> M ``end`` and ``end`` _ = M ``end`` a1 a2 a3 a a a not a b FnSMforrecognizingthe“aaab”pattern
  14. 14. STATE-SPACE EXPLOSION How about 3 ‘a’s and 3 ‘b’s in any order? Are 6 states enough? a1 a2 a3 b1 b2 b3 Need exponential # of states to recognize this pa
  15. 15. FNSM FOR 3 ‘A’S AND 3 ‘B’S IN ANY ORDERlet rec fStart = function | 'a' -> fAB 1 0 |> M | 'b' -> fAB 0 1 |> M | _ -> fStart |> M and fAB countA countB = function | 'a' when countA + 1 = 3 -> fB countB |> M | 'a' -> fAB (countA + 1) countB |> M | 'b' when countB + 1 = 3 -> fA countA |> M | 'b' -> fAB countA (countB + 1) |> M | _ -> failwith "invalid input" and fA count = function | 'a' when count + 1 = 3 -> fEnd |> M | 'a' -> fA (count + 1) |> M | _ -> failwith "invalid input" and fB count = function | 'b' when count + 1 = 3 -> fEnd |> M | 'b' -> fB (count + 1) |> M | _ -> failwith "invalid input" and fEnd _ = printf "done"; fEnd |> M 1 0
  16. 16. FNSM FOR WEARABLE DEVICE SENSOR DATA type SnsrEvent = { Snsr : SensorTypeEnum Ticks : long X : float; Y : float; Z : float} let rec start = function | {Snsr=LinearAcceleration; Z=z}  … | {Snsr=Gyroscope; Z=z; X=x}  … | {Snsr=s; Ticks=t} when …  … Matches an event where the sensor is Linear Acceleration and binds the Z-Axis value to the variable ‘z’ Given an event data structure of this type (record)... We can define FnSM with Pattern Matching code like this
  17. 17. F# FNSM MODULE IN PRACTICE - ANDROID ‘function-state’ recursive type Evaluation function takes a function-state and an event and returns the new function state PLUS whether something was recognized or not
  18. 18. OUTLINE
  19. 19. HANDLING NOISY DATA let rec start = function | {Snsr=LinearAcceleration; Z=z} when abs z > 5.0 // m/sec2  possible_swing z 1 |> M | _  start |> M and possible_swing prev_z count = function | {Snsr=LinearAcceleration; Z=z} when count < 30  let new_z = updateAvg (prev_z, count, z) possible_swing new_z (count + 1) |> M |{Snsr=LinearAcceleration; Z=z} -> when <avg z greater some threshold> -> … Compute avg. linear acceleration over some count number of events
  20. 20. PARAMETERIZE FNSM FOR TRAINABILITY let rec start cfg = function | {Snsr=LinearAcceleration; Z=z} when abs z > cfg.MIN_Z_THRESHOLD  possible_swing cfg z 1 | _  start and possible_swing cfg prev_z count = function | {Snsr=LinearAcceleration; Z=z} when count < cfg.COUNT_LIMIT  let curr_z = updateAvg (prev_z, count, z) possible_swing cfg curr_z (count + 1) | … FnSM parameterized with configuration data Optimal configuration values can be determined through machine learning
  21. 21. FNSM – OTHER FnSM as partially applied values in other FnSM  Hierarchical organization of state machines  Call-return semantics / modularization  Parallel execution of multiple ‘child’ state machines  E.g. recognize 1 of many gestures at the same time Reactive Extensions Operators  Write more compact and/or efficient custom operators for Rx  (used with Apache Storm topology)  Future: Project Brillo – Android for IoT
  22. 22. FNSM CONCEPTS – INTERIM SUMMARY
  23. 23. OUTLINE
  24. 24. FNSM TRAINING – PARAMETERS z_accel_front_thrsld = 2.74647951f; z_accel_back_thrsld = -4.25041485f; z_accel_avg_front_thrsld = 2.39706802f; z_accel_avg_back_thrsld = -4.60432911f; xy_accel_tolerance = 2.69525146f; avg_over_count = 1.0f; gstr_time_limit = 781282632L; xz_rot_tolerance = 2.24759841f; y_rot_tolerance = 3.57676268f; gstr_time_limit = 460464225L; x_accel_tolerance = 3.22107768f; y_accel_tolerance = 2.86686087f; ret_x_accel_tolerance = 2.9499104f; ret_y_accel_tolerance = 3.1444633f; xz_rot_tolerance = 2.97020698f; y_rot_tolerance = 0.526834428f; low_z_accel_limit = 3.09724712f Sample configuration parameters
  25. 25. FNSM TRAINING – PROCESS 2 Labelled training data: Perform each gesture X times and record sensor output Analyze gesture data and define parameterized FnSM for each gesture with initial (best guess) parameter values 1 3 Formulate a suitable optimization problem (fitness function) that uses the FnSM parameter values – this is key Run the evolutionary computing optimization algorithm (EA) – given the: 1) Training data 2) Parameterized gesture FnSM 3) Optimization formulation To search for parameter values that yield the best value of the objective function 4 arg 𝑚𝑎𝑥 𝑝 𝐹 𝑝 = W1 ∗ 𝑓𝑖𝑡 𝑝, 𝑔 𝑔 − W2 ∗ 𝑐𝑜𝑢𝑛𝑡𝐷 𝑝 − 𝐷𝐹 𝑝
  26. 26. FNSM TRAINING – OPTIMIZATION FORMULATION p = parm array values from a population individual g ∈ G ={Left, Right, Tap, Swipe}, the action gestures fit(p, g) = GC(g) – | count(p, g) – countNot (p, g) | is the fitness function for a single gesture. This function’s value is calculated by running the FnSM parameterized with p over the training set for g. (The Ideal score is 0 when all of the 10 gesture instances are recognized and none any other gestures are recognized). GC(g) = the # of gestures of type g in the training set for g (constant 10 for all gestures). count(p, g) = count of gesture of type g recognized when the p parameterized FnSM is run over the training set for g. (Ideal value is 10). countNot(p,g) = count of all gestures of type [G – {g}] recognized when the p parameterized FnSM is run over the training set for g. (Ideal value is zero). countD(p) = total count of any gesture in G recognized in the negative (driving) training set with a p parameterized FnSM. Ideal score is zero. This term is to reduce inadvertent recognition of any gestures. DF(p) = 𝑖,𝑗∈𝐺,𝑖 ≠𝑗 𝑓𝑖𝑡 𝑝, 𝑖 − 𝑓𝑖𝑡 𝑝, 𝑗 The sum of absolute differences between all individual fitness scores. This term favors p values that result in even gesture recognition performance across all gesture types in G. W1 and W2 are weighting factors to control the influence of some of the terms. In summary the fitness function is construed so that a perfect score of 120 is achieved when all 10 instances of a gesture type are recognized with the corresponding training set and none of any other type are recognized, for each of the action gesture types. arg 𝑚𝑎𝑥 𝑝 𝐹 𝑝 = W1 ∗ 𝑔 𝑓𝑖𝑡 𝑝, 𝑔 − W2 ∗ 𝑐𝑜𝑢𝑛𝑡𝐷 𝑝 − 𝐷𝐹 𝑝 Step (before / after CA) Fitness Score (max 120) Best guess parameter values -1.5 After CA optimization 92.5
  27. 27. SUMMARY Functional programming provides a powerful means of modeling FSM  State | Events | Transitions  function | pattern match | return a function Partial function application  Model complex patterns (state-space compression)  Handle probabilistic patterns through aggregation  Make FnSM amenable to training or machine learning through parameterization Training: Training data + Optimization + EA Human modelled, machine optimized, computationally efficient pattern recognition Upcoming paper in IEEE Conference Swarm / Human Blended Workshop  “Human-Defined, Machine-Optimized - Gesture Recognition using a mixed Approach”  SHBI2015 (Cleveland, Sept. 28, 29)
  28. 28. OUTLINE
  29. 29. A TAXONOMY OF EVOLUTIONARY COMPUTING ALGORITHMS EvolutionaryComputing Biological Genetic Algorithm Genetic Programming Evolutionary Algorithm Neural Net Artificial Immune System Social Ant Colony Optimization Particle Swarm Optimization Artificial Bee Colony Cultural Algorithm Molecular Chemical Reaction Optimization Simulated Annealing Hybrid algorithms + almost endless variations
  30. 30. “NO FREE LUNCH” THEOREM* Essentially states that if a search based algorithm is better on some problems, it will be worse on other problems (as compared to other search based algorithms) Explains the existence of the sheer variety of evolutionary computing algorithms Different search strategies are needed to tackle different problems * Wolpert & Macready (1997)
  31. 31. CULTURAL ALGORITHM FRAMEWORK* INSPIRED BY THE PROBLEM SOLVING ABILITIES OF CULTURE Population Space Belief Space accept() influence() Knowledge Sources Historical/Topological/Domain/Normative/S ituational *see Wikipedia page for introduction and further references Knowledg e Distributi on
  32. 32. CULTURAL ALGORITHM TYPES– F# (ALMOST) type CA = { Population : Population Network : Network KnowlegeDistribution : KnowledgeDistribution BeliefSpace : BeliefSpace AcceptanceFunction : Acceptance InfluenceFunction : Influence UpdateFunction : Update Fitness : Fitness } where: Topology = the network topology type of the population e.g. square, ring, global, etc. Knowledge = Situational | Historical | Normative | Topographical | Domain Individual = {Id:Id; Parms:Parm array; Fitness:float; KS:Knowledge} Id = int Parm = numeric value such as int, float, long, etc. Population = Individual array Network = Population  Id  Individual array Fitness = Parm array  float BeliefSpace = KnowledgeSource Tree Acceptance = BeliefSpace  Population  Individual array Influence = BeliefSpace  Population  Population Update = BeliefSpace  Individual array  BeliefSpace KnowledgeDistribution = Population  Network  Population KnowledgeSource = { Type : Knowledge Accept : Individual array  Individual array * KnowledgeSource Influence : Individual  Individual }
  33. 33. THANK YOU contact: fwaris / wayne.edu

×