Theory Meets Reality—Large Scale Frequent Pattern Mining with Apache Spark in the Real World with Kexin Xie and Wanderley Liu

Numberofcombinations
Number of items in set
8
256

8 20
256
1,048,576

8 20 140,000
256
1,048,576
???

Theory Meets Reality
Large Scale Frequent Pattern Mining with Apache Spark in the Real World
Kexin Xie, Architect of Marketing Cloud Einstein
kexin.xie@salesforce.com, @realstraw
Wanderley Liu, Senior Data Science Engineer
wanderley.liu@salesforce.com

Marketing Cloud Einstein Journey Insights
Track the entire consumer journey
Gather online and offline interactions to stitch together a
complete view of the consumer
Discover the optimal path to conversion
Use AI to analyze all journey permutations and
automatically recommend the best channels, offers and
sequences that lead to conversion
Learn how customers are actually interacting with your brand
GA

What is
Frequent Pattern
Mining
Mine Shaft Mural Painting by Frank Wilson

User Items
u-1 a, b
u-2 b, c, d
u-3 a, c, d, e
u-4 a, d, e
u-5 a, b, c
u-6 a, b, c, d
u-7 a
u-8 a, b, c
u-9 a, b, d
u-10 b, c, e

item support
a 8
b 7
c 6
d 5
e 3
User Items
u-1 a, b
u-2 b, c, d
u-3 a, c, d, e
u-4 a, d, e
u-5 a, b, c
u-6 a, b, c, d
u-7 a
u-8 a, b, c
u-9 a, b, d
u-10 b, c, e

item support
a 8
b 7
c 6
d 5
e 3
item support
a, b 5
a, c 4
a, d 4
a, e 2
... ...
User Items
u-1 a, b
u-2 b, c, d
u-3 a, c, d, e
u-4 a, d, e
u-5 a, b, c
u-6 a, b, c, d
u-7 a
u-8 a, b, c
u-9 a, b, d
u-10 b, c, e

item support
a 8
b 7
c 6
d 5
e 3
item support
a, b 5
a, c 4
a, d 4
a, e 2
... ...
Min Support = 4
User Items
u-1 a, b
u-2 b, c, d
u-3 a, c, d, e
u-4 a, d, e
u-5 a, b, c
u-6 a, b, c, d
u-7 a
u-8 a, b, c
u-9 a, b, d
u-10 b, c, e

item support
a 8
b 7
c 6
d 5
e 3
item support
a, b 5
a, c 4
a, d 4
a, e 2
... ...
Min Support = 4
item support
a 8
b 7
c 6
d 5
e 3
item support
a, b 5
a, c 4
a, d 4
a, e 2
... ...
User Items
u-1 a, b
u-2 b, c, d
u-3 a, c, d, e
u-4 a, d, e
u-5 a, b, c
u-6 a, b, c, d
u-7 a
u-8 a, b, c
u-9 a, b, d
u-10 b, c, e

item support
a 8
b 7
c 6
d 5
e 3
item support
a, b 5
a, c 4
a, d 4
a, e 2
... ...
Min Support = 4
item support
a 8
b 7
c 6
d 5
e 3
item support
a, b 5
a, c 4
a, d 4
a, e 2
... ...
User Items
u-1 a, b
u-2 b, c, d
u-3 a, c, d, e
u-4 a, d, e
u-5 a, b, c
u-6 a, b, c, d
u-7 a
u-8 a, b, c
u-9 a, b, d
u-10 b, c, e
L1 Patterns
L2 Patterns

A-priori Principle
A Priori in Berkeley, CA
“All sub-patterns of a frequent pattern are
frequent”

Min Support = 4
item support
a 8
b 7
c 6
d 5
e 3
item support
a, b ?
a, c ?
a, d ?
a, e ?
... ...

Min Support = 4
item support
a 8
b 7
c 6
d 5
e 3
item support
a, b ?
a, c ?
a, d ?
a, e ?
... ...
Min Support = 6
item support
a 8
b 7
c 6
d 5
e 3
item support
a, b ?
a, c ?
a, d ?
a, e ?
... ...

item support
a 8
b 7
c 6
d 5
e 3
root
a: 8 b: 2
b: 5
c: 3
d: 1
d: 1
c: 1
d: 1
e: 1
c: 2
d: 1
d: 1
e: 1
e: 1
Header Table

item support
a 8
b 7
c 6
root
a: 8 b: 2
b: 5
c: 3 c: 1
c: 2
a, b, c 3
a, c 1
b, c 2
c 6
FP Results
Header Table

FP-Tree | c
a, b 3
a 1
b 2
item support
b 5
a 4
root
b: 5
a: 4
c 6
FP Results
Header Table

c 6
item support
b 5
a 4
root
b: 5
a: 4
a 4
b 5
a, b 4
FP-Tree | c
FP Results
Header Table

c 6
a, c 4
b, c 5
a, b, c 4
item support
b 5
a 4
root
b: 5
a: 4
a 4
b 5
a, b 4
FP-Tree | c
FP Results
Header Table

Scaling Up
https://www.firestock.ru/strela-na-grafike-arrow-on-the-chart/

item support
a 8
b 7
c 6
d 5
e 3
root
a: 8 b: 2
b: 5
c: 3
d: 1
d: 1
c: 1
d: 1
e: 1
c: 2
d: 1
d: 1
e: 1
e: 1
User Items
u-1 a, b
u-2 b, c, d
u-3 a, c, d, e
u-4 a, d, e
u-5 a, b, c
u-6 a, b, c, d
u-7 a
u-8 a, b, c
u-9 a, b, d
u-10 b, c, e
Header Table

item support
a 8
b 7
c 6
d 5
e 3
user items
u-1 a, b
u-2 b, c, d
u-3 a, c, d, e
u-4 a, d, e
u-5 a, b, c
u-6 a, b, c, d
u-7 a
u-8 a, b, c
u-9 a, b, d
u-10 b, c, e
Header Table

item support
a 8
b 7
c 6
d 5
e 3 a
[a], b
[a, b], c
[a, b, c], d
[a, b, c, d], e
Header Table
user items
u-1 a, b
u-2 b, c, d
u-3 a, c, d, e
u-4 a, d, e
u-5 a, b, c
u-6 a, b, c, d
u-7 a
u-8 a, b, c
u-9 a, b, d
u-10 b, c, e

item support
a 8
b 7
c 6
d 5
e 3 a
[a], b
[a, b], c
[a, b, c], d
[a, b, c, d], e
u-2 b, c, (d)
u-3 a, c, (d, e)
u-5 a, b, c
u-6 a, b, c, (d)
u-8 a, b, c
u-10 b, c, (e)
Header Table
user items
u-1 a, b
u-2 b, c, d
u-3 a, c, d, e
u-4 a, d, e
u-5 a, b, c
u-6 a, b, c, d
u-7 a
u-8 a, b, c
u-9 a, b, d
u-10 b, c, e

item support
a 8
b 7
c 6
d 5
e 3 a
[a], b
[a, b], c
[a, b, c], d
[a, b, c, d], e
u-2 b, c, (d)
u-3 a, c, (d, e)
u-5 a, b, c
u-6 a, b, c, (d)
u-8 a, b, c
u-10 b, c, (e)
Header Table
user items
u-1 a, b
u-2 b, c, d
u-3 a, c, d, e
u-4 a, d, e
u-5 a, b, c
u-6 a, b, c, d
u-7 a
u-8 a, b, c
u-9 a, b, d
u-10 b, c, e
u-3 a, c, d, e
u-4 a, d, e
u-10 b, c, e

item support
a 8
b 7
c 6
d 5
e 3 a
[a], b
[a, b], c
[a, b, c], d
[a, b, c, d], e
u-2 b, c, d
u-3 a, c, d, (e)
u-4 a, d, (e)
u-6 a, b, c, d
u-9 a, b, d
Header Table
u-3 a, c, d, e
u-4 a, d, e
u-10 b, c, e
user items
u-1 a, b
u-2 b, c, d
u-3 a, c, d, e
u-4 a, d, e
u-5 a, b, c
u-6 a, b, c, d
u-7 a
u-8 a, b, c
u-9 a, b, d
u-10 b, c, e
u-2 b, c, (d)
u-3 a, c, (d, e)
u-5 a, b, c
u-6 a, b, c, (d)
u-8 a, b, c
u-10 b, c, (e)

item support
a 8
b 7
c 6
d 5
e 3 a
[a], b
[a, b], c
[a, b, c], d
[a, b, c, d], e
u-1 a, b
u-2 b, (c, d)
u-5 a, b, (c)
u-6 a, b, (c, d)
u-8 a, b, (c)
u-9 a, b, (d)
u-10 b, (c, e)
Header Table
user items
u-1 a, b
u-2 b, c, d
u-3 a, c, d, e
u-4 a, d, e
u-5 a, b, c
u-6 a, b, c, d
u-7 a
u-8 a, b, c
u-9 a, b, d
u-10 b, c, e
u-2 b, c, (d)
u-3 a, c, (d, e)
u-5 a, b, c
u-6 a, b, c, (d)
u-8 a, b, c
u-10 b, c, (e)
u-2 b, c, d
u-3 a, c, d, (e)
u-4 a, d, (e)
u-6 a, b, c, d
u-9 a, b, d
u-3 a, c, d, e
u-4 a, d, e
u-10 b, c, e

Number of rows
Numberofitems
u-1 a, b
u-2 b, (c, d)
u-5 a, b, (c)
u-6 a, b, (c, d)
u-8 a, b, (c)
u-9 a, b, (d)
u-10 b, (c, e)
u-2 b, c, (d)
u-3 a, c, (d, e)
u-5 a, b, c
u-6 a, b, c, (d)
u-8 a, b, c
u-10 b, c, (e)
u-2 b, c, d
u-3 a, c, d, (e)
u-4 a, d, (e)
u-6 a, b, c, d
u-9 a, b, d
u-3 a, c, d, e
u-4 a, d, e
u-10 b, c, e

Distribute rows to executors
Build FP-Trees on each node
and mine for patterns
Collect patterns
Build FP-tree header table

Distribute rows to executors
Build FP-Trees on each node
and mine for patterns
Collect patterns
val headerTable = data
.flatMap(_.items.map(_ -> 1L))
.reduceByKey(_ + _)
.filter(isFrequent)
.collect
.sorted
data
.flatMap(filterDataBasedHeaderTable (headerTable))
.groupByKey
.flatMap { case (k, rows) =>
mineForPatternsFor (k, rows)
}
.collect // If necessary
Build FP-tree header table

Minimum support
https://www.maxpixel.net/static/photo/1x/Cogs-Gears-Technical-Wh
eel-Cogwheel-Gearwheel-2279289.jpg

Differential Minimum Support (DMS)
Classify Items Into
Categories
Compute Min Support
Per Category
Run FP with Multiple
Min Supports

Pattern Frequency Test
CONDITION 1: Pattern Support ≥ Pattern Min Support
Pattern min support is defined as the lowest category minsup, given all items in the pattern
CONDITION 2 - Apriori Principle (Recursive)
If a pattern is frequent, all sub-patterns must be frequent

Condition 1: Pattern Support > Pattern Minimum Support
Item Cat Minsup Condition 1
A Common 100k
B Common 100k
C Rare 1k
Pattern Support Minsup Condition 1
A B 80k
A C 4k
B C 3k
A B C 2k

Condition 1: Pattern Support > Pattern Minimum Support
A Common 100k
B Common 100k
C Rare 1k
A B 80k 100k
A C 4k 1k
B C 3k 1k
A B C 2k 1k

Condition 1: Pattern support > Lowest minsup given all items in the pattern
Pattern Frequency
A Common 100k
B Common 100k
C Rare 1k
A B 80k 100k
A C 4k 1k
B C 3k 1k
A B C 2k 1k

Condition 2 - A priori principle
A Common 100k
B Common 100k
C Rare 1k
A B 80k 100k
A C 4k 1k
B C 3k 1k
A B C 2k 1k

val fpTreeResults = data
.flatMap(filterDataBasedHeaderTable(headerTable))
.groupByKey
mineForPatternsFor (k, rows)
}

val catMinsupMap = sc.broadcast( computeCatMinSup (data))
.groupByKey
mineForPatternsFor (k, rows, catMinsupMap.value )
}
CONDITION 1

val catMinsupMap = sc.broadcast( computeCatMinSup (data))
.groupByKey
mineForPatternsFor (k, rows, catMinsupMap.value )
}
val patternsMap = sc.broadcast(fpTreeResults.keys.collect)
fpTreeResults
.filter { case (pattern, support) =>
pattern.subsets.subsetOf (patternMap.value)
}
CONDITION 1
CONDITION 2

Not the end of the story ...
https://w-dog.net/wallpaper/nature-night-star-tree-trees-stars-background-wal
lpaper-widescreen-full-screen-hd-wallpapers-fullscreen/id/308950/

Low Level Optimization
• Handled case where array length > Integer.MAX_VALUE
Result Set Compaction
• Remove redundant and noisy result sets
• Very efficient compaction - 95% without loss of information
Result Set Ranking
• Score patterns with multiple criteria
Items with Feature Set
• Not only which combinations work best, but what makes them work best
• Well received feature, direct feedback on strategy

Theory Meets Reality—Large Scale Frequent Pattern Mining with Apache Spark in the Real World with Kexin Xie and Wanderley Liu

Theory Meets Reality—Large Scale Frequent Pattern Mining with Apache Spark in the Real World with Kexin Xie and Wanderley Liu

Recommended

Recommended

More Related Content

More from Databricks

More from Databricks (20)

Recently uploaded

Recently uploaded (20)

Theory Meets Reality—Large Scale Frequent Pattern Mining with Apache Spark in the Real World with Kexin Xie and Wanderley Liu