Pattern matching over event streams is increasingly being employed in many areas including financial services and click stream analysis. Flink, as a true stream processing engine, emerges as a natural candidate for these usecases. In this talk, we will present FlinkCEP, a library for Complex Event Processing (CEP) based on Flink. At the conceptual level, we will see the different patterns the library can support, we will present the main building blocks we implemented to support them, and we will discuss possible future additions that will further enhance the coverage of the library. At the practical level, we will show how the integration of FlinkCEP with Flink allows the former to take advantage of Flink's rich ecosystem (e.g. connectors) and its stream processing capabilities, such as support for event-time processing, exactly-once state semantics, fault-tolerance, savepoints and high throughput.

1. 1
Kostas Kloudas
@KLOUBEN_K
BERLIN BUZZWORDS
JUNE 13, 2017
Complex Event Processing with
Flink The state of FlinkCEP

2. 2
Original creators of Apache
Flink®
Providers of the
dA Platform, a supported
Flink distribution

3. What is CEP?
3

4. CEP: Complex Event Processing
 Detecting event patterns
 Over continuous streams of events
 Often arriving out-of-order
4

5. CEP: Complex Event Processing
5
Input

6. CEP: Complex Event Processing
6
Pattern
Input

7. CEP: Complex Event Processing
7
Pattern
Output
Input

8. CEP: use-cases
 IoT
 Intrusion detection
 Inventory Management
 Click Stream Analysis
 Trend detection in financial sector
 ...yours?
8

9. What is Stream Processing?
9

10. Stream Processing
10
Computation
Computations on
never-ending
“streams” of events

11. Distributed Stream Processing
11
Computation
Computation
spread across
many machines
Computation Computation

12. Stateful Stream Processing
12
Computation
State
Result depends
on history of
stream

13. 13
Stream Processors are a natural fit
for CEP

14. FlinkCEP
14
Pattern
Output
FlinkCEP
Input

15. What does FlinkCEP offer?
15

16. Pattern Definition
16
Pattern

17. Pattern Definition
 Composed of Individual Patterns
• P1(shape == rectangle)
• P2(shape == triangle)
17
Pattern
P2
P1

18. Pattern Definition
 Composed of Individual Patterns
• P1(shape == rectangle)
• P2(shape == triangle)
 Combined by Contiguity Conditions
• ...later
18
Pattern
P2
P1

19. FlinkCEP Individual Patterns
 Unique Name
 Condition : which elements to accept
• Simple e.g shape == rectangle
• Iterative e.g rectangle.surface < triangle.surface
 Quantifiers (or not)
• Looping/Optional oneOrMore(),times(#),optional()
19
Pattern
P2
P1

20. FlinkCEP Complex Patterns
 Combine Individual Patterns
 Contiguity Conditions
• how to select relevant events given an input mixing
relevant and irrelevant events
 Time Constraints
• within(time) e.g. all events have to come within 24h
20
Pattern
P2
P1

21. FlinkCEP Contiguity Conditions
21
Pattern
Input

22. FlinkCEP Contiguity Conditions
22
Pattern
OutputInput
Strict Contiguity
• matching events strictly follow each other

23. FlinkCEP Contiguity Conditions
23
Pattern
OutputInput

24. FlinkCEP Contiguity Conditions
24
Pattern
Relaxed Contiguity
• non-matching events to simply be ignored
Input Output

25. FlinkCEP Contiguity Conditions
25
Pattern
Input Output

26. FlinkCEP Contiguity Conditions
26
Pattern
Input Output

27. FlinkCEP Contiguity Conditions
27
Pattern
Input Output
Non-Deterministic Relaxed Contiguity
• allows non-deterministic actions on relevant events

28. FlinkCEP Contiguity Conditions
28
Pattern
NOT patterns:
• for strict and relaxed contiguity
• for cases where an event should invalidate a match
Input

29. FlinkCEP Summary
29
 Quantifiers
• oneOrMore(), times(), optional()
 Conditions
• Simple & Iterative
 Time Constraints
• Event and Processing time
 Different Contiguity Constraints
• Strict, relaxed, non-deterministic relaxed, NOT

30.  Trace all shipments which:
• start at location A
• have at least 5 stops
• end at location B
• within the last 24h
30
Running Example: retailer
A
B
M1
M2
M3
M4
M5

31.  Trace all shipments which:
• start at location A
• have at least 5 stops
• end at location B
• within the last 24h
31
Observation A Individual Patterns
Start
End
Mid
ev.from == A
ev[i].from
==
ev[i-1].to
ev.to == B
&&
size(“mid”) >= 5

32. 32
Observation B Quantifiers
 Start/End: single event
 Middle: multiple events
• .oneOrMore()
Start
End
Mid
ev.from == A
ev[i].from
==
ev[i-1].to
ev.to == B
&&
size(“mid”) >= 5

33. 33
Observation C Conditions
 Start -> Simple
• properties of the event
 Middle/End -> Iterative
• Depend on previous events
Start
End
Mid
ev.from == A
ev[i].from
==
ev[i-1].to
ev.to == B
&&
size(“mid”) >= 5

34. 34
 Trace all shipments which:
• start at location A
• have at least 5 stops
• end at location B
• within the last 24h
Observation D Time Constraints
Start
End
Mid
ev.from == A
ev[i].from
==
ev[i-1].to
ev.to == B
&&
size(“mid”) >= 5

35. 35
 We opt for relaxed continuity
Observation E Contiguity

36. Pattern<Event, ?> pattern = Pattern
.<Event>begin("start")
.where(mySimpleCondition)
.followedBy ("middle")
.where(myIterativeCondition1)
.oneOrMore()
.followedBy ("end”)
.where(myIterativeCondition2)
.within(Time.hours(24))
Start
Middle
End
Running Example Individual Patterns

37. Pattern<Event, ?> pattern = Pattern
.<Event>begin("start")
.where(mySimpleCondition)
.followedBy ("middle")
.where(myIterativeCondition1)
.oneOrMore()
.followedBy ("end”)
.where(myIterativeCondition2)
.within(Time.hours(24))
Start
Middle
End
Running Example Quantifiers

38. Pattern<Event, ?> pattern = Pattern
.<Event>begin("start")
.where(mySimpleCondition)
.followedBy ("middle")
.where(myIterativeCondition1)
.oneOrMore()
.followedBy ("end”)
.where(myIterativeCondition2)
.within(Time.hours(24))
Start
Middle
End
Running Example Conditions

39. Pattern<Event, ?> pattern = Pattern
.<Event>begin("start")
.where(mySimpleCondition)
.followedBy ("middle")
.where(myIterativeCondition1)
.oneOrMore()
.followedBy ("end”)
.where(myIterativeCondition2)
.within(Time.hours(24))
Start
Middle
End
Running Example Time Constraint

40. Running Example Pattern Integration
Pattern<Event, ?> pattern = ...
PatternStream<Event> patternStream = CEP.pattern(input, pattern);
DataStream<Alert> result = patternStream.select (
new PatternSelectFunction<Event, Alert>() {
@Override
public Alert select(Map<String, List<Event>> pattern) {
return parseMatch(pattern);
}
);

41. Running Example Pattern Integration
Pattern<Event, ?> pattern = ...
PatternStream<Event> patternStream = CEP.pattern(input, pattern);
DataStream<Alert> result = patternStream.select (
new PatternSelectFunction<Event, Alert>() {
@Override
public Alert select(Map<String, List<Event>> pattern) {
return parseMatch(pattern);
}
);

42. Running Example Pattern Integration
Pattern<Event, ?> pattern = ...
PatternStream<Event> patternStream = CEP.pattern(input, pattern);
DataStream<Alert> result = patternStream.select (
new PatternSelectFunction<Event, Alert>() {
@Override
public Alert select(Map<String, List<Event>> pattern) {
return parseMatch(pattern);
}
);

43. Documentation
 FlinkCEP documentation:
https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/libs/cep.html
43

44. 4
Thank you!
@KLOUBEN_K
@ApacheFlink
@dataArtisans

45. 45
Stream Processing
and Apache Flink®'s
approach to it
@StephanEwen
Apache Flink PMC
CTO @ data ArtisansFLINKFORWARD IS COMING BACKTO BERLIN
SEPTEMBER11-13, 2017
BERLIN.FLINK-FORWARD.ORG -

46. We are hiring!
data-artisans.com/careers