Fast data is big data, continuously streaming in, from which information is to be learned in (near) real time. This session demonstrates how Oracle Event Processing is used to analyze live streams of data to find patterns, deviations, and aggregates. The findings are reported in the form of business events that are pushed in live dashboards to Oracle Business Activity Monitoring, which also evaluates business rules on the business events and takes action when required. Examples to be demonstrated in this session include car sensors, website traffic, Twitter feeds, and bank run detection. Oracle SOA Suite 12c, WebSockets, Oracle Application Development Framework (Oracle ADF) active data visualization tools components, and JMS are used to process, forward, and act.

1. How Fast Data Is Turned into Fast
Information and Timely Action
Lucas Jellema
Oracle OpenWorld 2014, San Francisco, CA, USA

2. 2
Audience Challenge

3. 3
Audience Challenge

4. 4
Audience Challenge

5. 5
Audience Challenge

6. 6
Audience Challenge

7. 7
Audience Challenge

8. 8
Filter
Pattern Detection
Agregate

9. 9
Fast Data Example
14,0
16,1
14,1
16,1
16,0
13,1
14,0
16,0
13,1
13,0
14,1
16,0
14,1
13,0
14,1
16,0
13,1
14,0
Smart Processing
• Information
• Conclusion
• Alert
• Recommendation
• Action

10. 10
Demonstration:
Live Tennis
• Tennis Tournament
• Many matches played in parallel
• The data that is produced:
– At a rate of up to 10 events/minute
Match Id, Player [who scored]
14,0
16,1
14,1
16,1
16,0
13,1
14,0

11. 11
Demonstration: Live Tennis
• The information, conclusions &
actions we are looking for:
– Scoreboard per game, set, match
– Match start and completion (action:
inform next players for that court)
– Interrupted match (action: go and check
out the reason for the interruption)
Fast Data
Smart Processing
• Scoreboard
• Match start and
completion
• Interrupted match

12. 12
Real Time – from event to UI
Push through Web Sockets
Fast Data
Smart Processing
Oracle Event
Processor
WebLogic
event
WebSocket
Server
msg
msg
CQL queries

13. 13
WebSocket Powered
Scoreboard

14. 14
OEP application to process
fast tennis data
• Preparation
– Define event definitions
– Create local, in memory cache with static, enriching data
– Gather (in this case generate) tennis data through adapter
– Create Event Sink to consume all findings and publish to console
TennisMatch
Event
matchId
player

15. 15
Match Level events

16. 16
Rally’s to games
- The first player to have
won more than 4
points
- and have won two or
more points more than
his opponent
TennisMatch
Event
matchId
player

17. 17
Detect interrupted matches by
‘finding’ missing events
• When a match is interrupted, obviously no more ‘rally point events’ are
produced
• Detecting the absence of these events for a match [that has begun] is
equivalent to detecting an interruption of the match
– Unless the match is complete because someone won

18. 18
Detect interrupted matches by
‘finding’ missing events

19. 19
Complete EPN diagram for
Tennis Tournament Processor
• A single OEP application that consumes fine grained rally point events
and performs three-stage aggregation and enrichment
TennisMatch
Event
matchId
player
New
Match
Match
Finish
Interrupted
Match
Set
Game Won
Won

20. Overview
• What is [special about] Fast Data?
– Continuous, Volume|Velocity|Variety, Real Time
• Challenges
– Volatile, non persistent
– Data => Information, Conclusion, Alert, Recommendation, Action
• Strategies
– Smart gathering
– Discard – filter, aggregate, pattern
(and also look for missing events!)
– Promote (process, enrich)
– Visualize
• Technology/Tools
• Demonstration/Cases

21. 21
Fast Data
• Tweet
• Feed
• Beat
• Signal
• Measurement
• Message
• Mail
• Notification
• Tick
• Pulse

22. 22
New theme
(that brings it all together)

23. 23
Some event producing devices

24. 24

25. Most of these events….

26. 26

27. 27
Fast Data Processing
Fast Data
Smart Processing
• Information
• Conclusion
• Alert
• Recommendation
• Action

28. 28
Fast Data Processing
Multi-stage cleansing & aggregation
Fast Data
Smart Processing
• Information
• Conclusion
• Alert
• Recommendation
• Action

29. 29
Typical Flow and
Additional Challenge…
Business event
Business Value
Data captured
Analysis completed
Action taken
Fragmented
event entities
TIME

30. 30
The V-factor
Volume
Velocity
Variety
VALUE

31. 31
Key strategy
• Discard – as early as possible (close to the source)
– Ignore irrelevant events
– Filter out unneeded attributes
– Takes samples instead of entire stream
– Aggregate: merge multiple, correlated events into one

32. 32
Fast Data Processing:
Oracle Event Processor
Fast Data
Smart Processing
Oracle Event
Processor
RMI
File
REST
HTTP Channel
JMS
Database
Custom (Java)
SOA Suite EDN
Coherence
JMX
QuickFix (financial)
RMI
File
REST
HTTP Channel
JMS
Business Rule
Database
Custom (Java)
SOA Suite EDN
Coherence
JMX

33. 33
Oracle Event Processor
events
Event Processor
• Light weight, real-time (sub-sub-second), in-memory, continuous query engine
– Available in embedded form – with corresponding licence
• Interacts with many different channels – inbound and outbound
• Has internal caches to enrich events and temporarily retain events
• Uses CQL to:
– Filter, aggregate, enrich and detect patterns (including missing events)

34. 35
Fast Data Processing
Fusion Middleware Tooling
Fast Data
SOA Suite
12c EDN
Smart Processing
Coherence
Oracle Event
Processor
RMI
File
REST
HTTP Channel
JMS
Database
JMX
Custom (Java)
RMI
File
REST
HTTP Channel
JMS
Business Rule
JMX
Database
Custom (Java)

35. 36
Fast Data Processing
Fusion Middleware Tooling
Fast Data
Smart Processing
• Information
• Conclusion
• Alert
• Recommendation
• Action
OEP
BAM
Coherence SOA ADF
Suite
EDN
BPM
Suite
BPEL
Task
BI
RTD
ODI
Golden
Gate
NoSQL

36. 37
Business User Friendly Exploration of
Fast Data: Stream Explorer

37. 38
Credit Card Theft Detection
• Several situations in the past
– Credit card is stolen in the main terminal building
– Several purchases are made in shops on the way from that area to the main exit
• Purchases between $200-$500 dollar
• Purchases made within 5 minutes of each other
• Sometimes the purchases are made in not entirely the direct route to the exit
EXIT
Main
Terminal

38. 39
Credit Card Theft Detection
• Several situations in the past
– Credit card is stolen in the main terminal building
– Several purchases are made in shops on the way from that area to the main exit
• Purchases between $200-$500 dollar
• Purchases made within 5 minutes of each other
• Sometimes the purchases are made in not entirely the direct route to the exit
• To catch the perpetrator
– Consume the credit card purchase event stream for airport shops
– Spot situations where three or more purchases of $200-$500 are made within 5
minutes from each other and roughly in the terminal => exit physical order
– Publish an event to alert security staff
• To watch for any further purchases with that credit card
• To inform show staff for that credit card
• To send staff to the exit to try and apprehend the thief
(perhaps based on the shopping bags he is carrying
from the shops he bought stuff at)

39. 40
Catch me if you can
EXIT
Main
Terminal

40. 41
Catch me if you can
EXIT
Main
Terminal
$440
$300
$380
$250

41. 42
CQL to detect
‘funny string of transactions’

42. 43
Real Time – from Event to Task
OEP => SOA Suite 12c EDN
Fast Data
Smart Processing
Oracle Event
Processor
SOA Suite 12c
EDN
event event

43. 44
Real Time – from Event to Task
OEP => SOA Suite 12c EDN
Fast Data
Smart Processing
Oracle Event
Processor
SOA Suite 12c
EDN
BPEL Task
BPMN
Medi-ator
event event
event

44. 45
From OEP finding to EDN Business
Event triggering the SOA Suite

45. 46
Human consumers
• Slow at data processing
• Not electronically connected
• Visually oriented (1 picture > 1000 words)
• Frequently (though perhaps decreasingly so) the actor or decision maker
• Interact along human communication channels
• Use visualization to present findings, conclusions, recommended actions
– And as a second tier of fast data processing:
Highlight (filter), aggregate, patterns, extrapolate/interpolate, missing elements
• Sometimes take over from humans and just take action

46. 47
Visualize and Aggregate

47. 48
Real Time – from event to UI
Business Activity Monitoring
Fast Data
Smart Processing
Oracle Event
Processor
WebLogic
JMS
event
msg
msg BAM

48. 49
Real Time – from event to UI
ADF DVT Visualizations
Fast Data
Smart Processing
Oracle Event
Processor
WebLogic
JMS
event
msg
ADF
DVT
msg

49. 50
Visualize physical locations of
[string of] suspicious transactions

50. 51
Summary
• Fast Data (events): Vast, Continuous, Velocity, Variety
– Wanted: Near real time conclusions, recommendations, alerts, actions
• Strategy:
– Discard – as early as possible (Filter, Aggregate)
– Enrich, Correlate and Pattern Match, Missing Events, Retain, Publish higher level,
more coarse grained business event
– Repeat this cycle multiple times (such as rally point, game, set, match)
• Technology for Fast Data processing: Oracle Event Processor & CQL
– Interacts with JMS, EDN, RMI, HTTP (/REST), JMX, Database, Coherence
• New: Stream Explorer – business friendly, industry pattern based fast
data explorations and visualization
• To assist humans in Fast Data and Information Processing: Visualization
– Filter, Aggregate, Enrich, Pattern Match (1 picture > 1000 words)
– Technology: BAM (Dashboard and Rule processing), ADF Data Visualization
– Also: turn findings into actions using Human Task, BPEL and BPM via the SOA Suite
12c Event Delivery Network (EDN)

52. 53
Fast Data Example
14,0
16,1
14,1
16,1
16,0
13,1
14,0
16,0
13,1
13,0
14,1
16,0
14,1
13,0
14,1
16,0
13,1
14,0
Smart Processing
Oracle Event
Processor

53. 54
Demonstration:
Live Tennis
• Tennis Tournament
• Many matches played in parallel
• The data that is produced:
– At a rate of up to 10 events/minute
Match Id, Player [who scored]
14,0
16,1
14,1
16,1
16,0
13,1
14,0

54. 55
Demonstration: Live Tennis
• The information, conclusions &
actions we are looking for:
– Scoreboard per game, set, match
– Match start and completion (action:
inform next players for that court)
– Interrupted match (action: go and check
out the reason for the interruption)
Fast Data
Smart Processing
• Scoreboard
• Match start and
completion
• Interrupted match

55. 56
OEP application to process
fast tennis data
• Preparation
– Define event definitions
– Create local, in memory cache with static, enriching data
– Gather (in this case generate) tennis data through adapter
– Create Event Sink to consume all findings and publish to console
TennisMatch
Event
matchId
player

56. 57
Simple Time-slice Aggregation
• Produce aggegrates once every 30 seconds
– Count number of matches going on currently (meaning: in the last 30 seconds)
– Calculate average time per rally (over the last 30 seconds)
– Count total number of points played (over the last 30 seconds)

57. 60
Match Level events

58. 61
Rally’s to games
- The first player to have
won more than 4
points
- and have won two or
more points more than
his opponent
TennisMatch
Event
matchId
player

59. 62
Games to Sets
- The first player to have
won more than 5
games
- and have won two or
more games more
than his opponent

60. 63
Detect interrupted matches by
‘finding’ missing events
• When a match is interrupted, obviously no more ‘rally point events’ are
produced
• Detecting the absence of these events for a match [that has begun] is
equivalent to detecting an interruption of the match
– Unless the match is complete because someone won

61. 64
Detect interrupted matches by
‘finding’ missing events

62. 65
Complete EPN diagram for
Tennis Tournament Processor
• A single OEP application that consumes fine grained rally point events
and performs three-stage aggregation and enrichment
TennisMatch
Event
matchId
player
New
Match
Match
Finish
Interrupted
Match
Set
Game Won
Won