This document provides an overview of complex event processing (CEP) and Esper, an open source CEP engine. It defines CEP as analyzing and controlling interrelated events to understand modern distributed systems. Esper makes it easier to build CEP applications by providing an event processing language (EPL) to define event types, continuous queries, and event patterns similar to SQL. It supports features like filtering, aggregation, windows, correlations, and pattern detection on streaming event data. While powerful, Esper has limitations around memory usage, resilience, and distribution that must be considered.

1. Complex Event
Processing with Esper
@antonioalegria

2. Complex Event
Processing?
CEP

3. “Complex Event is an event
that could only happen if lots
of other events happened”
“CEP is a set of tools and
techniques for analyzing and
controlling the complex series
of interrelated events that drive
modern distributed information
systems”
David Luckham, 2002

4. Example
• Church bell ringing
• Appearance of a man in a tuxedo
• Appearance of a woman in a white gown
• Rice flying through the air

5. Example
• Church bell ringing
• Appearance of a man in a tuxedo
• Appearance of a woman in a white gown
• Rice flying through the air
Wedding has happened!

6. CEP Use Cases
• Are our business processes running on
time and correctly?
• Can we detect an opportunity for arbitrage
in our trading department?
• Are we servicing our call center customer’s
requests in a timely fashion?
• Was there a breach in our network?

7. It’s not a technology

8. It’s a Buzzword
like SOA!

9. It’s an Architectural
Pattern

11. What do you need for
CEP?

12. Event driven

13. (soft) Real-time

14. (soft) Real-time
Right

15. Across all layers of
organization

16. Event Aggregation

17. Event Relationships
• Causality
• Membership
• Timing

18. Event Patterns

19. Domain Specific
Language
for Event Processing

20. What you need for
CEP
• Event Driven
• Right-time
• Across all layers
• Aggregation, Correlation & Traceability
• Patterns
• DSL

21. Common CEP
Operations
• Windowing
• Transformation
• Aggregation/Grouping
• Merging/Union
• Filtering
• Sorting
• Correlation
• Pattern Detection

22. Esper
http://esper.codehaus.org

24. Esper makes it easier to
build a CEP app

25. Not meant to replace
Databases

26. But some parallels can
be made

27. Esper DB
• Stores queries • Stores data
• Continuous queries • On-demand queries
• Time is a dimension • Time is a data type

28. Esper DB
• EPL • SQL
• Event Streams • Tables
• Events • Rows

29. Esper Processing Model

30. EPL
Event Processing Language

31. Event Definition (1/2)
create schema Event (
id string, // Event unique identifier
ts long // Timestamp (milliseconds)
);
create schema Tweet (
user string, // username (e.g. ‘codebits’)
text string, // actual tweet
retweet_of string // references a Tweet.id
) inherits Event;

32. Event Definition (2/2)
create schema Hashtag (
tweet_id string, // references a Tweet.id
user string,
value string
) inherits Event;
// Create Url and Mention event types as a copy of Hashtag
create schema Url() copyfrom Hashtag;
create schema Mention() copyfrom Hashtag;

33. Looks like SQL...
// All events
select * from Event;
// Only tweets
select user, text as status
from Tweet;

34. Filtering
// Tweets from @codebits
select * from Tweet(user = 'codebits');
// Another way to do it
select * from Tweet where user = 'codebits';
// All occurrences of #codebits not posted by @codebits
select user,
value as hashtag,
current_timestamp() as ts
from Hashtag(value = 'codebits' and user != 'codebits');

35. Stream Creation and Redirection
insert into CodebitsTweets
select * from Tweet(user = ‘codebits’);
select * from CodebitsTweets;

36. Aggregation
insert into UrlsPerSecond
select count(*) as count from Url.win:time_batch(1 sec);
// Every second (driven by above rule) calculate for last minute
// - average Urls tweeted
// - total Urls tweeted
select avg(count), sum(count)
from UrlsPerSecond.win:length(60);

37. Grouping
select value as hashtag, count(*)
from Hashtag(value != null).win:time(30 seconds)
group by value;

38. Simple Event Views
select * from Tweet.win:time(5 min);
select * from Tweet.win:time_batch(1 hour);
select * from Tweet.win:length(10);
select * from Tweet.win:length_batch(10);

39. Other Standard Event Views
// Don’t use system clock, use event stream property
select * from Tweet.win:ext_timed(ts, 5 min);
// Last 10 tweets per user
select * from Tweet.std:groupwin(user).win:length(10);
// Top 5 Hashtags
select * from HashtagsPerMinute.std:sort(5, count desc);

40. You can create your
own custom Views

41. Correlation
// Associate hashtags used to describe a URL
insert into UrlTags
select u.value as url, h.value as hashtag
from Url.std:lastevent() as u,
Hashtag.std:lastevent() as h
where u.tweet_id = h.tweet_id;
insert into UrlTagsCount
select url,
hashtag,
count(*) as count
from UrlTags.win:time(1 hour)
group by url, hashtag;

42. Correlation (1/2)
// Every minute, output Top 3 hashtags per URL
select * from UrlTagsCount.ext:sort(3, count desc)
output snapshot at(*/1,*,*,*,*);

43. Event Patterns
// Measure how long it takes users to respond to Tweet
insert into ResponseDelay
select t.id as tweet_id,
t.user as author,
m.value as responder,
t.ts as start_ts,
m.ts as stop_ts,
m.ts - t.ts as duration
from pattern [
every (t=Tweet -> m=Mention(value = t.user))
];

44. Detecting Missing Events
// No Tweet from @codebits in 1 hour
select *
from pattern [ every Tweet(user = ‘codebits’) ->
(timer:interval(1 hour) and not Tweet(user = ‘codebits’))
];

45. Other features
• Subqueries
• Inner, outer joins
• Named windows
• 1 class integration with databases (JDBC)
st
• Regex-like Event Pattern matching (match-
recognize)

46. Esper is awesome!

47. It’s not a silver bullet
well, duh!

48. Memory Usage

49. Resilience &
Persistence

50. Weak Pattern matching

51. Drill-down not trivial

52. It’s NOT distributed!

53. Not full-stack

54. QA
For more: @antonioalegria