Introduction to Complex Event Processing (CEP). How FIWARE deals with CEP through FIWARE Perseo. How to connect FIWARE Perseo with FIWARE Orion Context Broker. How can we define an event with Event Processing Language (EPL) and what are the predefined actions to include in FIWARE Perseo.

1. Session 7 - Creating a Context-Aware Solution: Complex Event
Processing with Perseo
Fernando López, Cloud & Platform Senior Expert
fernando.lopez@fiware.org
@flopezaguilar
FIWARE Foundation, e.V.

2. Learning Goals
1
● What is a Complex Event Processing?
● How to connect Perseo with Orion Context Broker?
● How can I define an event with EPL?
● What are the available acBons?

3. Overview
2
▪ Context-aware CEP GE
• Listen to events from context information to identify patterns described by rules, and
immediately react by triggering actions.
• Designed to be NGSI-compliant, and thus, to implement systems that works seamless
and jointly with Orion CB.
□ Uses NGSI as communication protocol for events
□ Follows a PUSH model
• Based on Esper CEP
▪ It is focused on Complex Event Processing, not on Stream Processing.
• It defines a rule (query) engine, not a graph engine.
• Rules are set using SQL-like queries (EPL: Event Processing Language)

4. Introduction to Complex Event Processing
3
▪ Requirement to process events (or messages) in real-time or near real-time.
▪ Key considerations for these types of applications are throughput, latency and the
complexity of the logic required:
• High throughput - applications that process large volumes of messages (between 1,000 to
100k messages per second)
• Low latency - applications that react in real-time to conditions that occur (from a few
milliseconds to a few seconds)
• Complex computations - applications that detect patterns among events (event
correlation), filter events, aggregate time or length windows of events, join event series,
trigger based on absence of events etc.

5. Introduction to the Esper architecture
▪ Esper is a language, a language compiler and a runtime environment.
▪ Language
• The Esper language is the Event Processing Language (EPL).
• It is a declarative, data-oriented language for dealing with high frequency time-based
event data.
• EPL is compliant to the SQL-92 standard and extended for analyzing series of events
and in respect to time.
▪ Language Compiler
• Compiles EPL source code into Java Virtual Machine (JVM) bytecode.
• The resulting executable code runs on a JVM within the Esper runtime environment.
4

6. Introduction to the Esper architecture
▪ Runtime
• The Esper runtime runs on top of a JVM.
• You can run byte code produced by the Esper compiler using the Esper runtime.
▪ The Esper architecture is similar to Scala, Clojure and Kotlin.
5

7. Overview: CEP in FIWARE with Perseo
6
PERSEO
Context-Aware CEP
ACTIONS
RULES
ORION
Context
Broker
EVENTS
NGSI subscription
NGSI
Context
notification
Pattern
matching
REST API
ACTION: NGSI Update Context
ACTION: HTTP
ACTION: SMS
ACTION: EMAIL
ACTION: TWEET
Perseo-fe
Perseo-core
Events
Alarms
Actuate
Sensors
SMART IoT PLATFORM

8. ▪ Repositories
• Perseo Core (https://github.com/telefonicaid/perseo-core)
• Perseo Front-End (https://github.com/telefonicaid/perseo-fe)
▪ Docker images
• Perseo Core (https://hub.docker.com/r/fiware/perseo-core/)
• Perseo Front-End (https://hub.docker.com/r/fiware/perseo/)
Overview: CEP in FIWARE with Perseo
7

9. Notice vs. Event
▪ Notifications or notices from Context Broker
are processed by Perseo (perseo-fe) before
being sent to its core (perseo-core + Esper)
as events.
▪ They correspond to changes in the values of
an attribute of an entity to which Perseo is
subscribed to.
{
"subscriptionId":"5bfaf6ab27124e3d9b6c0b20",
"data":[{
"id":"streetlight:gothenburg:3",
"type":"Streetlight",
"status":{
"type":"Text",
"value":"ok",
"metadata":{}
},
"dateLastLampChange":{
"type":"DateTime",
"value":"2018-11-20T11:00:00.00Z",
"metadata":{}
},
"address":{
"type":"Address",
"value":{
"addressLocality":"Gothenburg",
"addressCountry":"Sweden",
"postalCode":"425 30",
"streetAddress":"Norra Ligården 1”
},
"metadata":{}
},
"areaServed":{
"type":"Text",
"value":"Karra-Sodra",
"metadata":{}
},
"illuminanceLevel":{
"type":”Number",
"value":”95",
"metadata":{}
},
"powerState":{
"type":"Text",
"value":"off",
"metadata":{}
}
}],
"subservice":"/demo/fiwaresummit19",
"service":”carthage”
}
Notice
8

10. Notice vs. Event
▪ Notifications or notices from Context Broker
are processed by Perseo (perseo-fe) before
being sent to its core (perseo-core + Esper)
as events.
▪ They correspond to changes in the values of
an attribute of an entity to which Perseo is
subscribed to.
{
"noticeId":"710e8dd0-f15a-11e8-90d5-7fa905f73247",
"noticeTS":1543223138094,
"id":"streetlight:gothenburg:3",
"type":"Streetlight",
"isPattern":false,
"subservice":"/demo/fiwaresummit19",
"service":"carthage",
"status__type":"Text",
"status":"ok",
"address__type":"Address",
"address__addressLocality":"Gothenburg",
"address__addressCountry":"Sweden",
"address__postalCode":"425 30",
"address__streetAddress":"Norra Ligården 1",
"areaServed__type":"Text",
"areaServed":"Karra-Sodra",
"illuminanceLevel __type":"Number",
"illuminanceLevel": 95
"powerState__type":"Text",
"powerState":"off”
}
Event
Restriction:
▪ An attribute cannot be named id or type
9

11. Events, datatypes and NGSIv2
▪ Perseo flattens the data structures to access the internal values of each data type, and add the
next parsed values in the event. This attributes can be used in the rules individually.
"location__type":"geo:point",
"location__lat":12.0011546,
"location__lon":57.7934324,
"location__x":586374.5004314554,
"location__y":1326805.6699024632,
Location (geo:point, geo:line, geo:box, geo:polygon and geo:json):
"location": {
"type": "geo:point",
"value": "12.0011546, 57.7934324",
"metadata": {}
}
"dateLastLampChange__type": "DateTime",
"dateLastLampChange__iso": 2017-09-10T11:00:00.00Z,
"dateLastLampChange__ts": 1505041200000,
"dateLastLampChange__day": 10,
"dateLastLampChange__month": 9,
"dateLastLampChange__year": 2017,
"dateLastLampChange__hour": 13,
"dateLastLampChange__minute": 0,
"dateLastLampChange__second": 0,
"dateLastLampChange__millisecond": 0,
"dateLastLampChange__dayUTC": 10,
"dateLastLampChange__monthUTC": 9,
"dateLastLampChange__yearUTC": 2017,
"dateLastLampChange__hourUTC": 11,
"dateLastLampChange__minuteUTC": 0,
"dateLastLampChange__secondUTC": 0,
"dateLastLampChange__millisecondUTC": 0,
DateTime:
"dateLastLampChange": {
"type": "DateTime",
"value": "2017-09-10T11:00:00.00Z",
"metadata": {}
}
10

12. Rules
{
"name":“streetlight_column_alarm",
"text":"select ev.id? as id, ev.status? as laststatus, ev.dateLastLampChange__iso? as lastchange
pattern [every ev=iotEvent(type="Streetlight" and ev.status? = ‘columnIssue’)]",
"action”:{
"type":"update",
"parameters":{
"id":"${id}_alarm_column",
"type”:"StreetlightAlarm",
"version":"2",
"attributes": [
{
"name":"status",
"type":"Text",
"value":"${laststatus}"
},
{
"name":”lastChange",
"type":"DateTime",
"value":"${lastchange}"
}
]
}
}
Rule example. (create/update a StreetlightAlarm entity when a problem is detected in the column/pole)
▪ Perseo rules have three mandatory sections: name, text and action
11

13. Rules: names
▪ Simple description of the rule.
12
"name":“streetlight_column_alarm”
Name

14. Rules: text
▪ Based on Esper EPL SQL-like
rules
▪ Pattern matching:
• from pattern
• match_recognize
▪ Event stream called iotEvent
and instanciated as ev
▪ We can use predefined
functions, such as cast(),
avg(), etc…
▪ It is no longer necessary to add
the RuleName in the "text" field
SELECT ev.id? as id, ev.status? as laststatus, ev.areaServed?
as areaServed
FROM pattern
[every ev=iotEvent(type=‘Streetlight’ and ev.status? != ‘ok’)]
EPL using paGern clause
SELECT *
FROM iotEvent(type=‘Streetlight’).win:time(1 minute)
MATCH_RECOGNIZE (
measures A.status? as status
pattern (A B)
define(
A as A.status? != ‘ok’,
B as B.status? != ‘ok’ and
B.id? != A.id)
EPL using match_recognize
13

15. Rules: actions
Rule actions
▪ Options:
• Update context broker entity
• SMS
• Email
• HTTP invocation
• Send a tweet
▪ We can use placeholders, ${X},
to insert data from the events
"action":”:{
"type":"update",
"parameters":{
"id":"${id}_alarm_column",
"type”:"StreetlightAlarm",
"version":"2",
"attributes": [
{
"name":"status",
"type":"Text",
"value":"${laststatus}"
},
{
"name":”lastChange",
"type":"DateTime",
"value":"${lastchange}"
}
]
}
Update CB entity action
"action": {
"type": "sms",
"template": "${id} status has change to ${laststatus}",
"parameters": {
"to": "123456789"
}}
SMS action
14

16. Rules: actions
Rule actions
▪ Options:
• Update context broker entity
• SMS
• Email
• HTTP invocation
• Send a tweet
▪ We can use placeholders, ${X},
to insert data from the events
"action": {
"type": "email",
"template": "${id} status has change to ${laststatus}”,
"parameters": {
"to": "someone@fiware.org",
"from": "cep@system.org",
"subject": ”Problems with StreetLamp ${id}. The
status has change to ${laststatus}"
}
}
Email action
15

17. Rules: actions
Rule actions
▪ Options:
• Update context broker entity
• SMS
• Email
• HTTP invocation
• Send a tweet
▪ We can use placeholders, ${X},
to insert data from the events
"action":{
"type": "post",
"template": "${id} status has change to ${laststatus}",
"parameters":{
"url": "http://localhost:9182/${type}/${id}",
"method": "PUT",
"headers": {
"Content-type": "text/plain",
"X-${type}-status": "${laststatus}"
},
"qs": {
”status": "${laststatus}",
}
}
}
HTTP acLon
16

18. Rules: actions
Rule actions
▪ Options:
• Update context broker entity
• SMS
• Email
• HTTP invocation
• Send a tweet
▪ We can use placeholders, ${X},
to insert data from the events
"action": {
"type": "twitter",
"template": "${id} status has change to ${laststatus}"
"parameters": {
"consumer_key": "xvz1evFS4wEEPTGEFPHBog",
"consumer_secret":
"L8qq9PZyRg6ieKGEKhZolGC0vJWLw8iEJ88DRdyOg",
"access_token_key": "xvz1evFS4wEEPTGEFPHBog",
"access_token_secret":
"L8qq9PZyRg6ieKGEKhZolGC0vJWLw8iEJ88DRdyOg"
}
}
Tweet acLon
17

19. Timed Rules
{
"name":”10_sec_timer_rule",
"text":"select * from pattern [every timer:interval(10 sec)]",
"action":{...}
}
Rule example: fires each 10 seconds
{
"name":”2_week_timer_rule",
"text":"select * from pattern [every timer:interval(2 weeks)]",
"action":{...}
}
▪ timer:interval(<number|variable_name> <time-period>)
Rule example: fires each 2 weeks
Time Periods:
• " years" | "year"
• " months" | "month “
• " weeks" | "week“
• "days" | "day“
• "hours" | "hour“
• "minutes" | "minute" | "min“
• "seconds" | "second" | "sec“
• "milliseconds" | "millisecond" | "msec"
18

20. Timed Rules
▪ timer:at (minutes, hours, days of month, months, days of week [, seconds [, time zone]]).
{
"name":”5_min_past_hour",
"text":"select * from pattern [every timer:at(5, *, *, *, *)]",
"action":{...}
}
Rule example: fires once an hour, at minute five
{
"name":”17_PST",
"text":"select * from pattern [timer:at (0, 17, *, *, *, *, 'PST')]",
"action":{...}
}
Rule example: only fires 1 time at 5:00 pm Pacific Standard Time
19

21. Timed Rules
▪ timer:schedule(repetitions, date, period)
{
"name":"specific_date",
"text":"select * from pattern[every timer:schedule(repetitions: 2, date: '2018-03-01T13:00:00Z',
period: 1 year 2 month 10 days 2 hours 30 minutes)]",
"action":{...}
}
Rule example: fires 2 times, first callback (2018-03-01 at 13:00:00 UTC) second callback (2019-05-11 at 15:30:00 UTC)
{
"name":"next_minute",
"text":"select * from pattern[every timer:schedule(date: '2019-10-01T05:52:00Z')]",
"action":{...}
}
Rule example: fires on 2019-10-01T05:52:00Z
Note: There are many other kind of timed Esper rules, such as time_batch , ext_timed_batch, match_recognize intervals, etc.
20

22. Summary: Terms
21
● CEP, Complex Event processing is event processing that combines data from mulBple sources to infer
events or paYerns that suggest more complicated circumstances. (source: Wikipedia).
● Esper is an event stream processor that aims to enable a short development cycle from incepBon to
producBon for these types of applicaBons. (source: EsperTech).
● EPL, Event Processing Language (EPL) defined to be the Esper language and designed for Complex
Event Processing and Streaming AnalyBcs. (source: EsperTech).

23. References
22
● FIWARE Catalogue
o https://www.fiware.org/developers/catalogue
● FIWARE Academy
o https://fiware-academy.readthedocs.io/en/latest/index.html
● Perseo documentation
o https://perseo.readthedocs.io/en/latest/

24. References
23
● GitHub repos
o Perseo Core (https://github.com/telefonicaid/perseo-core)
o Perseo Front-End (https://github.com/telefonicaid/perseo-fe)
● Docker images
o Perseo Core (https://hub.docker.com/r/fiware/perseo-core)
o Perseo Front-End (https://hub.docker.com/r/fiware/perseo)
● Esper Doc
o http://esper.espertech.com/release-7.1.0/esper-reference/html/index.html

25. Question & Answer
24
fiware-tech-help@lists.fiware.org

27. 2
6