Original (german) title: Orientierung im Nebel: Unterschiede zwischen Stream Processing in Edge und Cloud. See https://www.buildingiot.de/veranstaltung-7812-orientierung-im-nebel%3A-unterschiede-zwischen-stream-processing-in-edge-und-cloud.html?source=0&id=7812

1. Orientierung im Nebel:
Unterschiede zwischen Stream
Processing in Edge und Cloud

2. • IIoT vs IoT
• (Stream) processing at the Edge
• How Apache Edgent can help
What I’ll talk about

3. • Julian Feinauer
• Phd in mathematics
• Founder of pragmatic minds GmbH and pragmatic industries GmbH
• Apache PLC4X & Apache Edgent (P)PMC member
• Twitter: @jfeinauer2
Who am I?

4. Industry 4.0, Digitalization and the
Industrial Internet of Things?!

5. • Everybody wants Industrie 4.0 or Digitalization
• „Can we do something with Big Data?“
• „We need a Data Lake“
• Can AI solve this for us?“
• This means that the Shopfloor is confrontend with
• ... connectivity
• ... modern IT infrastructure
• ... Cloud technology
Current Trend in the Industry

6. • #Neuland for the Industry
• Industry is always conservative regarding IT infrastructure
• Suppliers try to close the gap but are themselves faced with a
completly new situation and new technologies
• Very dynamic situation but Industry likes to plan
• In many industrial sites we have bad bandwidths
But...

7. • „Universal Protocol
Adapter“
• Connectivity for PLCs
• Bridging the GAP between
Shopfloor and IT
However...

8. The Industrial Internet of Things!

9. The Industrial Internet of Things
PLCs
Sensors
Actors

10. The Industrial Internet of Things
PLCs
Sensors
Actors
Hundreds of internal variables
• 10-100 Sensors
• Scaling
• Data types
• 10-100 Actors
• Different technologies
• Metadata is important

11. The Industrial Internet of Things
PLCs
Sensors
Actors
• No discrete Events
• Timeseries!
• Events have to be extracted based
on
• ... absolute values (thresholds)
• ... (bit-)triggers / value changes
• ... derivatives
• ...
• è Stream Processing at the Edge!

12. Stream Processing at the Edge

13. (Classic-)Definition:
“A data stream can be seen as a continuous and
potentially infinite stochastic process in which events
occur independently from another”

14. Streaming is easy!

15. The Industrial Internet of Things
PLCs
Sensors
Actors
• No discrete Events
• Timeseries!
• Events have to be extracted based
on
• ... absolute values (thresholds)
• ... (bit-)triggers / value changes
• ... derivatives
• ...
• è Stream Processing at the Edge!

16. A simple Example…

17. Example: Cycle Detection
• A (boolean) marker is true during cycle, false otherwise
• We want to detect cycles as events, and record their start / end

18. Example: Cycle Detection

19. Example: Cycle Detection
• A (boolean) marker is true during cycle, false otherwise
• We want to detect cycles as events, and record their start / end
• „Oh, we sometimes jitter, so a cycle is only a real cycle if its
longer than 10 seconds“

20. Example: Cycle Detection

21. Example: Cycle Detection
• A (boolean) marker is true during cycle, false otherwise
• We want to detect cycles as events, and record their start / end
• „Oh, we sometimes jitter, so a cycle is only a real cycle if its
longer than 10 seconds“
• „Ah, we forgot, it would be good to also store the pressure at the
end of the cycle“

22. Example: Cycle Detection

23. Example: Cycle Detection
• A (boolean) marker is true during cycle, false otherwise
• We want to detect cycles as events, and record their start / end
• „Oh, we sometimes jitter, so a cycle is only a real cycle if its longer than
10 seconds“
• „Ah, we forgot, it would be good to also store the pressure at the end
of the cycle“
• „In fact, we also need the current, rpm and temperature for the cycle“

24. Example: Cycle Detection
...

25. Example: Error Logging
• Exceptions are stored as bit mask
• About 2300 bits / exceptions on the machine
• Task
• Read all bits
• See which toggled
• Generate Event for Exception on / off

26. ...
Example: Error Logging

27. Example: Complex Actions
• We need production parameters, at defined points
• First, wait until bit0 is set, then take the start time
• Then, wait until bit1 is set, then take the pressure
• Then, wait until bit2 is set, then take the rpm
• ...

28. Example: Complex Actions
...

29. Part II – Which Engine to use?

30. Streaming is easy!

31. Stream Processing is not Stream
Processing
Property Cloud Edge
Distribution Yes No
Scalable Yes No
Fast Yes Easy
Stateless Yes* No
Message Delivery Exactly-Once At least once sufficient

32. From Imperative to Declarative

33. CRUNCH it!
• CRUNCH - A framework for declarative industrial data analysis
• Not another streaming framework, ABOVE the streaming layer
• Created by pragmatic minds GmbH
• Open Source sind end of 2018
• Apache 2.0 License
• Visit: https://github.com/pragmaticminds/crunch

34. CRUNCH it!
• CRUNCH - A framework for declarative industrial data analysis
• Not another streaming framework, ABOVE the streaming layer
• Created by pragmatic minds GmbH
• Open Source sind end of 2018
• Apache 2.0 License
• Visit: https://github.com/pragmaticminds/crunch

35. Introducing Apache Edgent (incubating)

36. Best practices... λ-Architecture

37. λ-Architecture at the Edge?

38. λ-Architecture at the Edge?

39. • Incubating project at the Apache Software Foundation
• Established 2016
• Edge runtime with
• a streaming runtime
• real-time analytics capabilities
• and cloud communication
for constrained devices (i.e., with small footprint), at the edge.
Apache Edgent

40. Apache Edgent

41. Apache Edgent

42. Edge to Cloud...

43. But...
• All have their own specific API
• Partly open source, but usually very vendor driven
• Vendor-Lock in
• Entrance barrier is low but this should be a strategic decision
• Can become very expensive, long term

44. Edgent to the rescue!
• All Open Source (Apache 2.0 Licence)
• Cloud Connectors for different providers, no vendor lock-in
• Tight integration with other Apache Frameworks, like
• Apache PLC4X (incubating)
• Apache IoTDB (coming soon)
• Complete toolkit to build analytic applications at the edge!

45. What‘s (probably) Next?
• More Cloud Connectors!
• Tight integration of CRUNCH
• Edge Integration Patterns!
• Improve Routing / Rule Engine at the Edge („Camel at the Edge“)
• Revive the Community

46. Interested?
• Visit www.edgent.apache.org
• Sign up to our mailing-list: dev@edgent.apache.org
• Follow us on twitter: @ApacheEdgent
• Start using Edgent
• File bugs: https://issues.apache.org/jira/projects/EDGENT
• Source code: https://github.com/apache/incubator-edgent
• Examples: https://github.com/apache/incubator-edgent-samples
• Join our meetup in Frankfurt on Mai 6th in Frankfurt (am Main)

47. Questions?
Julian Feinauer
T +49 7022 24933 01
M +49 176 2488 3977
E j.feinauer@pragmaticminds.de
PRAGMATICMINDS GmbH
Zementwerk 1
72622 Nürtingen
www.pragmaticminds.de