IoT and Edge Integration with Open Source Frameworks: Internet of Things (IoT) and edge integration is getting more important than ever before due to the massively growing number of connected devices year by year. This session shows open source frameworks built to develop very lightweight microservices, which can be deployed on small devices or in serverless architectures with very low resources and wire together all different kinds of hardware devices, APIs and online services. The focus of this session lies on showing open source projects such as Eclipse Kura, Node-RED or Flogo, which offer a framework plus zero-code environment with web IDE for building and deploying integration and data processing directly onto connected devices using IoT standards such as MQTT, WebSockets or CoaP, but also other interfaces such as Twitter feeds or REST services. The end of the session discusses the relation to other components in a IoT architecture including cloud IoT platforms and big data respectively streaming analytics solutions (such as Apache Storm, Flink, Spark Streaming, Samza, StreamBase, Apama).

1. Kai Wähner
Technology Evangelist
kontakt@kai-waehner.de
LinkedIn
@KaiWaehner
www.kai-waehner.de
November 2016
Comparison of Open Source Frameworks
for Integrating the Internet of Things

2. © Copyright 2000-2016 TIBCO Software Inc.
• used to develop lightweight IoT edge apps and microservices
• part of a hybrid integration architecture
• complementary to IoT cloud platforms and streaming analytics
Key Takeaways
IoT integration frameworks are...

3. © Copyright 2000-2016 TIBCO Software Inc.
1) IoT Trends and Challenges
2) Hybrid Integration Architecture
3) IoT Integration
o Dataflow Pipeline
o Stream Processing
o Process Engine
4) Open Source IoT Process Engines
o Eclipse Kura (+ Apache Camel)
o Node-RED
o Flogo
5) Cloud IoT Platforms
Agenda

4. © Copyright 2000-2016 TIBCO Software Inc.
1) IoT Trends and Challenges
2) Hybrid Integration Architecture
3) IoT Integration
o Dataflow Pipeline
o Stream Processing
o Process Engine
4) Open Source IoT Process Engines
o Eclipse Kura (+ Apache Camel)
o Node-RED
o Flogo
5) Cloud IoT Platforms
Agenda

5. © Copyright 2000-2016 TIBCO Software Inc.
Internet of Things (IoT)
… refers to the ever-growing network of physical
objects that feature an IP address for internet
connectivity, and the communication that occurs
between these objects and other Internet-enabled
devices and systems.

6. © Copyright 2000-2016 TIBCO Software Inc.
Forecast for Year 2020 à 20+ Billion Devices

7. © Copyright 2000-2016 TIBCO Software Inc.
IoT Landscape 2016
The IoT Landscape 2016 Edition - Matt Turck & David Rogg, First Mark Capital

8. © Copyright 2000-2016 TIBCO Software Inc.
Integration is Key for Success of IoT Projects
Without Integration
there is no
Internet of Things!
Gartner
Market Guide for IoT Integration
“Through 2018, half the cost of
implementing IoT solutions will be
spent on integration. “
Strategic Planning Assumption,
Benoit J. Lheureux, Massimo Pezzini, Alfonso Velosa
!

9. © Copyright 2000-2016 TIBCO Software Inc.
Processes Analytics
APIs Streams & Events
Interconnect Everything
Devices
Equipment
CORE
Run the Business
EDGE
Change the Business
People

10. © Copyright 2000-2016 TIBCO Software Inc.
IoT raises several new challenges
• Devices are not connected to the cloud
• Devices have low bandwidth to connect
• Latency of connectivity is significant
• Connectivity is not reliable
• Connectivity is not cost-effective
Great IoT standards available
• MQTT, CoaP, OPC US, many more
• Different abstraction levels
• Not the one single standard
Challenges

11. © Copyright 2000-2016 TIBCO Software Inc.
1) IoT Trends and Challenges
2) Hybrid Integration Architecture
3) IoT Integration
o Dataflow Pipeline
o Stream Processing
o Process Engine
4) Open Source IoT Process Engines
o Eclipse Kura (+ Apache Camel)
o Node-RED
o Flogo
5) Cloud IoT Platforms
Agenda

12. © Copyright 2000-2016 TIBCO Software Inc.
Integration Frameworks
Decision
Data
Conversion
Transitions
Process
Call
Process Design Tools
Integration Processes
Connectors to vendor
applications
Connectors to technologies
Connectors
(e.g. JMS, MQTT, FTP) (e.g. SF.com, SAP)

13. © Copyright 2000-2016 TIBCO Software Inc.
Enterprise Integration Patterns
“The goal of EIPs is to document technology-
independent design guidance that helps
developers and architects describe and develop
robust integration solutions.”
http://www.eaipatterns.com

14. © Copyright 2000-2016 TIBCO Software Inc.
Pervasive Integration with different Technologies and User Roles
https://www.linkedin.com/pulse/how-implement-hybrid-integration-platform-tackle-massimo-pezzini
There is no “one-size-fits-all” IoT integration!

15. © Copyright 2000-2016 TIBCO Software Inc.
Hybrid Integration Platform (HIP)
Application
Integration
Application
Integration
(on a PaaS)
iPaaS
Integration
Platform as a Service
iSaaS
Integration
Software as a Service
Cloud-Ready Cloud-Native
API
Management
On Premise
Public Cloud
Public Cloud Public Cloud
On Premise
Public Cloud
Process
Integration
Business Process
Management Platform
Streaming
Analytics
Fog Computing
Edge Devices
Edge
Integration
Analysts are sure:
HIP is “the new default”
in most enterprises!

16. © Copyright 2000-2016 TIBCO Software Inc.
1) IoT Trends and Challenges
2) Hybrid Integration Architecture
3) IoT Integration
o Dataflow Pipeline
o Stream Processing
o Process Engine
4) Open Source IoT Process Engines
o Eclipse Kura (+ Apache Camel)
o Node-RED
o Flogo
5) Cloud IoT Platforms
Agenda

17. © Copyright 2000-2016 TIBCO Software Inc.
Fog Computing
http://www.slideshare.net/Angelo.Corsaro/20141210-fog
… keep data closer
‘to the edge’
!

18. Typical IoT Integration Scenario
What’s wrong with
Cloud-only IoT Integration?
• Connectivity is unreliable
• Non-trivial latency in hops
• Networks costs add up quickly

19. A better IoT Integration Blueprint
Benefits:
• Local control = more reliable
• Less traffic = Lower TCO
• Edge integration & event processing

20. © Copyright 2000-2016 TIBCO Software Inc.
1) IoT Trends and Challenges
2) Hybrid Integration Architecture
3) IoT Integration
o Dataflow Pipeline
o Stream Processing
o Process Engine
4) Open Source IoT Process Engines
o Eclipse Kura (+ Apache Camel)
o Node-RED
o Flogo
5) Cloud IoT Platforms
Agenda

21. © Copyright 2000-2016 TIBCO Software Inc.
Dataflow Pipeline – Extract, Transform, Load
https://www.linkedin.com/pulse/data-pipeline-hadoop-part-1-2-birender-saini

22. © Copyright 2000-2016 TIBCO Software Inc.
• Flow-based Programming
• Ingest data from various sources
• Extract (E) – Transform (T) – Load (L)
• Transform / Filter / Route / Aggregate / Enrich …
• High-throughput straight-through data flows
• Data lineage
• Synchronous and asynchronous communication
• Batch or stream processing
• Visual coding with flow editor
• Not to be mixed-up with stream processing for event correlation
Dataflow Pipeline - Characteristics

23. © Copyright 2000-2016 TIBCO Software Inc.
Dataflow Pipeline – Examples All except Informatica use
open source / open core business model

24. © Copyright 2000-2016 TIBCO Software Inc.
1) IoT Trends and Challenges
2) Hybrid Integration Architecture
3) IoT Integration
o Dataflow Pipeline
o Stream Processing
o Process Engine
4) Open Source IoT Process Engines
o Eclipse Kura (+ Apache Camel)
o Node-RED
o Flogo
5) Cloud IoT Platforms
Agenda

25. © Copyright 2000-2016 TIBCO Software Inc.
Stream Processing
time
1 2 3 4 5 6 7 8 9
Event Streams
• Continuous Queries
• Sliding Windows
• Filter
• Aggregation
• Correlation
• …
Stream
Processing
H20.ai
Open
Source
R
TERR
Spark
ML
MATLAB
SAS
PMML

26. © Copyright 2000-2016 TIBCO Software Inc.
Stream Processing - Examples
OPEN SOURCE CLOSED SOURCE
PRODUCT
FRAMEWORK
Azure Microsoft
Stream Analytics

27. © Copyright 2000-2016 TIBCO Software Inc.
Streaming Analytics à Predictive Maintenance
Voltage
Temperature
Vibration
Device
history
Temporal analytic: “If vibration spike is followed by temp spike then
voltage spike [within 4 hours] then flag high severity alert.”

28. © Copyright 2000-2016 TIBCO Software Inc.
1) IoT Trends and Challenges
2) Hybrid Integration Architecture
3) IoT Integration
o Dataflow Pipeline
o Stream Processing
o Process Engine
4) Open Source IoT Process Engines
o Eclipse Kura (+ Apache Camel)
o Node-RED
o Flogo
5) Cloud IoT Platforms
Agenda

29. © Copyright 2000-2016 TIBCO Software Inc.
• Integrate and orchestrate various data sources
• Wire together hardware devices, APIs and online services
• Transform / Filter / Route / Aggregate / Enrich …
• Error handling, Re-Try, Re-Routing, Wait, Resume, ...
• Not just Extract-Transform-Load
• Synchronous and asynchronous communication
• Typically real time processing
• Visual coding with flow editor
• Deployed at the edge
Process Engine

30. © Copyright 2000-2016 TIBCO Software Inc.
Real World IoT Use Cases for a Process Engine

31. © Copyright 2000-2016 TIBCO Software Inc.
1) IoT Trends and Challenges
2) Hybrid Integration Architecture
3) IoT Integration
o Dataflow Pipeline
o Stream Processing
o Process Engine
4) Open Source IoT Process Engines
o Eclipse Kura (+ Apache Camel)
o Node-RED
o Flogo
5) Cloud IoT Platforms
Agenda

32. © Copyright 2000-2016 TIBCO Software Inc.
Commonalities:
• Open source frameworks
• Connectivity to IoT technologies (MQTT, CoaP, REST, …)
• Web UI for visual coding, testing, debugging
• Deployable “at the edge”
• For developers / integration specialists / citizen integrators
• Extendable SDKs / APIs
Open Source IoT Process Engines

33. © Copyright 2000-2016 TIBCO Software Inc.
1) IoT Trends and Challenges
2) Hybrid Integration Architecture
3) IoT Integration
o Dataflow Pipeline
o Stream Processing
o Process Engine
4) Open Source IoT Process Engines
o Eclipse Kura (+ Apache Camel)
o Node-RED
o Flogo
5) Cloud IoT Platforms
Agenda

34. © Copyright 2000-2016 TIBCO Software Inc.
Eclipse Kura
http://www.eclipse.org/kura

35. © Copyright 2000-2016 TIBCO Software Inc.
• Focus on IoT Gateway
• Set of Java and OSGi services
• including I/O services, Data Services, Cloud Services, Networking, etc.
• Web UI for Configuration (Devices, Network, Protocols, etc.)
• Eclipse Public License 1.0
• Mature framework (~3 years old, code based committed in December 2013)
• Best of breed with focus on coding instead of a visual designer
• Eclipse IoT Projects
• IoT Gateway --> Eclipse Kura
• Services like Eclipse SmartHome
• Standard implementations like Mosquitto (MQTT Server)
• Connectivity via Eclipse Paho (MQTT Client)
• Apache Camel Connector
• Integration Framework (connectivity, enterprise integration patterns)
• Visual coding via JBoss or Talend (both available as open source edition)
• For Developers / Integration Specialists only
• More complex to install and build a first IoT integration flow
• Write source code
• Can be run on a variety of platforms (on premise, cloud, edge devices, container)
Eclipse Kura (+ Apache Camel) à Facts

36. © Copyright 2000-2016 TIBCO Software Inc.
Eclipse Kura – Functionality Overview and Architecture
https://www.youtube.com/watch?v=ia8cLnR1uFI
http://www.slideshare.net/Eurotechchannel/kuram2miotgateway

37. © Copyright 2000-2016 TIBCO Software Inc.
Eclipse Kura - Architecture

38. © Copyright 2000-2016 TIBCO Software Inc.
Eclipse Kura – Hello World
https://eclipse.github.io/kura/doc/hello-example.html

39. © Copyright 2000-2016 TIBCO Software Inc.
Apache Camel
http://www.kai-waehner.de/blog/2012/05/04/apache-camel-tutorial-introduction/

40. © Copyright 2000-2016 TIBCO Software Inc.
Apache Camel (in conjunction with Eclipse Kura)
https://dzone.com/articles/creating-camel-routes-for-eclipse-kura

41. © Copyright 2000-2016 TIBCO Software Inc.
Live Demo
Kura (+ Apache Camel) in Action

42. © Copyright 2000-2016 TIBCO Software Inc.
1) IoT Trends and Challenges
2) Hybrid Integration Architecture
3) IoT Integration
o Dataflow Pipeline
o Stream Processing
o Process Engine
4) Open Source IoT Process Engines
o Eclipse Kura (+ Apache Camel)
o Node-RED
o Flogo
5) Cloud IoT Platforms
Agenda

43. © Copyright 2000-2016 TIBCO Software Inc.
Node-RED
https://nodered.org/

44. © Copyright 2000-2016 TIBCO Software Inc.
Node-RED
http://industrialinternet.co.uk/category/node-red/

45. © Copyright 2000-2016 TIBCO Software Inc.
• Focus on Integration on an IoT Gateway
• Built on Node.js and Javascript
• Very easy to install and build a first IoT integration flow
• For developers / integration specialists and ciziten integrators
• Visual coding via web designer
• Apache 2.0 License
• Mature framework (~3 years old) with many examples and docs
• Native integration into IBM Bluemix cloud platform
• Leverage other Bluemix services
• Can be run on a variety of platforms (on premise, cloud, edge devices, container)
• Flows can be shared as JSON strings / files
• No binaries to share and deploy on devices
Node-RED à Facts

46. © Copyright 2000-2016 TIBCO Software Inc.
Live Demo
Node-RED in Action

47. © Copyright 2000-2016 TIBCO Software Inc.
1) IoT Trends and Challenges
2) Hybrid Integration Architecture
3) IoT Integration
o Dataflow Pipeline
o Stream Processing
o Process Engine
4) Open Source IoT Process Engines
o Eclipse Kura (+ Apache Camel)
o Node-RED
o Flogo
5) Cloud IoT Platforms
Agenda

48. © Copyright 2000-2016 TIBCO Software Inc.
Flogo
http://www.flogo.io

49. Project Flogo IoT Example

50. © Copyright 2000-2016 TIBCO Software Inc.
• Focus on Integration on an IoT Gateway and very lightweight edge applications
• Powered by Golang
• Very easy to install and build a first IoT integration flow
• Ultra-lightweight footprint with zero dependency model
• For developers / integration specialists and ciziten integrators
• Visual coding via web designer
• BSD-style License
• Developer Preview (released in October 2016)
• Can be run on a variety of platforms (on premise, cloud, edge devices, container)
• Flows can be shared as JSON strings / files
• Lightweight binaries with zero dependencies can be shared and deployed on devices
Flogo à Facts

51. Web-Native Step-Back Debugger
• Step back in a flow
• Interactively design/debug
your process
• Simulate sensor events
• Change data or configuration
and run from Trigger or any
Activity
• Underlying technology could
be used to support remote
ops debugging of failed flows

52. © Copyright 2000-2016 TIBCO Software Inc.
Live Demo
Flogo in Action

53. What technology to build edge apps?
Java
• Heavy (>200MB)
• Not agile
• Robust
• Licensing risk
Node.js
• Lighter (~70MB)
• Agile
• Not IoT robust
• Clear licensing
C/C++
• Lightest
• Not Agile
• Robust
• Clear licensing

54. © Copyright 2000-2016 TIBCO Software Inc.
Why Go?
Modern programming language
Getting a lot of traction due to usage in frameworks like Docker or Kubernetes
Concurrency
Built into the language (Goroutines, channels, no explicit thread programming)
Memory management
Modern Garbage collector
Statically typed
Type-safe development with no “surprises” during runtime
“Partly” object-oriented
Simple and flexible type system, composition instead of inheritance
Zero-dependency programming model
All-included binary file, no dependent “external” libraries
Speed
Ultra fast compilation and startup time, very lightweight footprint

55. Infrastructure Layers
Hardware
OS
VM - JVM
Framework - OSGi
App
Eclipse Kura
Hardware
OS
VM – V8
Framework – Node.js
App
Node-RED
Hardware
OS
App
Flogo
Typically 100s of
dependencies –
100MB+
“All included” zero-
dependency model
enabled by Go

56. Hardware
OS
Security Logging Metrics
Hardware
OS
V8
Protocol
Bridges
Diagnostics
Sensor
Processing
V8
Node.js
Logging
Node.js
Security
Whoa!
Overhead for every app
means you run out of room
for your apps
Node-RED Flogo

57. Comparison – Resource Requirements of IoT Frameworks
Eclipse Kura Node-RED Flogo
VM JVM V8 Golang
Base Disk Space 59 MB 56 MB ~ 0 MB
Base Runtime Memory ~ 170 MB (with Open JDK)
~ 20 MB (with Oracle Embedded Java)
> 50 MB ~ 5 MB
Startup time Slow (~8 sec) Slow (~5 sec) Fast (~1 sec)
Application Build Slow Not Applicable Fast (~2 secs)

58. Outlook: Flogo Nanoservices on Microcontrollers
Flogo nanoservices that target
microcontrollers like ARM M0
Offload I/O centric compute
farther off to the edge.

59. © Copyright 2000-2016 TIBCO Software Inc.
1) IoT Trends and Challenges
2) Hybrid Integration Architecture
3) IoT Integration
o Dataflow Pipeline
o Stream Processing
o Process Engine
4) Open Source IoT Process Engines
o Eclipse Kura (+ Apache Camel)
o Node-RED
o Flogo
5) Cloud IoT Platforms
Agenda

60. Intel IoT Reference Architecture
Potential use cases
• Event-driven flows & rules
• Device Lifecycle Mgmt
• User Experience
Core Integration
Agile Microservices
API Management

61. Amazon IoT Reference Architecture
Potential use cases
• Event-driven Flows
• Device Lifecycle Mgmt
• Orchestration
Core Integration
Agile Microservices
API Management

62. Google IoT Reference Architecture
Potential edge apps

63. © Copyright 2000-2016 TIBCO Software Inc.
Serverless Architecture with IBM’s OpenWhisk (Open Source)
https://apiguy.tokyo/2016/03/01/openwhisk-a-world-first-in-open-serverless-architecture/
Edge
App
Edge
Gateway
Ultra-
Lightweight
Serverless
App

64. © Copyright 2000-2016 TIBCO Software Inc.
Hybrid Integration Architecture is the New Default

65. © Copyright 2000-2016 TIBCO Software Inc.
• used to develop lightweight IoT edge apps and microservices
• part of a hybrid integration architecture
• complementary to IoT cloud platforms and streaming analytics
Key Takeaways
IoT integration frameworks are...

66. Questions? Please contact me!
Kai Wähner
Technology Evangelist
kontakt@kai-waehner.de
@KaiWaehner
www.kai-waehner.de
LinkedIn