More Related Content Similar to Make Streaming IoT Analytics Work for You (20) More from Hortonworks (20) Make Streaming IoT Analytics Work for You2. 2 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
IoT - Relevance
7.2 Billion active SIM Cards
25 Billion connected things
500 Million tweets per day
– Average life of a tweet is 18 minutes
2 Zettabytes per year of Global IP Traffic
– 80% is unstructured
3. 3 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
The Four Vs of Big Data http://www.ibmbigdatahub.com/infographic/four-vs-big-data
4. 4 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
5 Attributes of a Streaming Platform
Ingest
Process
Analyze
Respond
Visualize
5. 5 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Data Flow versus Stream Analytics for IoT
Data Flow
– Ingest and route terabytes data into a ”unified firehose”
– Actively performance manage the latency and quality of these data flows –
• Across high variability of data formats, size of data and speed of data
Real Time Stream Analytics
– Sub second event processing with linear scalability to billions of events
– Predictive Analytics at Scale
• Real time data aggregation across edge nodes while processing 10s of millions of events and
100s of gigabytes per second
– Guaranteed no data loss and events processed in order
6. 6 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
6 Key Focus Areas for Building a Streaming Platform for IoT
1. Common Abstraction Layer
2. Latency
3. Lambda Architecture
1. “Orchestrate” over static and real time data
4. Scale-out
5. Rapid Application Development
6. Data Visualization
7. 7 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Focus Areas for Streaming Platform for IoT
1. Common Abstraction Layer
– Select one or more streaming engines
– Select one or more cloud providers
– Select one or more resource managers
– Select one or more event sources
– …Future Proof
8. 8 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Focus Areas for Streaming Platform for IoT
2. Latency
– 500 millisecond or less- Dashboards, Security Incidents, Asset Performance
– 20 milliseconds or less - Ad Networks, Preventive Maintenance
– …
9. 9 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Focus Areas for Streaming Platform for IoT
3. Lambda Architecture
– Integrate static and real time data
– Enrichment
– Orchestration of Batch workflows
– Predictive Classification and Scoring
10. 10 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Focus Areas for Streaming Platform for IoT
4. Scale Out
– Linear Scale out or scale down
– Resource Management
– Handle Transient workloads
11. 11 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Focus Areas for Streaming Platform for IoT
5. Rapid Application Development
– Industry vertical specific applications
– Aggregations
– Filters
– Multi Stream Correlations
– Splits
– Joins
– Normalizations
– Business Rules Editor
12. 12 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Focus Areas for Streaming Platform for IoT
6. Data Visualization
– Time Series Visualizations
– Metrics Dashboarding
– Trends
– Comparisons
– Thresholds
– Custom UI extensions
13. 13 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
IoT - Real Time Stream Analytics versus Streaming Engine
Abstracts underlying Streaming Engine- Storm, Spark Streaming, Flink..
OOTB Support for multiple (cloud) event sources - Kafka, AWS Kinesis, Azure Event Hub
Built-in Operators for Complex Event Processing
Built-in Real Time Dashboarding- Metrics and Events
PMML Support
Pluggable Workflow Management
Business Rules Editor and Rapid Application Development Framework
Cloud Deployment
Scalable Architecture
Handles different latency requirements
15. 15 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
© Hortonworks Inc. 2011
Hortonworks IoT Platform – The Big Picture
Page 15
Data Acquisition
Edge Processing
Real Time Stream Analytics
IoT Services
Rapid Application Development
IoT
ANALYTICS
CLOUD
16. 16 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
IoT Edge
Data Acquisition & Processing
Device Specific
Protocol
Device Data
Acquisition
Edge
Processing
IoT Analytics
CloudHTTP(s)
Edge Processing
• Reliable Delivery
• Buffering and Flow Control
• Simple Event Processing
• Edge Analytics
High Availability
• Scale out and Clustering
• Multi Data Center Support
Security
• 2 way SSL
• Data Element Masking
• Flexible Routing based on Data
Sensitivity
Kafka Enablement
• Data Ingest and Drain based on
Kafka Consumer and Producer
Support
17. 17 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Device Management
Agent
• Send Diagnostics
• Receive F/W updates
• Connectivity Logs
Enrollment &
Authentication
• 3-legged OAuth
based flow
• Encryption
• Confidentiality
Registry
• Device Metadata
Catalog
User Linkage
• Device - User Linkage
• May be many - many
between Device and
User
• Temporary or
Persistent
De-
Authorization
• Deregister
• Unlink User
• Disable
19. 19 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Connected Car - Generator of Big Data:
http://www.slideshare.net/AditiTechnologies/how-internet-of-things-iot-is-reshaping-the-automotive-sector-infographic
20. 20 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Connected Car - Use Cases http://gelookahead.economist.com/infograph/car-os/
21. 21 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Reference Architeucture: Connected Car Platform and Hortonworks
For more details contact sales@hortonworks.com
Hortonworks Data Platform
Core Data
Engineers (CAD)
Systems of Record (ERP)
Customers (CRM)
22. 22 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Apache NiFi Demo: https://www.youtube.com/watch?v=r4NsWbE4_-I
(Download here: http://hortonworks.com/products/hdf/
23. 23 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Questions?
http://community.hortonworks.com