Introducing HDF 3.0, the open source IoT platform that everyone can easily start using right now.

1. 1 © Hortonworks Inc. 2011 – 2017 All Rights Reserved1 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
IoT Platform for Everyone
~Hortonworks DataFlow (HDF) 3.0~
Yifeng Jiang
Lead Solutions Engineer
July 1, 2017

2. 2 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
About Me
à 蒋 逸峰 (しょう いつほう / Yifeng Jiang)
à Lead Solutions Engineer, NAPAC, Hortonworks
– Hadooper since 2009
– HBase book author
– Software engineer, cloud, PaaS, DevOps
à Jogger, hiker
à Twitter: @uprush

3. 3 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

4. 4 © Hortonworks Inc. 2011 – 2017 All Rights Reserved
Drive IoT Data into Data Lake
Constrained
High-Latency
Localized Context
Hybrid – Cloud/On-Premise
Low-Latency
Global Context

5. 5 © Hortonworks Inc. 2011 – 2017 All Rights Reserved
Management
Constrained
High-Latency
Localized Context
Hybrid – Cloud/On-Premise
Low-Latency
Global Context
Management
Distributed Agent
Central Control
ß à
Multi-Tenant
Interactive C&C
Push or Pull

6. 6 © Hortonworks Inc. 2011 – 2017 All Rights Reserved
Security
Constrained
High-Latency
Localized Context
Hybrid – Cloud/On-Premise
Low-Latency
Global Context
Security
Sign or Encrypt
(data and control) ß à
TLS, Obfuscation,
Dynamic Entitlements,
Kerberos, PKI, AD/DS, etc.
‘ACK’

7. 7 © Hortonworks Inc. 2011 – 2017 All Rights Reserved
Beyond Management: Real-time Analysis
Constrained
High-Latency
Localized Context
Hybrid – Cloud/On-Premise
Low-Latency
Global Context

8. 8 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
DATA AT REST
DATA IN
MOTION
ACTIONABLE
INTELLIGENCE
Modern IoT Data Applications
PERISHABLE
INSIGHTS
HISTORICAL
INSIGHTS
INTERNET
OF
ANYTHING
Hortonworks
DataFlow
Hortonworks
Data Platform
Hortonworks Delivers
Connected Data Platforms

9. 9 © Hortonworks Inc. 2011 – 2017 All Rights Reserved
Hortonworks DataFlow (HDF) 3.0

10. 1
0
© Hortonworks Inc. 2011 – 2016. All Rights Reserved

11. 1
1
© Hortonworks Inc. 2011 – 2016. All Rights Reserved
Deeper Ecosystem Integration: 200+ Processors
Hash
Extract
Merge
Duplicate
Scan
GeoEnrich
Replace
ConvertSplit
Translate
Route Content
Route Context
Route Text
Control Rate
Distribute Load
Generate Table
Fetch
Jolt Transform
JSON
Prioritized Delivery
Encrypt
Tail
Evaluate
Execute
All Apache project logos are trademarks of the ASF and the respective projects.
Fetch
HTTP
Syslog
Email
HTML
Image
HL7
FTP
UDP
XML
SFTP
MQTT
WebSock
et

12. 12 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
NiFi vs. MiNiFi Java Processor, Smaller Footprint ~40 MB
NiFi Framework
Components
MiNiFi
NiFi Framework
User Interface
Components
NiFi

13. 13 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Data Movement Management with Nifi & Minifi
§ Constrained
§ High-latency
§ Localized context
§ Hybrid – cloud/on-premises
§ Low-latency
§ Global context
SOURCES
REGIONAL
INFRASTRUCTURE
CORE
INFRASTRUCTURE

14. 1
4
© Hortonworks Inc. 2011 – 2017 All Rights Reserved
Introducing Hortonworks Streaming Analytics Manager (SAM)
Streaming Analytics
Manager
Design, develop, deploy and
manage streaming analytics
app with a drag-and-drop
paradigm
“Make it a delightful experience to build
streaming analytics apps. Provide the same
type of awesome experience for streaming
analytics that customers have today when
building flow management apps with Nifi”

15. 15 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
SAM’s 3 Components
Builder – Ops - Insight
BUILD
OPERATEANALYZE

16. 1
6
© Hortonworks Inc. 2011 – 2017 All Rights Reserved
Introducing Hortonworks Schema Registry
A shared repository for schemas
allowing applications to save,
retrieve and reuse schemas and
flexibly interact with each other

17. 1
7
© Hortonworks Inc. 2011 – 2017 All Rights Reserved
Ambari – HDF Cluster Manager

18. 1
8
© Hortonworks Inc. 2011 – 2017 All Rights Reserved
Stream Builder
⬢ Builder components, shown on the canvas
palette, are the building blocks used by the
app developer to build streaming apps.
⬢ Drag and drop to build a working streaming
application without writing a single line of
code.
⬢ 4 Types of Components: Sources,
Processors, Sinks and Custom

19. 1
9
© Hortonworks Inc. 2011 – 2017 All Rights Reserved
Schema Registry
⬢ Data Governance
– Centralized registry to provide reusable
schema
– Version management to define
relationship between schemas
– Validation to enable generic format
conversion and generic routing
⬢ Operational Efficiency
– Centralized registry to avoid attaching
schema to every piece of data
– Version management to enable
consumers and producers can evolve at
different rates
– Validation to ensure data quality

20. 2
0
© Hortonworks Inc. 2011 – 2017 All Rights Reserved
Prescriptive Analytics
⬢ Routing
⬢ Filtering
⬢ UDF-based projections
⬢ Data delivery
⬢ Alerting

21. 2
1
© Hortonworks Inc. 2011 – 2017 All Rights Reserved
Predictive Analytics
⬢ Model Registry
– Sam has repository to store and
manage PMML based predictive
models
– First class features like version,
evolution policies, etc, will be
added in future release
⬢ PMML Processor
– Processor that can use model
from the registry and score the
models based on the input
stream of events coming in

22. 2
2
© Hortonworks Inc. 2011 – 2017 All Rights Reserved
Descriptive Analytics
⬢ Create time-series and real-
time analytics dashboards,
charts and graphs
⬢ 30+ visualization charts out of
the box with customization
capability
⬢ Powered by Druid and the
Stream Insight

23. IoT Case Studies
power by
Connected Data Platform

24. 2
4
© Hortonworks Inc. 2011 – 2017 All Rights Reserved
三菱ふそう
トラック・バス様

25. 2
5
© Hortonworks Inc. 2011 – 2017 All Rights Reserved
Coca-Cola East Japanさま
https://www.slideshare.net/HadoopSummit/from-a-single-droplet-to-a-full-bottle-our-
journey-to-hadoop-at-cocacola-east-japan

26. 2
6
© Hortonworks Inc. 2011 – 2017 All Rights Reserved
Coca-Cola East Japanさま
https://www.slideshare.net/DamienContreras/damien-contreras-futureofdata20170428

27. 27 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Build Your First Streaming Analytics
App in Under 10 Minutes

28. 2
8
© Hortonworks Inc. 2011 – 2017 All Rights Reserved
Trucking company w/ large fleet of international trucks
A truck generates millions of events for a given
route; an event could be:
§ 'Normal' events: starting / stopping of the vehicle
§ ‘Violation’ events: speeding, excessive acceleration
and breaking, unsafe tail distance
§ ‘Speed’ Events: The speed of a driver that comes in
every minute.
Company uses an application that monitors truck
locations and violations from the truck/driver in real-
time
Route?
Truck?
Driver?
Analysts query a broad
history to understand if
today’s violations are
part of a larger problem
with specific routes,
trucks, or drivers

29. 2
9
© Hortonworks Inc. 2011 – 2017 All Rights Reserved

30. 3
0
© Hortonworks Inc. 2011 – 2017 All Rights Reserved
7 Streaming Analytics Requirements the Developer has to Implement

31. 3
1
© Hortonworks Inc. 2011 – 2017 All Rights Reserved

32. 3
2
© Hortonworks Inc. 2011 – 2017 All Rights Reserved
DEMO

33. 3
3
© Hortonworks Inc. 2011 – 2017 All Rights Reserved
Summary
⬢Connected Data Platform for IoT
⬢Hortonworks DataFlow
–Web User Interface
–Configurable
–Scalable
–Easy to Manage
–Designed for Extension
⬢IoT and Big Data Are Transforming the
Industries

34. 34 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Hortonworksを始める為の５ステップ
1. Hortonworksサンドボックスを試す
hortonworks.com/sandbox
2. チュートリアルを試す
hortonworks.com/tutorials
3. Future of Data 勉強会に参加する
futureofdata.connpass.com
4. Hortonworks Community Connect (HCC)に参加する
hortonworks.com/community
5. サポートサービスを検討する
http://hortonworks.com/support

35. 3
5
© Hortonworks Inc. 2011 – 2017 All Rights Reserved
THANK YOU
Yifeng Jiang
@uprush