More Related Content Similar to Hortonworks Data In Motion Series Part 4 (20) More from Hortonworks (20) Hortonworks Data In Motion Series Part 42. 2 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Agenda
Overview of Hortonworks DataFlow (HDF)
How HDF transforms data movement – months to minutes
HDF Use Cases
Real-World HDF Use Cases
Apache NiFi Case Studies from Hadoop Summit
Other Apache NiFi/HDF Use Cases
3. 3 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Connected Data Platforms
4. 4 © Hortonworks Inc. 2011 – 2016. All Rights Reserved4 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Payment
Tracking
Due
Diligence
Social
Mapping
Product
Design
M & A
Call
Analysis
Machine
Data
Defect
Detecting
Factory
Yields
Customer
Support
Basket
Analysis
Segments
Customer
Retention
Sentiment
Analysis
Optimize
Inventories
Supply
Chain
Cross-
Sell
Vendor
Scorecards
Ad
Placement
Cyber
Security
Disaster
Mitigation
Investment
Planning
Ad
Placement
Risk
Modeling
Proactive
Repair
Inventory
Predictions
Next
Product Recs
OPEX
Reduction
Historical
Records
Mainframe
Offloads
Device
Data
Ingest
Rapid
Reporting
Digital
Protection
Data
as a
Service
Fraud
Prevention
Public
Data
Capture
INNOVATE
RENOVATE
E X PLO R E O PTIMIZE TR A NS FO R M
ACTIVE
ARCHIVE
ETL
ONBOARD
DATA
ENRICHMENT
DATA
DISCOVERY
SINGLE
VIEW
PREDICTIVE
ANALYTICS
5. 5 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Constrained
High-latency
Localized context
Hybrid – cloud / on-premises
Low-latency
Global context
Core
Infrastructure
Hortonworks DataFlow Manages Data in Motion
Regional
InfrastructureSources
6. 6 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Hortonworks DataFlow Manages Data in Motion
Core
InfrastructureSources
Constrained
High-latency
Localized context
Hybrid – cloud / on-premises
Low-latency
Global context
Regional
Infrastructure
7. 7 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
451 Analyst Report on Stream Processing and Streaming Integration
http://hortonworks.com/info/value-streaming-integration/
8. 8 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Dataflow Management
9. 9 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Problems Today: Timely Access to Data and Decisions
http://diginomica.com/2016/04/22/royal-mail-starts-to-deliver-on-hortonworks-data-in-motion-promise
“HDF helps us to streamline the flow
of data and build models and
visualisations quickly, so that my team
can work iteratively with business
colleagues on building solutions
that work for the business.“
Royal Mail
10. 10 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
HDP
HORTONWORKS
DATA PLATFORM
Powered by Apache Hadoop
HDF Makes Big Data Ingest Easy
Complicated, messy, and takes weeks to
months to move the right data into Hadoop
HDP
HORTONWORKS
DATA PLATFORM
Streamlined, Efficient, Easy
HDP
HORTONWORKS
DATA PLATFORM
Powered by Apache Hadoop
11. 11 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Hortonworks DataFlow, Powered by Apache NiFi. Demo Time
12. 12 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Create a live dataflow in minutes
How would that change your business?
13. 13 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Add processor for data intake. Time: 1 minute
1 Drag and drop processor from top menu
14. 14 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Choose the specific processor
2 Choose one of the processors – currently 170+ available
15. 15 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Example: Pick Twitter Processor
16. 16 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Configure the processor. Time: 2 minutes
3
4
Select processor and choose
option to Configure
Adjust
parameters as
required
17. 17 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Another processor for data output. Time: 1 minute
5
6 Filter for and select a “Put” processor
Drag and drop processor from top menu
18. 18 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Configure second processor. Time: 1 minute
7 Configure 2nd processor
19. 19 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Connect processors, configure connection. 2 minutes
Configure Connection8
Note: Sample Flow is different from previous example of PutHDFS. This dataflow is PutFile. Same concepts apply.
20. 20 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Click Start to Begin Processing. Time total: 7 minutes
9 Click start “play” to being processing
(will run continuously until you select stop)
22. 22 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Core
Infrastructure
Hortonworks DataFlow Use Cases
Regional
InfrastructureSources
Dataflow Management
• On-ramp into Hadoop
• Log Collection / Splunk Optimization
• Cyber Security
• IoT Ingestion
• Deliver data into stream processing engines
Real-time Event Processing
(Kafka, Storm)
Move data between
from on-prem and
cloud environments
23. 23 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Optimize Log Analytics with Content Based Routing
24. 24 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
IoT Data Ingestion
Constrained
High-Latency
Localized Context
Hybrid – Cloud/On-Premise
Low-Latency
Global Context
Resolves real world connectivity and transmission issues often overlooked by assuming connectivity is always perfect
25. 25 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Enterprise Data Movement and Hybrid Cloud
• Seamlessly fuse dataflows between data centers
• Data center to data center,
• Remote location to data center,
• Data center to cloud
HDF
Between Data Centers
HDF
HDF
Remote to Data Center
HDF
HDF
HDF
HDF
Between Data Centers & Cloud
HDF
26. 26 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
© Hortonworks Inc. 2011
Stream Processing
Page 26
Data Acquisition
Edge Processing
Real Time Stream Analytics
Rapid Application Development
IoT
ANALYTICS
CLOUD
28. 28 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Royal Mail’s Journey
Satisfied customers
with better service
levels & improved
retention
Actionable intelligence improves
customer experience
Data management transformed to deliver
specific, actionable insights to line of
business departments
Analysis delivered within days & weeks
rather than months per project deadlines
Churn modelling project identified customers
at risk by vertical sector in order to take
preventative action
Improved accuracy of delivery times for
business customers & highlighted trends
related to volumes of mail expected
Governance & compliance simplified due to
central data platform
Satisfied customers
with better service
levels & improved
retention
P R E D I C T I V E
A N A L Y T I C S
A C T I V E
A R C H I V E
S I N G L E
V I E W
A C T I V E
A R C H I V E
S I N G L E
V I E W
E T L
O N B O A R D
D A T A
E N R I C H M E N T
P R E D I C T I V E
A N A L Y T I C S
D A T A
E N R I C H M E N T
Parcel
distribution
Customer
Acquisition
S I N G L E
V I E W
Customer
Support
Inventory
Predictions
Investment
Planning
Data-as-a-
Service
Public data
capture
Rapid
reporting
EDW offload
OPEX
reduction
Innovate
Renovate
New Data
Products
29. 29 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Open Energi Uses HDF for Electricity Demand Response
As a result of its investment in Hortonworks
Dataflow, Open Energi is Already:
Reducing costs thanks to 10-15% less data
being transmitted across a mobile
network
Creating a full transparent trail for data
provenance that Open Energi can share
with customers
Enabling line of business teams to
contribute to building dataflow rules and
processes
Standardizing the output of data across
various end point devices
Open Energi: hortonworks.com/blog/data-fuel-open-energi-virtual-power-station-hortonworks-dataflow/
30. 30 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Prescient Traveler transformed the travel risk
management market
$0.5MM
Savings in development costs due
to Hortonworks HDF
700%
Improvement in analyst
productivity in determining
actual threats
49,000
Number of data sources currently
being analyzed to identify threats
31. 31 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Centrica’s Journey
1.3 Million
Smart Meters
EDW
Offload
Mobile App for
Customer Sites
Ingest 300
GB per Day
Product
Cross-Sell
Building a Data-Driven
Energy Utility Business
Self-service analytics for 3
million customers in UK & North
America
HDP and HDF simplify IT estate
Ingest of 300 GB/day
rationalizes maintenance work
Personalized customer
communications replaced
impersonal up-sell messages
Legacy EDWs decommisioned
Innovate
Renovate
Smart, Efficient
Homes
D A T A
D I S C O V E R Y
D A T A
E N R I C H M E N T
P R E D I C T I V E
A N A L Y T I C S
S I N G L E
V I E W
A C T I V E
A R C H I V E
E T L
O N B O A R D
SINGLE
VIEW
S I N G L E
V I E W
P R E D I C T I V E
A N A L Y T I C S
On-site customer
data capture
Optimized
engineering
schedule
Tailored
servicing
Customer
sentiment
33. 33 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
From Zero to DataFlow: http://www.slideshare.net/HadoopSummit/from-zero-to-data-flow-in-hours-with-apache-nifi-64032731
34. 34 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Make Streaming Analytics Work For You
http://www.slideshare.net/HadoopSummit/make-streaming-analytics-work-for-you-the-devil-is-in-the-details
35. 35 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Hadoop Summit Keynote: Apache Metron
Ingest log data into their cyber security data lake
https://youtu.be/Nffx8SKn7l4?t=1h37m50s
36. 36 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Hadoop Summit Keynote: Improving Customer Experience
https://youtu.be/BY_0HB9uyXQ
38. 38 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Data Hacks & Demos Keynote: Retail Simulation at Hadoop Summit
Live Voting, Electronic Conversation, Real-Time Facial Recognition
Intro, Demo 1, Demo 2, Demo 3, Demo 4.
https://www.youtube.com/watch?v=BY_0HB9uyXQ&feature=youtu.be&t=49m10s
39. 39 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
GENIVI Alliance: Open Source, In-Vehicle Infotainment Software
The GENIVI Alliance is a nonprofit industry
alliance committed to driving the broad adoption
of specified, open source, In-Vehicle Infotainment
software.
40. 40 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
More Use Cases
Using Apache NiFi to read children’s books” https://twitter.com/KayLerch/status/721455415456882689
41. 41 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Yet More Use Cases
https://twitter.com/4everfusiongal/status/7351585225
39855872
https://www.linkedin.com/pulse/making-rain-apache-nifi-
jeremy-dyer?trk=prof-post
www.linkedin.com/hp/update/6138493082129149952
https://community.hortonworks.com/articles/30636/how-to-simulate-a-
sales-executive-with-hdf.html
42. 42 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Even More Use Cases
https://community.hortonworks.com/articles/47854/accessing-facebook-
page-data-from-apache-nifi.html
https://community.hortonworks.com/content/kbentry/32605/runn
ing-nifi-on-raspberry-pi-best-practices.html
Accessing Facebook Data
43. 43 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
And Even More Use Cases
https://community.hortonworks.com/articles/20318/visualize-patients-
complaints-to-their-doctors-usi.html
Visualize patients' complaints to their doctors
using NiFi and Solr/Banana
http://hortonworks.com/blog/qualcomm-hortonworks-showcase-
connected-car-platform-tu-automotive-detroit/
Connected Car
44. 44 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Questions? Hortonworks Community Connection:
Data Ingestion and Streaming
https://community.hortonworks.com/
Contact Us:
http://hortonworks.com/contact-us/
Editor's Notes Hortonworks: Powering the Future of Data 4 Hortonworks: Powering the Future of Data Hortonworks: Powering the Future of Data Hortonworks: Powering the Future of Data 10 Hortonworks: Powering the Future of Data 35