Introduction to Apache NiFi - Seattle Scalability Meetup

Introducing #ApacheNiFi
Saptak Sen [@saptak]
Technical Product Manager, Hortonworks
© Hortonworks Inc. 2011 – 2015. All Rights Reserved
#seascale

Agenda
• New Data Sources and the Rise of the Internet of Anything
• Introducing: Hortonworks DataFlow powered by Apache NiFi
• Key concepts, architecture, and use cases
• Demo
• Q&A

IoAT Data Grows Faster Than We Consume It
Much of the new data
exists in-flight, between
systems and devices as
part of the Internet of
AnythingNEW
TRADITIONAL
The Opportunity
Unlock transformational business value
from a full fidelity of data and analytics
for all data.
Geolocation
Server logs
Files & emails
ERP, CRM, SCM
Traditional Data Sources
Internet of Anything
Sensors
and machines
Clickstream
Social media

Interconnectedness Demands User Centricity
Changes Organizations into Data Companies
Hortonworks Data Platform
for rich historical insights
from data-at-rest
NEW Hortonworks DataFlow
for securely collecting,
conducting, and curating
data-in-motion while ALSO
driving value for data-at-rest
analytics and use cases
Source: Gartner - Architecture Options for Big Data Analytics on Hadoop, July 2015

Simplistic View of IoAT & Data Flow
The Data Flow Thing
Process and
Analyze Data
Acquire Data
Store Data

Global interactions with customers, business partners, and things
spanning different volume, velocity, bandwidth, and latency needs
Realistic View of IoAT and Data Flow

Meeting IoAT Edge Requirements
GATHE
R
DELIVER
PRIORITIZE
Track from the edge Through to the datacenter
Small Footprints
operate with very little power
Limited Bandwidth
can create high latency
Data Availability
exceeds transmission bandwidth
Data Must Be Secured
throughout its journey

Hortonworks Acquires Onyara
Turn Internet of Anything Data Into Actionable
Insights
• Onyara is the creator of and key contributor to Apache NiFi,
an open source solution for processing and distributing data.
• Over the past 8 years, Onyara engineers developed the U.S.
government software project called “Niagara Files”, the
precursor to Apache NiFi.
• Apache NiFi was made available as an Apache Incubator
project through the NSA Technology Transfer Program in the
Fall of 2014.
NEW Hortonworks DataFlow offering will
securely and easily collect, conduct and curate
any data, from anything, anywhere.

The IoAT Data Flow
powered by Apache Hadoop
powered by Apache Hadoop
Enrich
Context
Store Data
and Metadata
Internet
of Anything
Hortonworks DataFlow
powered by Apache NiFi
Perishable
Insights
Historical
Insights
Introducing Hortonworks DataFlow powered by
Apache NiFi
Hortonworks DataFlow and the Hortonworks Data Platform
deliver the industry’s most complete solution for management of Big Data.

Apache NiFi: Three key concepts
• Manage the flow of information
• Data Provenance
• Secure the control plane and data plane

Apache NiFi – Key Features
• Guaranteed delivery
• Data buffering
- Backpressure
- Pressure release
• Prioritized queuing
• Flow specific QoS
- Latency vs. throughput
- Loss tolerance
• Data provenance
• Recovery/recording
a rolling log of fine-
grained history
• Visual command and
control
• Flow templates
• Pluggable/multi-role
security
• Designed for extension
• Clustering

Common Apache NiFi Use Cases
Predictive Analytics
Ensure the highest value data is captured and available for analysis
Compliance
Gain full transparency into provenance and flow of data
IoT Optimization
Secure, Prioritize, Enrich and Trace data at the edge
Fraud Detection
Move sales transaction data in real time to analyze on demand
Big Data Ingest
Easily and efficiently ingest data into Hadoop
Value Resources
Gain visibility into how data sources are used to determine value

OS/Host
JVM
Flow Controller
Web Server
Processor 1 Extension N
FlowFile
Repository
Content
Repository
Provenance
Repository
Local Storage
OS/Host
JVM
Flow Controller
Web Server
FlowFile
Repository
Content
Repository
Provenance
Repository
Local Storage
Architecture
OS/Host
JVM
NiFi Cluster Manager – Request Replicator
Web Server
Master
NiFi Cluster
Manager (NCM)
OS/Host
JVM
Flow Controller
Web Server
FlowFile
Repository
Content
Repository
Provenance
Repository
Local Storage
Slaves
NiFi Nodes
High Availability: Control plane vs Data plane…

HDF – Powered by Apache NiFi

Add processor for data intake
1 Drag and drop processor icon from the top menu

Choose the specific processor
2 Choose one of the processors – currently 90 available – designed for extension

Example: Pick Twitter Processor

Configure the processor
3 Select processor and
choose option to Configure
4
Adjust
parameters as
required

Another processor for data output
5 Drag and drop processor icon from the top menu
6 Example: choose PutHDFS processor

Configure second processor
7 Configure 2nd processor

Connect processors, configure connection
8

Click Start to begin processing
9

See processors update with real time changes
10
As data flows, GUI interface updates in real
time.

Dynamically adjust and tune data flow as needed
11 Dynamically adjust and tune dataflow as needed, in
real time. Can also replicate data for testing and
comparison.

Understand the data path with Data Provenance
14 Select Data Provenance

Trace lineage of a particular piece of data
15
Icon for Data Lineage

Every change to data is tracked: processing, views
16
Provenance event is tracked

Updates as changes happen
17 Updates as data flows

Easily access and trace changes to dataflow

Audit trail of Hortonworks DataFlow User Actions

Operations: Planned

Introduction to Apache NiFi - Seattle Scalability Meetup

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Introduction to Apache NiFi - Seattle Scalability Meetup

Similar to Introduction to Apache NiFi - Seattle Scalability Meetup (20)

More from Saptak Sen

More from Saptak Sen (9)

Recently uploaded

Recently uploaded (20)

Introduction to Apache NiFi - Seattle Scalability Meetup

Editor's Notes