Future of Data: Apache NiFi 勉強会 〜データフローの自動化〜 http://futureofdata.connpass.com/event/35428/ で発表したスライドです。皆さん是非ダウンロードしてNiFi, HDFを使ってみてください!

1. Koji Kawamura
Hortonworks Software Engineer
July 27, 2016

2. 2 © Hortonworks Inc. 2011 –2016. All Rights Reserved
- Hortonworks DataFlow Team – Software Engineer
- ijokarumawak
- kkawamura hortonworks.com

3. 3 © Hortonworks Inc. 2011 –2016. All Rights Reserved
Agenda
HDF, Apache NiFi
Step 1.
Step 2.
Step 3.

4. 4 © Hortonworks Inc. 2011 –2016. All Rights Reserved
DataFlows

5. 5 © Hortonworks Inc. 2011 –2016. All Rights Reserved

6. 6 © Hortonworks Inc. 2011 –2016. All Rights Reserved
The Connected Data Platform

7. 7 © Hortonworks Inc. 2011 –2016. All Rights Reserved
H O R T O N W O R K S D A T A F L O W
H O R T O N W O R K S D A T A F L O W
HORTONWORKS DATAFLOW

8. 8 © Hortonworks Inc. 2011 –2016. All Rights Reserved
&
Powered by
Apache NiFi

9. 9 © Hortonworks Inc. 2011 –2016. All Rights Reserved
8 NSA
NSA
-- NSA
Linda L. Burger
NiFi

10. 10 © Hortonworks Inc. 2011 –2016. All Rights Reserved
HTTP
Syslog
HTML
HL7
FTP
UDP
XML
SFTP

11. 11 © Hortonworks Inc. 2011 –2016. All Rights Reserved

12. 12 © Hortonworks Inc. 2011 –2016. All Rights Reserved
Agenda
HDF, Apache NiFi
Step 1.
Step 2.
Step 3.

13. 13 © Hortonworks Inc. 2011 –2016. All Rights Reserved
à Apache NiFi
à HDP
à Docker
à Apache NiFi

14. 14 © Hortonworks Inc. 2011 –2016. All Rights Reserved
$ ./bin/nifi.sh start
$ ./bin/nifi.sh install
[1] https://nifi.apache.org/download.html
[2] https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#system_properties

15. 15 © Hortonworks Inc. 2011 –2016. All Rights Reserved
HDF
[1] http://hortonworks.com/downloads/#dataflow

16. 16 © Hortonworks Inc. 2011 –2016. All Rights Reserved

17. 17 © Hortonworks Inc. 2011 –2016. All Rights Reserved
Agenda
HDF, Apache NiFi
Step 1.
Step 2.
Step 3.

18. 18 © Hortonworks Inc. 2011 –2016. All Rights Reserved
Step 1 :

19. 19 © Hortonworks Inc. 2011 –2016. All Rights Reserved
Ã
Ã
à Flow File
à Processor
Ã
à Processor
à Processor
à Relationship

20. 20 © Hortonworks Inc. 2011 –2016. All Rights Reserved
Agenda
HDF, Apache NiFi
Step 1.
Step 2.
Step 3.

21. 21 © Hortonworks Inc. 2011 –2016. All Rights Reserved
Step 2 :
logAttributes
^([d]{4}-[d]{2}-[d]{2}
[d]{2}:[d]{2}:[d]{2},[d]{3})
([A-Z]+) (.+)$

22. 22 © Hortonworks Inc. 2011 –2016. All Rights Reserved
à Data Flow
à Process Group
à Flow File
Ã
à TailFile
à Flow File
à Dynamic Property

23. 23 © Hortonworks Inc. 2011 –2016. All Rights Reserved
NiFi
NiFi
*_repository
logs, run, state, work

24. 24 © Hortonworks Inc. 2011 –2016. All Rights Reserved
Agenda
HDF, Apache NiFi
Step 1.
Step 2.
Step 3.

25. 25 © Hortonworks Inc. 2011 –2016. All Rights Reserved
Step 3 :

26. 26 © Hortonworks Inc. 2011 –2016. All Rights Reserved
Ã
Ã
Ã
Ã

27. 27 © Hortonworks Inc. 2011 –2016. All Rights Reserved

28. 28 © Hortonworks Inc. 2011 –2016. All Rights Reserved
Provenance

29. 29 © Hortonworks Inc. 2011 –2016. All Rights Reserved

30. 30 © Hortonworks Inc. 2011 –2016. All Rights Reserved
Thank You