В докладе Игорь продемонстрирует архитектуру хранилища, базирующуюся на технологиях от Cloudera и Oracle. Он расскажет об опыте интеграции множества источников данных с помощью самописных решений и использованием специализированных инструментов, как Apache NiFi. Выступление заинтересует технических специалистов, которые уже знакомы со стеком Hadoop. Интересуетесь анализом данных? Присоединяйтесь к нашей группе на Facebook: https://www.facebook.com/groups/DataTalks/

1. BUILDING A DATA WAREHOUSE
WITH HADOOP
10.10.2015
IGOR NAKHVAT, DATA INTEGRATION ENGINEER

2. CONTENT TABLE
I. Building a Data Warehouse with Hadoop
A.Data sources
B.Data storage
C.Data flow
D.ETL tool
E.Conclusions

3. 3
DATA SOURCES
GamesDATA SOURCES
DATA STORAGE
DATA FLOW
ETL TOOL
CONCLUSIONS

4. 4
SPA Payment ForumeSport
Clan warsUpdate
DATA SOURCES
ServicesDATA SOURCES
DATA STORAGE
DATA FLOW
ETL TOOL
CONCLUSIONS

5. 5
DATA SOURCES
GeographyDATA SOURCES
DATA STORAGE
DATA FLOW
ETL TOOL
CONCLUSIONS

6. 6
DATA SOURCES
Total: 294222
72
Tables
Total: 1264
Relational databasesDATA SOURCES
DATA STORAGE
DATA FLOW
ETL TOOL
CONCLUSIONS

7. 7
DATA SOURCES
Non - Relational data sourcesDATA SOURCES
DATA STORAGE
DATA FLOW
ETL TOOL
CONCLUSIONS

8. 8
• It is an open architecture.
• Cost effective.
• Many interfaces to data (SQL, Spark, Java, Scala, Python).
• Many ways/formats for storing the data.
• Many tools available for the data analytics.
DATA STORAGE
Why Hadoop?DATA SOURCES
DATA STORAGE
DATA FLOW
ETL TOOL
CONCLUSIONS

9. 9
Keep in mind
• Lack of employees
• Security
DATA STORAGE
Why Hadoop?DATA SOURCES
DATA STORAGE
DATA FLOW
ETL TOOL
CONCLUSIONS

10. 10
DATA STORAGE
Hadoop ecosystem
SQOOP
PIG
DATA SOURCES
DATA STORAGE
DATA FLOW
ETL TOOL
CONCLUSIONS

11. 11
DATA STORAGE
How HDFS works?DATA SOURCES
DATA STORAGE
DATA FLOW
ETL TOOL
CONCLUSIONS

12. 12
DATA STORAGE
How HDFS works?DATA SOURCES
DATA STORAGE
DATA FLOW
ETL TOOL
CONCLUSIONS

13. 13
DATA STORAGE
How HDFS works?DATA SOURCES
DATA STORAGE
DATA FLOW
ETL TOOL
CONCLUSIONS

14. 14
DATA FLOW
ShellSQL CSV + GZIP
Check count
rows
Compute
stats
Parquet
DATA SOURCES
DATA STORAGE
DATA FLOW
ETL TOOL
CONCLUSIONS

15. 15
DATA FLOW
Parquet
Shell
DATA SOURCES
DATA STORAGE
DATA FLOW
ETL TOOL
CONCLUSIONS

16. 16
DATA FLOW
Parquet
DATA SOURCES
DATA STORAGE
DATA FLOW
ETL TOOL
CONCLUSIONS

17. 17
DATA FLOW
Shell
Aggregation
Presentation
Audience
Balance
Finance
Data
scientist
Manager
DATA SOURCES
DATA STORAGE
DATA FLOW
ETL TOOL
CONCLUSIONS

18. 18
ETL TOOL
Continuous integration tool
+ =
ETL toolPlugins
5 - 20 Hours 1000+ Jobs
Jenkins
DATA SOURCES
DATA STORAGE
DATA FLOW
ETL TOOL
CONCLUSIONS

19. 19
ETL TOOL
Apache NiFi
DATA SOURCES
DATA STORAGE
DATA FLOW
ETL TOOL
CONCLUSIONS

20. 20
Apache NiFi
• Drag – n – drop works!
• Great visualization.
• Data provenance.
• Flow can be modified at runtime.
ETL TOOL
DATA SOURCES
DATA STORAGE
DATA FLOW
ETL TOOL
CONCLUSIONS

21. 21
Apache NiFi
Keep in mind
• Multiuser development.
• No templates.
• NiFi is not an orchestration tool.
ETL TOOL
DATA SOURCES
DATA STORAGE
DATA FLOW
ETL TOOL
CONCLUSIONS

22. 22
CONCLUSIONS
DATA SOURCES
DATA STORAGE
DATA FLOW
ETL TOOL
CONCLUSIONS
• Hadoop is good for data warehousing
• Poor Hadoop security
• Impala (SQL on Hadoop) performs and scales
• Data format choice is a key (Avro, Parquet)

23. TANKS A LOT!