Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (13)
Similar to Data Engineering Quick Guide
Similar to Data Engineering Quick Guide (20)
More from Asim Jalis
Recently uploaded
Recently uploaded (20)
Data Engineering Quick Guide
- 1. DATA ENGINEERING QUICK GUIDE ASIM JALIS GALVANIZE
- 2. BIG DATA
- 3. WHY HADOOP? How can we create a supercomputer Using cheap Linux boxes?
- 4. WHAT IS HADOOP? Operating system for cluster of machines Combines small weak computers To create Big Data system Unified disk and processing power
- 5. HADOOP
- 7. WHY HDFS? How can we store a petabyte-sized file Using cheap Linux boxes?
- 8. WHAT IS HDFS? Split petabye file into 128 MB blocks Distribute blocks across Hadoop cluster Make 3 copies of each block for insurance
- 9. HDFS
- 10. WHY MAPREDUCE? How can we process the data in HDFS Without pulling it out and pushing the result back?
- 11. WHAT IS MAPREDUCE? Send program to where the data is on HDFS Process petabyte file by processing each block Then combining the result
- 12. MAPREDUCE
- 13. WHY HIVE? How can people who don’t know Java Write MapReduce jobs?
- 14. WHAT IS HIVE? Hive translates SQL to MapReduce jobs
- 15. HIVE SELECT * FROM sales WHERE amount > 400;
- 16. HIVE
- 17. WHY PIG? How can people who don’t know Java or SQL Write MapReduce jobs?
- 18. WHAT IS PIG? Pig translates PigLatin to MapReduce jobs PigLatin is a scripting language comparable to SQL
- 19. PIG high_sales = FILTER sales_data BY amount > 400;
- 20. PIG
- 21. WHY SPARK? How can we make MapReduce faster And the API less clunky?
- 22. WHAT IS SPARK? Spark is like MapReduce Spark has a cleaner API and is faster Speed up because it saves intermediate results in memory
- 23. SPARK sc.textFile("shakespeare.txt"). flatMap(line => line.split("W+")). map(word => (word,1)). reduceByKey((count1,count2) => (count1 + count2)). saveAsTextFile("output")
- 24. SPARK
- 25. WHY SPARK SQL? How can people who don’t know Scala, Python, or Java Write Spark code?
- 26. WHAT IS SPARK SQL? Spark SQL is like Hive for Spark Hive translates SQL to MapReduce Spark SQL translates SQL to Spark
- 27. SPARK SQL SELECT * FROM sales WHERE amount > 400;
- 28. SPARK SQL
- 29. WHAT IS SPARK MLLIB? Machine Learning algorithms on Spark Analyze data to extract insights
- 30. WHAT IS MACHINE LEARNING? Technique Question Regression Predict revenue next month Classification Is tumor cancerous or benign Clustering Which customers are similar to each other Recommendation Which movie will you like
- 32. WHAT IS THE DIFFERENCE BETWEEN REAL-TIME AND BATCH? Term Means Example Real- Time Process data when it arrives Reject credit card transaction Batch Process data periodically Flag suspicious transaction at night
- 33. BATCH Processing Layer SQL Layer MapReduce Hive, Pig Spark Spark SQL
- 35. WHY HBASE? How can we store petabytes of data on HDFS And do fast read and writes like a database?
- 36. WHAT IS HBASE? HBase is a NoSQL database on top of HDFS Can store petabytes of data Reads/writes much faster than traditional database and HDFS
- 37. HBASE
- 38. WHY KAFKA? How can we hold onto incoming data and not lose it When we are getting a million messages per second?
- 39. WHAT IS KAFKA? Kafka is TiVo for the cluster It stores real-time data as it comes in Can store a week of data Queuing system for Hadoop cluster
- 40. KAFKA
- 41. WHY SPARK STREAMING? How can we process data as it comes in Instead of every night (using Spark or MapReduce)
- 42. WHAT IS SPARK STREAMING? Spark Streaming is a library on top of Spark It allows processing data as soon as it comes in Sits in front of Kafka
- 43. SPARK STREAMING
- 44. WHY LAMBDA ARCHITECTURE? How can we watch historical trends and what is happening right now? How can we show bestsellers from this year and from last hour?
- 45. WHAT IS LAMBDA ARCHITECTURE? Big Data system which can handle both batch and real- time Uses historical data as well as real-time data Best of both worlds
- 47. REVIEW
- 48. BATCH REVIEW Technology Description Hadoop Cluster operating system HDFS Stores petabytes of data on 100s or 1000s of machines MapReduce Processes data in HDFS Hive SQL MapReduce Pig PigLatin MapReduce Spark Faster MapReduce Spark SQL SQL Spark
- 49. REAL-TIME REVIEW Technology Description HBase Fast NoSQL database on top of HDFS Kafka Queues incoming data into cluster Spark Streaming Process in real-time Lambda Architecture Combines real-time and batch