Hadoop Ecosystems overview and diagrams - helps to understand list of subprojects in hadoop in diagramatic way.

1. Hadoop Technologies
Architecture Overview
@senthil245
Mail - senthil245@gmail.com

2. DISTRIBUTED CLUSTER ARCHITECTURE: MASTER/SLAVE

3. HADOOP CORE

4. MAPREDUCE PATTERNS

5. WHEN MAPREDUCE
Since the MapReduce is running within a
cluster of computing nodes, the architecture is
very scalable.
• In other words, if the data size is increased by
the factor of x, the performance should be still
constant if we are adding a predictable/fixed
factor of y.
The graph on the right is illustrating the
relationship between the size of the data (xaxis) and processing time (y-axis).
•The blue color curve is the process using
traditional programming. On the other hand, the
black color curve is the process using Hadoop.
When the data size is small, traditional
programming is better performance because the
bootstrap of Hadoop is expensive (Copy the data
within the cluster, inter-nodes communication,
etc.).
Once the data size is big enough, the penalty
of the Hadoop bootstrap becomes invisible.
•Hence Hadoop is best suited for Big Data
crunching ideally in terms of petaBytes and is
not suited for implementing common data
integration patterns

7. APACHE SQOOP

8. APACHE FLUME

9. APACHE CHUKWA

10. HDFS

11. APACHE OOZIE – WORKFLOW SCHEDULER (CHECK AZKABAN & LINKEDIN OPENSOURCE)

12. PIG AND HQL (DO
NOT USE
HQL)

13. APACHE S4 (STREAM PROCESSING)(ALSO CHECK KAFKA
AND
STORM)

14. APACHE ZOOKEEPER SERVICE (ALSO CHECK APACHE HUE)

15. APACHE HIVE

16. APACHE HCATALOG, HIVE
AND
HBASE