講師：Dell Enterprise Technologist 李俊邦 (CP Li)

1. 解構⼤大數據架構
⼤大數據系統的伺服器與網路資源規劃
“How to eat an elephant – one byte at a time”
CP Li 李俊邦
Enterprise Technologist
Enterprise Solutions & Alliances, Greater China
Dell

2. 2
議程
1. 不同的伺服器⾓角⾊色
1. Manager
2. Name Nodes
3. Edge Nodes
4. Data Nodes
2. Hadoop Cluster設計
3. Etu+Dell
4. Futures / Roadmap
5. Questions?

3. 3
Server Roles - Manager
• 系統安裝圖形介⾯面/ 主控台
• ⼤大多安裝在Edge Node
• 常⾒見版本
– Cloudera Manager
– Apache Ambari

4. 4
Server Roles – Name Nodes
• 存放HDFS的metadata
• Job Manager for YARN data-processing framework
• Primary
– Heartbeats from data nodes
– 10th heartbeat is a block report from which it generates
metadata
• Standby
– Checks in every hour to mirror metadata / block map
– Not a hot-spare – requires manual fail-over
• High Availability (HA) can be added in some
distributions
– Results in a dedicated HA node that acts as a witness
to the Name Node cluster

5. 5
Server Roles - Edge Nodes
• 資料進出Hadoop叢集的主要端⼝口
• 可擴展
• Hadoop叢集裡唯⼀一的多網段節點
PowerEdge
R730
–
Name
Node
PowerEdge
R730
–
Standby
Name
Node
PowerEdge
R730
–
Edge
Node(s)
PowerEdge
R730
–
HA
Node
Corporate
Network Data
Network
Corporate
Data
Network
Data
Network
Data
Network
Data
Network
PowerEdge
R730XD
–
Data
Nodes
Data
Network

6. 6
Server Roles - Data Node
• HDFS的主要存放處
• 執⾏行YARN資源管理所指定的資料處理
• 主要屬性
– 記憶體
› 標配64GB
› 更多服務（Impala/Spark) 需要更多記憶體
– 很多的本地硬碟 (JBOD / Non-RAID mode)
› SFF (2.5”) for performance-based workloads
› LFF (3.5”)for capacity-centric workloads
– CPUs – legacy recommendation of 1:1 core:spindle ratio
› SSDs, faster HDD (10K+), and in-memory workloads make this less of an issue
› 10 and 12 core are the best practice default

7. Hadoop Cluster
Design

8. 8
Hadoop Cluster Design – Hardware Considerations

9. 9
Hadoop Cluster Deployment – Installation Best
Practices
• Use pre-built, assembled & cabled racks from vendor
• ⾃自動佈署⼯工具 (ex: Open Crowbar)
• Purchase nodes in standard size groups for easy capacity growth and ordering, not in single node
increments
– Common increments are ½ or full rack for easy deployment and sizing
• For each type of hardware, purchase spare components to keep on site for easy, rapid repair

10. 10
Core Hadoop Use Cases
歸檔
⾼高硬碟／CPU⽐比
記憶體使⽤用低
法規需求
⻑⾧長期歸檔
資料處
理
⾼高硬碟／CPU⽐比
記憶體使⽤用中等
DW offload
ETL offload
EDH
質量分析
IT Log分析
分析
⾼高核⼼心數
記憶體使⽤用⾼高
市場分析
詐欺預防
網路分析

11. 11
Common Hadoop Use Case to Ecosystem Tool Mapping

12. 12
Hadoop Use Case to Ratio Mapping
歸檔
1:2:1
資料處理
1:4:1
分析
2:8:1
CPU (Cores) : Memory (GB) : Disk (數量) – Data Node

13. 13
Node Considerations
Dell PowerEdge R730 Dell PowerEdge R730 Dell PowerEdge R730
Dell PowerEdge R730xd

14. 14
Node Considerations

15. 15
HDFS Capacity
• HDFS protects information through replication of the data between nodes, the default Replication
Factor is 3, but is configurable.
• HDFS Raw Capacity = Number of Compute Nodes x Number of Drives x Capacity of Drives
• HDFS Usable Capacity = HDFS Raw Capacity/Replication Factor

16. 16
Big Data Networking Best Practices
• Traditional Ethernet is used since it’s affordable and already prevalent.
• 1GbE networking was used initially in early drafts of the solution but with the reduction in cost it’s
much more efficient to go with 10GbE.
• Multiple ports are teamed both for redundancy and throughput. LACP or software bonding are the
most common methods.
• IPv4 is most widely used. IPv6 has limited support at the OS and Hadoop level.

17. 17
Attributes of a Good Switch for Big Data
• Non-blocking backplane
• Deep per-port packet buffers (shared buffers do not work well). During sort/shuffle phases of
map/reduce operations network traffic is so chaotic that it can saturate any and all shared buffers,
impacting multiple host’s network performance.
• Good choices:
– 1GbE
› S55
› S60
– 10GbE
› S4810
› S5000
– 40GbE
› Z9000
› Z9500
› S6000

18. 18
Dell Hadoop Solution Logical Diagram

19. 19
Scale-out Aggregation Layer

20. 20
Dell Points of Integration
• VLT / VRRP is a very affordable way to team switches both at the ToR and the aggregation tiers.
This makes the Dell Networking Force10 switches a great choice.
• Active Fabric Manager
– Speeds up the creation and administration of the required VLT / VRRP configuration on the switches.
– Helps with capacity-planning as customer scale

21. 21
Big Data Networking Futures
• 40GbE onboard LOMs will begin to be used for high-volume clusters. Right now the cost:benefit
ratio isn’t there yet.
• As HPC and Big Data converge, we’ll start to see the use of IB for node-to-node connectivity.
• In-memory (Spark / Impala) workloads are reducing the bottlenecks that used to exist at the disk
and now move to the processor and network. Expect customers to be looking to increase core
counts and network speed to overcome this.

22. @Dell_Enterprise Enterprise Solutions
Etu+Dell = complete Hadoop/Big Data solution provider
Best of breed
Cloudera partners
- Etu
Analytic software
solutions for Big Data
Dell Professional Services for Big Data
Dell PowerEdge
13G servers
Dell Networking
solutions
Installation and configuration service
Complete end-to-end implementation
Discover Plan ImplementInvestigate

23. 2. Store1. Integrate
4. Act
3. Analyze
Solution architecture
Analytical output
Toad Data Point
Desktop – integrate, cleanse
Dell Boomi
Cloud – integrate, correlate
Toad Intelligence
Central
Data aggregation
and virtualization
Dell STATISTICA
Customer data
Order data
Events
Stock market data
Advanced
Analytics
Marketing campaigns
Dell Statistica Big Data
Desktop – crawl, save
Social Media

24. 24
Futures
• Speed Improvements in Map / Reduce
• More in-memory workloads
– Possible move to Spark to replace Map/Reduce
• Virtualized Hadoop
– VMWare Big Data Extensions
– Openstack Sahara
– Microsoft HDInsights (Hortonworks)

25. 25
Dell In-Memory Appliance for Cloudera Enterprise
Configurations at a glance
Mid-Size Configuration
16 Node Cluster
PowerEegeR720- 4 Infrastructure Nodes
with ProSupport
PowerEdgeR720XD- 12 Data Nodes with
ProSupport
Cloudera Enterprise
Force10- S4810P
Force10- S55
Dell Rack 42U
~528TB (disk raw space)
Starter Configuration
8 Node Cluster
PowerEdge R720- 4 Infrastructure Nodes
with ProSupport
PowerEdgeR720XD- 4 Data Nodes with
ProSupport
Cloudera Enterprise
Force10- S4810P
Force10- S55
Dell Rack 42U
~176TB (disk raw space)
Small Enterprise
Configuration
24 Node Cluster
PowerEdgeR720- 4 Infrastructure Nodes
with ProSupport
PowerEdgeR720XD- 20 Data Nodes with
ProSupport
Cloudera Enterprise
Force10- S4810P
Force10- S55
Dell Rack 42U
~880TB (disk raw space)
Expansion Unit- PowerEdgeR720XD-4 Data Nodes w ProSupport, Cloudera Enterprise, Scales in
Blocks