Multi-Tenant HBase Cluster - HBaseCon2018-final

© 2018 Bloomberg Finance L.P. All rights reserved.
Me to We: Moving to a
Multi-Tenant HBase Cluster
HBaseCon 2018
June 18, 2018
Biju Nair

Outline
• Need for multi-tenancy
• HBase features for multi-tenancy
• Experience with multi-tenancy features
• Summary

Bloomberg in a nutshell
The Bloomberg Terminal delivers a
diverse array of information on a single
platform to facilitate financial decision-
making.

Bloomberg by the numbers
• Founded in 1981
• 325,000 subscribers in 170 countries
• Over 19,000 employees in 192 locations
• More News reporters than The New York Times + Washington Post + Chicago Tribune

Bloomberg technology by the numbers
• 5,000+ software engineers
• 150+ technologists and data scientists devoted to machine learning
• One of the largest private networks in the world
• 120 billion pieces of data from the financial markets each day, with a peak of more than
10 million messages/second
• 2 million news stories ingested / published each day (500+ news stories ingested/second)
• News content from 125K+ sources
• Over 1 billion messages and Instant Bloomberg (IB) chats handled daily

HBase at Bloomberg
• Started with v0.94.6
• 2B+ reads per day
• 1B+ writes per day
• 51+ TB of compressed data stored in HBase

One Cluster Per Application
Photo By: Todd Pruden, Fort Hood Public Affairs

Consolidation

Multiple Applications Per Cluster

Streamline Use of Clusters

Cost
• 550 Wh – 4,800 KWh/Year
• Greenhouse gas emissions – 8,755 miles of driving a car
• CO2 emissions from – 0.535 of 1 year home electricity use
• Sequestered by – 4.2 acres of U.S. forests in 1 year
https://www.zdnet.com/article/toolkit-calculate-datacenter-server-power-usage/
https://www.epa.gov/energy/greenhouse-gas-equivalencies-calculator

Namespace & ACL

Namespace & ACL
• hbase> create_namespace ‘PH‘
• create ' PH:red', 'f1'
• hbase> grant ‘phadmin', 'RWXCA', '@PH‘
• hbase> revoke ‘phadmin', '@PH‘
• Updates to hbase-site.xml
— hbase.security.authorization
— hbase.coprocessor.region.classes
— hbase.coprocessor.master.classes
— hbase.coprocessor.regionserver.classes
— hbase.security.exec.permission.checks

Region Server Groups

Architecture
HMaster
RS1 RS2 RS3 RS4 RS5 RS6 RS7 RS8
system
ZK

Internals
HMaster
RSGroupAdminEndpoint
RSGroupBasedLoadBalancer
FilterBy
Group
LoadBalancer
RSGroupInfoManager
RSGroup
Table
RSGroup
on ZK
Achieving HBase multitenancy with regionserver groups and favored nodes – DataWorks Summit 2017

RSgroup Creation
<property>
<name>hbase.coprocessor.master.classes</name>
<value>org.apache.hadoop.hbase.rsgroup.RSGroupAdminEndpoint</value>
</property>
<property>
<name>hbase.master.loadbalancer.class</name>
<value>org.apache.hadoop.hbase.rsgroup.RSGroupBasedLoadBalancer</value
</property>

RSgroup Creation
• add_rsgroup ‘derivatives’
• move_servers_rsgroup ‘derivatives’,[‘rs1:16020’,’rs2:16021’]
• move_tables_rsgroup ‘derivatives’,[‘tbl1’,’tbl2’]
• list_rsgroups ‘derivatives’
• move_servers_namespaces_rsgroup ‘derivatives’,[‘ns1’,’ns2’]*
• create_namespace 'ns',{'hbase.rsgroup.name' => ’derivatives'}

Storage Quota

Internals
Master
1 Create Quota
3 Compute Space
4 Update Current Use
Region Server
2 Report region size
5 Read Current Use
6 Enforce Policy
HBase
Quota

Enable Quota
<property>
<name>hbase.quota.enabled</name>
<value>true</value>
</property>

Space Quota Creation
• set_quota TYPE => SPACE, TABLE => 'tb1', LIMIT => '100G', POLICY =>
NO_INSERTS
• set_quota TYPE => SPACE, NAMESPACE => 'ns', LIMIT => '150T', POLICY
=> NO_WRITES
• Other Policies: NO_WRITES_COMPACTIONS, DISABLE, NONE
• Table level quota takes precedence
• Preserve quota definition - hbase.quota.remove.on.table.delete

CPU/Network Quota

Internals
Master
Create Quota
Region Server
Verify current use and
restrict
HBase
Quota

Query Throttling
• set_quota TYPE => THROTTLE, USER => 'u1', LIMIT => '10req/sec'
• set_quota TYPE => THROTTLE, NAMESPACE => 'ns1', LIMIT => '10req/sec'
• set_quota TYPE => THROTTLE, THROTTLE_TYPE => WRITE, TABLE => 't1',
LIMIT => '10M/sec'
• set_quota USER => 'u1', GLOBAL_BYPASS => true

Meta-Data Quota

Internals
Master
Create Quota
Verify Quota Use
Region Server
Region Split
HBase
Quota

Meta-Data Quota Creation
• create_namespace 'ns1', {'hbase.namespace.quota.maxregions'=>'10'}
• create_namespace 'ns2', {'hbase.namespace.quota.maxtables'=>'2’}
• alter_namespace 'ns3', {METHOD => 'set',
'hbase.namespace.quota.maxtables'=>'5’}
• alter_namespace 'ns3', {METHOD => 'unset', NAME =>
'hbase.namespace.quota.maxtables'}

Request Queues

Request Queues
• Number of request handler per region server
— hbase.regionserver.handler.count
• Number of queues per request handler
— hbase.ipc.server.callqueue.handler.factor
• read vs write queue ratio
— hbase.ipc.server.callqueue.read.ratio
• get vs scan queue ratio
—hbase.ipc.server.callqueue.scan.ratio

Rubber meets the road

RSGroup - Performance
678189 687062 667110
760310 758847
815204
Run 1 Run 1 Run 3
Write Performance Avg
Avg Avg RG
444978
481672
520719
619840 606806 603353
Run 1 Run 1 Run 3
Write Performance Min
Min Min RG

RSGroup – Data Locality
0.00%
8.16%
1.87%
23.81%
10.01%
2.66%
14.76%
4.90%
15.43% 14.71%
16.53% 15.99%
6.32%
21.72%
No Fail Run 1 Run 2 Run 3 Run 4 Run 5 Run 6
Non local blocks
Shared Cluster RS Group

RSGroup
• Replication doesn’t recognize RSgroup
— Vote for HBASE-15531 J
• RSes in groups should be distributed across racks
• Group for SYSTEM tables should be large to withstand failures
• Failure of RSes in a group requires master restart for recovery
— HBASE-20728
• Reverting RSgroup feature requires additional steps
• RSgroup uses SLB and hence supports region replication

RSGroup
• Altering namespace to set RSGroup is not supported in 1.x
• Failure of all RSes in SYSTEM RSgroup makes cluster unavailable
— HBASE-20632
• JIRA’s
— HBASE-20566
— HBASE-20606
— HBASE-20607
— HBASE-20666

Space Quota
Limit, 10
Actual, 210.9
0
50
100
150
200
250
300
350
Run 1 - def Run 2 - def Run3 - def Run 4 - 60 sec Run 5 - 30 sec
StoragesizeinGB
Space Quota - Limit vs Actual

Space Quota
• Space Quota Properties
— hbase.regionserver.quotas.policy.refresher.chore.period
— hbase.regionserver.quotas.policy.refresher.report.percent
— hbase.regionserver.quotas.region.size.reporting.chore.period
— hbase.master.quotas.observer.chore.period
• HBASE-18133
— Low-latency space quota size reports
• HBASE-20588
— LIMIT => NONE doesn’t take effect
• HBASE-20662
— Disabled table due to space violation doesn’t get re-enabled

Quota
• Limit on region count can lead to large regions
• HBASE-20705
— Having RPC Quota on a table prevents Space quota to be recreated/removed
• Setting HDFS quota on a namespace can bring down RS

Summary

HBase Multi-Tenancy Story is Good
HBase Feature Multi-Tenant Cluster NS* Table User
Namespace + ACL Logical grouping of tenants X X X
Regionserver Groups Allocate CPU/Memory/Network per tenant X X
Space Quota Guarantee storage space per tenant X X
Request Throttling Control CPU/Network use X X X
Meta-Data Quota Control number of objects created X
* NS - Namespace

Improvements
• HBASE-15531
• SYSTEM table availability
• Memory Quota

Acknowledgement
• HBASE-6721 – Region server groups
• HBASE-16961 – Storage quota
• HBASE-11598 – Network/CPU Quota
• HBASE-8410 – Meta-Data Quota
• HBASE-8015 – Namespace and ACL
• HBASE-11355 – Request Queue

Thank You!
Reference: http://hbase.apache.org
Connect with Hadoop Team: hadoop@bloomberg.net

We are hiring!
Questions?
https://www.bloomberg.com/careers

Multi-Tenant HBase Cluster - HBaseCon2018-final

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Multi-Tenant HBase Cluster - HBaseCon2018-final

Similar to Multi-Tenant HBase Cluster - HBaseCon2018-final (20)

More from Biju Nair

More from Biju Nair (15)

Recently uploaded

Recently uploaded (20)

Multi-Tenant HBase Cluster - HBaseCon2018-final