An update on the State of the Database, for Apache Big Data EU 2015.

1. State
of
the
Database
@HBase
h)p://hbase.apache.org
2015-­‐09-­‐28
Nick
Dimiduk
(@xefyr)
h)p://n10k.com
#apachebigdata

2. Agenda
o State
of
the
Project
o State
of
the
SoMware
o State
of
the
Ecosystem
o Latest
Releases
o Bonus
Content!
o Q
&
A

3. STATE
OF
THE
PROJECT
Who
we
are,
what
we
do,
why
we
do
it

4. Project:
Vision
Simple,
steady,
and
powerful:
“A
first
class
high
performance
horizontally
scalable
data
storage
engine
for
Big
Data,
suitable
as
the
store
of
record
for
mission
cri]cal
data.”

5. Project:
Usage
o Data
access
for
medium-­‐
and
high-­‐scale
services
o Hundreds
of
enterprises
and
startups
o Some
of
the
largest
Internet
companies
in
the
world
o Running
major
produc]on
workloads
since
2011
o Use-­‐cases
o messaging,
security,
measurement/“IoT”,
collabora]on,
digital
media,
digital
adver]sing,
telecommunica]ons,
computa]onal
biology,
clinical
informa]cs/healthcare,
insurance

7. Project:
Goals
o Availability:
Always
more,
always
faster
o Stability
and
operability
o Scaling
up,
scaling
down
o Up-­‐to-­‐date
with
“commodity”
hardware
o Mul]-­‐tenancy
o Diversity
of
ecosystem

9. STATE
OF
THE
SOFTWARE
Regarding
the
codebase

10. State
of
the
SoMware
o Mature
codebase
o 100+
contributors
(40+
commi)ers)
o 1.1M
lines
of
code
(each
ac]ve
branch)
o est.
1200+
human-­‐years’
effort
o Clusters
sizes
from
10
to
1000+
machines
o that
we
know
of!
o Runs
on
HDFS,
MapR,
Gluster,
GPFS,
Lustre
o HBase
as
a
Service
o AWS/EMR,
HDInsight,
Qubole,
Google
(sort-­‐of)

11. SoMware:
Releases

12. SoMware:
Seman]c
Versioning
MAJOR-­‐MINOR-­‐PATCH[-­‐iden]fier]
o Client/Server
wire
compa]bility
o Server/Server
feature
compa]bility
o API
compliance
guarantees
o ABI
compliance
guarantees
h)p://hbase.apache.org/book.html#hbase.versioning

13. SoMware:
Ac]ve
Development
o Smaller
regions,
more
regions
o Less
write
amplifica]on
o 1M+
region
clusters
o Stability
o ProcedureV2
o Assignment
improvements/stability
o Backup,
restore
tools
o Built
on
snapshots,
easier
opera]ons

14. SoMware:
Ac]ve
Development
o Adap]on:
Workloads
o HBase
as
Medium
Object
Store
(MOB)
o Tunable
Availability
o Region
replicas
o TIMELINE
consistency
o Coprocessor
API
stability
o Less
GC,
more
RAM
(off-­‐heap)

15. SoMware:
Ac]ve
Development
o Mul]-­‐tenancy
o Table
groups
o Quotas
o Priori]es
o Improved
machine
u]liza]on
o More
RAM
(100’s
of
GB)
o IOPS
o Be)er
concurrency

16. STATE
OF
THE
ECOSYSTEM
The
whole
enchilada

17. State
of
the
Ecosystem
o OpenTSDB
o Transac]on
Managers
o Themis,
Tephera,
Omid2,
LeanXcale
o Graph
engines
o Titan,
Giraph,
Zen,
S2Graph
o Myriad
SQL’s
o Other
Hadoop
components
o Google
Cloud
Bigtable

18. Ecosystem:
SQL

19. Ecosystem:
Hadoop
Components
o YARN-­‐2928
Applica]on
Timeline
Service
o HIVE-­‐9452
HBase
to
store
Hive
metadata
o AMBARI-­‐5707
Ambari
Metrics
System

20. LATEST
RELEASES
Come
and
get
it!

21. Release:
0.94
o Last
(final?)
release:
0.94.27,
2015-­‐03-­‐26
o “ancient
history”
o No
new
deployments
o Exis]ng
users
highly
encouraged
to
upgrade
o Requires
down]me
to
upgrade
😫
😡
(╯°□°)╯︵
┻━┻

22. Release:
0.98
o Last
release:
0.98.14,
2015-­‐08-­‐31
o “legacy”
o Most
produc]on
deploys
(probably)
o Largest
produc]on
clusters
(probably)
o New
features
back-­‐ported
when
possible

23. Release
1.x
o Last
release:
1.1.2,
2015-­‐09-­‐01
o “stable”
o Produc]on
deploys
moving
here
o Ac]ve
development
o Rolling
upgrade
from
0.98.x
😄
😍
ヽ(´ー｀)ﾉ

24. Release
1.0
o Released
1.0.0,
2015-­‐02-­‐24
o Adop]ng
seman]c
versioning
o Patch
releases
don’t
quite
follow
spec
yet
o Client
/
Server
API
cleanup
o Interfaces,
builder
pa)ern,
@InterfaceAudience
o Region
Replicas
o Trade
Consistency,
resources
for
Availability
github.com/ndimiduk/hbase-­‐1.0-­‐api-­‐examples

25. Region
Replicas
o Mul]ple
Region
Servers
host
each
region
o Primary
+
N
read
replicas
(usually
N=2)
o Primary
is
authority
on
reads
and
writes
o Replicas
tail
replicate
edits,
offer
TIMELINE
view
o Client’s
choice
o Read
primary
only
for
“classic”
strong
consistency
o Fan-­‐out
reads
for
faster,
poten]ally
TIMELINE
results

26. Release
1.1
o Release
1.1.0,
2015-­‐05-­‐15
o Async
RPC
client
o Scanner
improvements
o RPC
chunking,
heartbeat
messages,
API
o RPC
thro)ling
o quotas
for
per
user,
table,
namespace
o Compac]on
thro)ling,
monitoring
o ProcedureV2
o Improved
opera]onal
reliability

27. ProcedureV2
o Distributed,
fault-­‐tolerant
opera]ons
o Mul]ple
steps
on
mul]ple
machines
o Roll-­‐back
in
case
of
failure
o Coordina]on
of
long-­‐running
procedures
o Compac]ons,
splits,
&c.
o Progress
tracking
o No]fica]ons
across
mul]ple
machines
o Current
status
inquiries

28. Branch-­‐1.2
o Next
up
in
1.x
line
o Java
8
support
o Na]ve
checksums
o SyncTable
o Flush-­‐per-­‐store
o ProcV2
all
the
things!
o (More)
Compac]on
improvements
o Region
normalizer

29. Region
Normalizer
o An]-­‐entropy
for
region
size
o Converge
towards
uniform
size
o Compliments
balancer
working
toward
uniform
distribu]on
o Managed
by
Master,
runs
in
the
background
(like
balancer)
o Pluggable
normaliza]on
strategies
(“simple”
default)
o Use-­‐cases
o Merge
away
regions
from
expired
]meseries
data
o Smooth
uneven
bulk
loads
o Correct
operator
ini]al
split
guesses
o Ease
upgrades
from
ancient
versions
(0.92/1g
vs.
today/20g)

30. Thanks!
@HBase
h)p://hbase.apache.org
2015-­‐09-­‐28
Nick
Dimiduk
(@xefyr)
h)p://n10k.com
#apachebigdata

31. BONUS
CONTENT!
Ask
and
you
shall
receive

32. Agenda
o Replica]on
o Filters
o Coprocessors

33. Replica]on
o Keep
data
synchronized
between
clusters
o Supports
mulOple
desOnaOons
o Cyclical
graphs
supported
o Configurable
at
Column
Family
granularity
o Uses
WAL
shipping
to
propagate
data
o Replica]on
state,
status
stored
in
ZooKeeper
o General
purpose
interface
for
asynchronously
shipping
edits
from
a
cluster
o Other
HBase
clusters,
Region
Replicas,
SOLR/Elas]cSearch
hbase.apache.org/book.html#_cluster_replica]on

35. Filters
o Addi]onal
applied
to
reads
o Use
in
conjunc]on
with
specifying
start,
end
rows,
&c.
o Run
on
the
Region
Servers
o Included
in
GET,
SCAN
request
o Explicitly
exclude
data
based
on
criteria
o I.E.,
value
>=
10
o Implicitly
exclude
data
by
hin]ng
seeks
o INCLUDE_AND_NEXT_COL,
NEXT_ROW,
SEEK_NEXT_USING_HINT
o Operate
on
data
read
from
BlockCache

36. Filters
o 30+
Filters
included
in
distribu]on
o Mini-­‐language
for
use
in
ThriM,
REST
o "(PrefixFilter ('row2') AND (QualifierFilter (>=,
'binary:xyz'))) AND (TimestampsFilter ( 123, 456))"
o hbase.apache.org/book.html#thriM.filter_language
o Simple
interface,
Implement
your
own!

37. public class PageFilter extends FilterBase {
public PageFilter(long pageSize) {…}
public boolean filterRowKey(Cell c) {
return false;
}
public ReturnCode filterKeyValue(Cell c) {
return ReturnCode.INCLUDE;
}
public boolean filterAllRemaining() {
return this.rowsAccepted >= this.pageSize;
}
public filterRow() {
this.rowsAccepted++;
return this.rowsAccepted > this.pageSize;
}
}

38. Coprocessors
o Extension
points
for
HBase
o Think
Linux
Kernel
Module,
not
Stored
Procedure
o I.E.,
customize
compac]ons,
Table
constraints
o Observers
o pre-­‐
and
post-­‐execu]on
logic
o I.E.,
MasterObserver#preTruncateTable,
RegionObserver#postScannerNext
o Endpoints
o Cluster
RPC
extensions
o I.E.,
RowCountEndpoint,
BulkDeleteEndpoint

39. public class RowCountEndpoint implements
ExampleProtos.RowCountService {
public void getRowCount(…) {
Scan = new Scan();
InternalScanner scanner =
env.getRegion().getScanner(scan);
…
do {
count++;
} while (scanner.next());
// return count
}
}

40. Thanks!
@HBase
h)p://hbase.apache.org
2015-­‐09-­‐28
Nick
Dimiduk
(@xefyr)
h)p://n10k.com
#apachebigdata