Introduction to Aerospike

ROCKET ENGINE
FOR CONTEXT DRIVEN APPS THAT

PERSONALIZE THE INTERNET
BY YOUNG PAIK
DIRECTOR SALES ENGINEERING,
AEROSPIKE
Aerospike aer . o . spike [air-oh- spahyk]
noun, 1. tip of a rocket that enhances speed and stability
© 2014 Aerospike. All rights reserved. Confidential

1

Aerospike NoSQL Database


2

AGENDA
1. How the game has changed,
driving need for next-gen NoSQL
2. Who uses Aerospike and why
3. Architecture Overview


3

Internet Enterprises have changed the game…
Simple, Personalized, Instant

Complex, Standardized, Silo-ed


4

Consumers Expect and “Want it All”
1. Instant Response
■“Every 100ms latency costs
Amazon 1% in sales”
– Greg Linden, Amazon
■“An extra ½ sec in search page
generation dropped traffic 20%”
– Google (average 1.5 sec)
■“A 1 sec delay can cause 7%
decline in conversion”
– Walmart


5

1. Instant Response
2. Intuitive Service
■

Personalized & Consistent
across channels
■
■

Mobile devices, tablet, car…
Web, mobile, social media…


6

1. Instant Response
■

Personalized & Consistent
across channels
■
■

■

Mobile devices, tablet, car…
Web, mobile, social media…

Seamless across the business
■

Marketing, sales, support…


7

1. Instant Response


1. Always-On
■

How much does down-time cost?


8

Enterprises must “Deliver it All”
■ Use every swipe, search, share to delight
- Instantly, Intuitively, Always-On
■ (DIY or SaaS)

CONTEXT
■ IDENTITY
■ SessionIDs, Cookies, DeviceIDs, ip-Addr

■ ATTRIBUTES
■ Demographic, geographic

■ BEHAVIOR (REAL-TIME)
■ Presence, swipe, search, share..
■ Channels – web, phone, in-store..
■ Services – frequency, sophistication

■ SEGMENTS (PRE-CALCULATED)
■ Attitudes, values, lifestyle, history..

■ TRANSACTIONS
■ Payments, campaigns


9

How Big is Real-time Big Context?
# People

Per Profile

10 M Customers *

25 kb

250 GB

500 M Prospects *

1 kb

+ 500 GB

Real-time
Context

= 750

CONTEXT

GB

■ IDENTITY
■ SessionIDs, Cookies, DeviceIDs, ip-Addr

■ ATTRIBUTES

■ How many Objects?

■ Demographic, geographic

■# People * # Devices * # Browsers
■People move around, cookies get
cleared..
■* 2x Replication

■ “100M people ≈ 2 Billion cookies”
- eBay

■ BEHAVIOR (REAL-TIME)
■ Presence, swipe, search, share..
■ Channels – web, phone, in-store..
■ Services – frequency, sophistication

■ SEGMENTS (PRE-CALCULATED)
■ Attitudes, values, lifestyle, history..

■ TRANSACTIONS
■ Payments, campaigns


10

Aerospike Database – Powering Context-Driven
Apps
■ Apps that Personalize the Internet
Instantly, Intuitively & Always-On

1. ROCKET ENGINE
- In-Memory,
Flash-optimized
2. WEB SCALE
– Distributed,
Shared nothing
3. ACID RELIABILITY
– Immediate Consistency,
High Availability
4. NoSQL FLEXIBILITY
– Distributed Queries,
Real-time Analytics


11

New Architecture for Context Computing
APP SERVERS

RDBMS

DATA
WAREHOUSE
AEROSPIKE
CLUSTER
MILLIONS OF CONSUMERS
BILLIONS OF DEVICES

SEGMENTS
R/W
REAL-TIME
CONTEXT

CONTEXT
IDENTITY
Cookies, device
ID..
ATTRIBUTES
Age, Gender..

+

REAL-TIME
ANALYTICS

Calculate models,
Discover
SEGMENTS eg
early adopter,
bargain hunter,
mass affluent…

BATCH
ANALYTICS

QUERIES &
AGGREGATIONS
Risk scores,
best sellers,
trending now…

BEHAVIOR
Click, search…
SEGMENTS

12

Pioneered by Ad-Tech
AppNexus

[x+1]

Federated
Media


eXelate

13

Powering the profile store for AppNexus (RTB)
■ “For the last three years, Aerospike’s
database has been managing our vast
volumes of user data. With Aerospike, we
process many terabytes of data daily
across our global data centers at a rate in
excess of a million requests per second.
– Mike Nolet, CTO
■ “AppNexus* operates at massive scale
while paying close attention to the
economics of the platform. Aerospike’s
flash optimizations running on top of
Intel® SSDs have given us the price,
performance, reliability, and serviceability
we need to grow our business.”
– Timothy G Smith,
SVP Technical Operations

•

50 Billion Ad +
300 Billion Bid Requests/day
for Microsoft Ad Exchange,
Interactive Media
(Deutsche Telekom),
Collective…

•

6 Billion Mobile Ads/day
for Millennial Media Exchange

•

100ms SLA from click to view

14

Powering the eXelate Data Exchange (DMP)
■ 200 publishers/ marketers access real-time context on
700Million Consumers
■ Demographics, purchase intent, behavioral propensities
from online /offline sources eg Nielsen, MasterCard Advisors,
Bizo

■ Found SQL DBs an order of magnitude too expensive,
considered several NoSQL DBs
■ 200 servers ingest 2 TB clickstream data per day
to Aerospike and an analytics DWH
■ Models calculated in DWH, loaded into Aerospike:
■ 20 TB data, 60 Billion transactions per month
■ 50/50 balanced reads/writes
■ 12 node clusters, 7 SSDs/128GB DRAM per node
■ Data synchronized across clusters in 4 data centers

■ Aerospike delivered on all of these requirements.”
– Elad Efraim, CTO of eXelate

15

Powering [x+1] Origin Digital Marketing Hub
(DMP + DSP)
■ Marketing Hub

■ Multi-channel analytics and personalization of messages
across touchpoints
■ Integrated with leading CRM platforms
■ Deployed at many Fortune 500 companies

■ “It was a challenge to find an extremely highperformance, high availability database…
■ 4 TB of data, 2 Billion profiles
■ 5,000-10,000 attributes per profile analyzed in 4ms
■ 10 relevant recommendations suggested in 50ms
- each time a visitor clicks on a website

…providing fast reliable access to data in real-time
is simple to say, but it’s not easy to do….
Aerospike has proven that our choice to buy, not build was
the right decision.” – Patrick DeAngelis, CTO, [x+1]

16

Internet-scale Context Computing Platforms
EMAIL
RETAIL
E-COMMERCE

SEARCH

WEB

OMNI
CHANNEL

GAMING

VIDEO
MOBILE

SOCIAL


17

Context driven Apps - Use Cases
ADVERTISING
• REAL-TIME BIDDING
• DEMAND SIDE PLATFORM
(DSP)
• DATA MGMT PLATFORM
(DMP)
• SUPPLY SIDE PLATFORM
(SSP)
MARKETING
• MULTI-SCREEN OFFERS
• MULTI-CHANNEL
PERSONALIZATION
• REAL-TIME
RECOMMENDATIONS
• ONE TIME COUPONS
• LOYALTY REWARDS
• DEALS NEAR YOU
• RELATED ITEMS
SALES
• PRODUCT AVAILABILITY
• DYNAMIC PRICING
• RISK SCORES
• FRAUD PREVENTION
• STREAM ANALYSIS


SUPPORT
• REAL-TIME DASHBOARDS
• PERSONAL FINANCE
PORTFOLIOS

18

Next Gen NoSQL


19

Aerospike Database / Context Computing
Platform
■ Powering Context driven Apps that
Personalize the Internet
Instantly, Intuitively & Always-On!

1. ROCKET ENGINE
- In-Memory,
Flash-optimized
2. WEB SCALE
– Distributed,
Shared nothing
3. ACID RELIABILITY
– Immediate
Consistency, High Availability
4. NoSQL EXTENSIBILITY
Real-time Analytics


20

ROCKET ENGINE

Ask me. I’ll look up the answer and then tell it
to you.

• Indexes in DRAM
• Data in DRAM / SSD
• Balanced Reads &
Writes
• Highly Parallelized
Ask and I’ll tell you now.
• Lock-free + ACID

OTHER DATABASE

AEROSPIKE

OS FILE SYSTEM

HYBRID MEMORY SYSTEM™

PAGE CACHE

BLOCK INTERFACE OPEN NVM

BLOCK INTERFACE
HDD

SSD

SSD

OTHER
DATABASE

SSD

SSD

FLASH OPTIMIZED
IN-MEMORY DATABASE


21

Aerospike Certification Tool (ACT) for SSDs
■ Industry Standard Flash (SSD / PCI-E) Benchmark
■ Open Source Tool used by Flash Vendors to certify drives


22

10X Faster for Balanced Read/Write loads
HIGH THROUGHPUT

LOW LATENCY

350,000

Balanced Workload Read Latency
Average Latency, ms

300,000

250,000
200,000
150,000

10
Aerospike

7.5

Cassandra

5

MongoDB

2.5
0
0

100,000

Throughput, ops/sec

50,000

Balanced
Cassandra

Read-Heavy
MongoDB

Couchbase 2.0*

*We were forced to exclude Couchbase...since when run with either disk
or replica durability on it was unable to complete the test.”
– Thumbtack Technology

Average Latency, ms

Balanced Workload Update Latency

0

Aerospike

50,000 100,000 150,000 200,000

16
Aerospike

12

Cassandra

8

MongoDB

4
0
0

4 Node Cluster, each with:

50,000 100,000 150,000 200,000
Throughput, ops/sec

CPU: 8 x Intel(R) Xeon(R) CPU E5-2665 0 @
2.40GHz, RAM: 31 GB
SSD: 4 x INTEL SSDSA2CW120G3, 120 GB (94 GB overprovisioned)
HDD: ST500NM0011, 500 GB, SATA III, 7200 RPM
OS: Ubuntu Server 12.04.1 64-bit (Linux kernel v.3.2.0)

23

ONLY
186 SERVERS REQUIRED

14 SERVERS

Actual customer analysis
99% < 1ms
500K TPS
10 TB Storage
2x Replication
OTHER DATABASES
DRAM & HDD

SSD & DRAM

Storage /server
TPS /server
Cost /server
Server costs
Power /server
Power (2 years) $0.12 per kWh ave.
US
Maintenance (2 years) $3,600
/server

180 GB (196 GB)
500,000
$8,000
$1,488,000
0.9 kW

2.4 TB (4 x 700 GB)
500,000
$11,000
$154,000
1.1 kW

$352,000

$32,400

$670,000

$50,400

Total

$2,510,000

$236,800


24

WEB SCALE
■ Distributed Hash Table with No Hotspots
■ Every key hashed with RIPEMD160
into a 20 byte (fixed length) string

cookie-abcdefg-12345678

■ Hash + additional (fixed 64 bytes) data
stored in DRAM in the index
■ Some bits from hash value are used to
calculate the Partition ID (4096 partitions)
■ Partition ID maps to Node ID in the cluster

182023kh15hh3kahdjsh

■ Shared Nothing architecture

…

1

4

2

3

3

2

4096

■ No Load Balancers required

Replica
Node ID

1821

■ Client just calculates Partition ID, determines Node ID

Master
Node ID

1820

■ 1 Hop to data

Partition ID

4

1

■ Every node is identical


25

WEB SCALE with ACID RELIABILITY

OHI
O

1)

No Hotspots
– DHT with RIPEMD160
simplifies data
partitioning

2)

Shared Nothing
Architecture,
every node identical

Single row ACID
– synch replication in
cluster

5)

Smart Cluster, Zero
Touch
– auto-failover,
rebalancing, rolling
upgrades..

Smart Client – 1 hop to
data, no load balancers

3)

4)

6)

Transactions and long
running tasks prioritized
© 2014 Aerospike.
real-time All rights reserved. Confidential

7) XDR – asynch replication
across data centers ensures
Zero Downtime

26

XDR ensures Zero Downtime
■ Cross Data Center Replication (XDR) enables geographic
redundancy and location proximity
■ Maximum flexibility
■ Replication set at namespace level
■ Active-Passive /Active-Active modes
■ Changes in one data center can be
■ replicated to multiple data centers
■ forwarded to another data center

■ Clusters can have different number of nodes
■ Automatic failure handling ensures continuity
in spite of node failures

■ Super Storm Sandy 2012
■ Power outage, NYC Cluster down for 17 hours
■ Once power returned, XDR synched in 1 hour

“Aerospike allows us to handle business
continuity and reliability across 4 data centers
seamlessly.”
- Elad Efraim, CTO

27

APP SERVER

Architected for Context Computing

APPLICATION

APP/WEB
SERVER

AEROSPIKE SMART CLIENT™
• APIs (C, C#, Java, PHP, Python, Ruby, Erlang…)
• Transactions, Cluster awareness

EXTENSIBLE DATA MODEL

AEROSPIKE SERVER

Patents pendin
Written in „C‟

• Str, Int, Lists, Maps
• Lookups, Queries, Scans

• User Defined Functions
• Distributed Aggregations

4) NOSQL FLEXIBILITY

MONITORING &
MANAGEMENT
• Aerospike Monitoring
Console™
• Command Line Tools

AEROSPIKE
SMART CLUSTER™

2) WEB SCALE
AEROSPIKE
HYBRID
MEMORY SYSTEM™

AEROSPIKE (XDR)
CROSS DATA CENTER
REPLICATION™

3) ACID RELIABILITY

AEROSPIKE
CLUSTER

• PluginsNaglos, Graphite, Zabbi
x

AEROSPIKE
REAL-TIME
ENGINE™

1) ROCKET FAST

28

NOSQL EXTENSIBILITY
■ Namespaces (policy containers)
■ Determine storage - DRAM or Flash
■ Determine replication factor
■ Contain records and sets

■ Sets (tables) of records
■ Arbitrary grouping

■ Records (rows)
■ Max 128k, contain key and bins
■ Bin with same name can contain values of
different types
■ String, integer, bytes (raw, blob, etc)
■ list ( an ordered collection of values )
■ map ( a collection of keys and values )

■ Bins can be added anytime


29

Real-time Analytics on Operational Data
DISTRIBUTED QUERIES
1. “Scatter” requests to all nodes
2. Indexes in DRAM for fast map of secondary  primary keys
3. Indexes co-located with data to guarantee ACID,
manage migrations
4. Records read in parallel from all SSDs
using lock free concurrency control
5. Aggregate results on each node

6. “Gather” results from all nodes on client
STREAM AGGREGATIONS
1. Push Code/ Security Policies/ Rules to Data with UDFs
2. Pipe Query results through UDFs to
Filter, Transform, Aggregate.. Map, Reduce
REAL-TIME ANALYTICS on OPERATIONAL DATA (No ETL)
■ In Database, within the same Cluster
■ On the same Data, on XDR Replicated Clusters


30

Aerospike Database / Context Computing
Platform
■ Powering Context driven Apps
that Personalize the Internet
Instantly, Intuitively & Always-On!

1. ROCKET ENGINE
- In-Memory,
Flash-optimized
2. WEB SCALE
– Distributed,
Shared nothing
3. ACID RELIABILITY
– Immediate
Consistency, High Availability
4. NoSQL EXTENSIBILITY
Real-time Analytics


31

Recognized as the only Visionary in Gartner's Magic
Quadrant for Operational Database Management
Systems

Gartner, Magic Quadrant for Operational Database
Management Systems Donald Fienberg et al. October 23, 2013
This graphic was published by Gartner, Inc. as part of a larger
research document and should be evaluated in the context of the
entire document. The Gartner document is available at
www.aerospike.com .
Gartner does not endorse any vendor, product or service depicted in
its research publications, and does not advise technology users to
select only those vendors with the highest ratings. Gartner research
publications consist of the opinions of Gartner's research organization
and should not be construed as statements of fact. Gartner disclaims
all warranties, expressed or implied, with respect to this
research, including any warranties of merchantability or fitness for a
particular purpose.


32

Introduction to Aerospike

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (16)

Similar to Introduction to Aerospike

Similar to Introduction to Aerospike (20)

More from Aerospike, Inc.

More from Aerospike, Inc. (14)

Recently uploaded

Recently uploaded (20)

Introduction to Aerospike

Editor's Notes