Video and slides synchronized, mp3 and slide download available at URL http://bit.ly/149ghtf. This talk goes over the design motivation for Zen and describe its internals including the API, type system and HBase backend. Filmed at qconsf.com. Xun Liu is a software engineer in the infrastructure team at Pinterest. He worked in many areas and is currently focusing on storage and caching solutions at Pinterest. Raghavendra Prabhu is engineering manager for the infrastructure team at Pinterest, which is responsible for core backend infrastructure including storage systems, caching, service framework and core business logic.

2. InfoQ.com: News & Community Site
• 750,000 unique visitors/month
• Published in 4 languages (English, Chinese, Japanese and Brazilian
Portuguese)
• Post content from our QCon conferences
• News 15-20 / week
• Articles 3-4 / week
• Presentations (videos) 12-15 / week
• Interviews 2-3 / week
• Books 1 / month
Watch the video with slide
synchronization on InfoQ.com!
http://www.infoq.com/presentations
/zen-pinterest-graph-storage-service

3. Purpose of QCon
- to empower software development by facilitating the spread of
knowledge and innovation
Strategy
- practitioner-driven conference designed for YOU: influencers of
change and innovation in your teams
- speakers and topics driving the evolution and innovation
- connecting and catalyzing the influencers and innovators
Highlights
- attended by more than 12,000 delegates since 2007
- held in 9 cities worldwide
Presented at QCon San Francisco
www.qconsf.com

4. Raghavendra Prabhu (RVP)
Zen:
Pinterest’s Graph Storage Service

5. “Given how robust the messenger is on
day one, it’s surprising to learn that
Pinterest built the entire product
in three months.” — The Verge

7. What does it take to do this consistently?
Many different things
•Hire the best
•Culture
•Focus
Infrastructure that doesn’t get in the way

8. Challenge for Infrastructure

9. Our approach
•Make it part of the team mission statement
•Design systems with ‘move fast’ in mind
•Separation of concerns: feature vs reliability

10. Persistent Storage
Even with a distributed database, app needs to deal with:
•Schema design
•Fault tolerance
•Capacity management
•Performance tuning

11. Solution 1: UserMetaStore
Storage-as-a-Service: Key-value thrift API on top of HBase
Features:
•Key partitioning to balance load
•Master-slave clusters, semi automatic failover
•Speculative execution
•Multi-tenancy with traffic isolation

12. Storage-as-a-service is a great step forward,
but can we do better?

13. Example: Messages Data Model
Conversation
Message 1
Message 2
Message N
User
User
Participates Contains

14. Realization
•These object models closely resemble a graph
•Objects are nodes, edges represent relationships
•Typical needs:
• retrieve data for a node or edge
• get all outgoing edges from a node
• get all incoming edges from a node
• count incoming or outgoing edges for a node

15. Enter Zen!
•Provides a graph data model instead of key-value
•Automatically creates necessary indexes
•Materializes counts for efficient querying
•Implemented on top of HBase, but can plug in other backends

16. Why the name Zen?
•Data model inspired by Facebook’s TAO
•But internally a very different system
•Zen:
• “evolution of Buddhism under Taoist conditions”
• “simplified version of Taoism”
• basically Pinterest’s take on the TAO idea :)

17. What Zen is NOT
•NOT a full fledged graph database
•NO advanced graph operations
•Basically an object-relationship data model on top of existing
databases to simplify app development

18. Zen API
Nodes:
• addNode, removeNode, getNode
• Node id: globally unique 64-bit integer
ID 123
Prop 1 Val 1
Prop 2 Val 2

19. Zen API
Edges:
• addEdge, removeEdge, getEdge
• Edge Ref: (edgeType, fromId, toId)
• Score for ordering
Edge Ref 120, 123, 4567
Prop 1 Val 1
Prop 2 Val 2

20. Zen API
Edge Queries:
• getEdges, countEdges, removeEdges
struct EdgeQuery {
1: required NodeId nodeId;
2: required EdgeDirection direction;
3: optional TypeId edgeType;
}

21. Zen API
Property Indexes
•Unique index
•Ensures a property value is unique across all nodes of a type
•Non-unique index
•Allows retrieval by property value
•Works for both nodes and edges

22. Zen API
Type System
•Declare node and edge types
•Specify type schema, e.g. unique and non-unique index properties
•Fully online: no deploy, no config
•Internally implemented on top of Zen itself!

23. Illustration: Messages on Zen
Id:1234 Id:2345 Id:3456
Type: Participates Type: Contains
Type: Conversation
Started: 12 Aug 2014 08:00
Header: “Great pin!”
Pin Id: 10001 [non-unique]
Type: User
Name: “Ben Smith” [unique]
Status: Active
Type: Message
Sent: 12 Aug 2014 08:00
Text: “Great pin!”

24. Zen: Current Usage
Products:
• smart feed, messages, network news, interest graph and other
upcoming features
Numbers:
• ~10 clusters
• 100,000+ requests per second at peak
• Over 5 million HBase operations per second

25. Xun Liu
Internals and Production Learnings

26. Zen Backends
•HBase backend implemented in fall 2013
•Currently working on MySQL backend
•Other potential backends in future

27. HBase Data Model Overview
Data

28. HBase Data Model Overview
Data
col1 col2
row-key-1 val1 val2

29. HBase Data Model Overview
Data
col1 col2 col3
row-key-1 val1 val2
row-key-2 val3 val4

30. HBase Data Model Overview
Data
col1 col2 col3 col4
row-key-1 val1 val2
row-key-2 val3 val4
row-key-3 val5

31. Zen - Property
Data
type name score distance
12345 (node) 10 Ben Smith
12345-20-67890 (edge) 1000 1 mile

32. Zen - Property Index
Data
ID
<hash>-unique-10-name=ben smith 12345
<hash>-nonuniq-10-lastname=smith-12345
<hash>-nonuniq-10-lastname=smith-67890

33. Zen - Edge Score Index
Data
12345-out-20-1000-67890
12345-out-20-1001-67891
12345-in-30-990-67892
12345-in-30-991-67893

34. Zen - Edge Count
Data
Count
12345-out-20 2
12345-in-30 4

35. Status - Soft Delete
New Features

36. Built-in Cache
New Features
Zen
Cache
HBaseClient Zen HBaseClient
Cache
Before After

37. Namespace
New Features
Node
Namespace 1
Edge Index Node
Namespace 2
Edge Index

38. New Features
•Online type schema change
•Optional reverse edge
•Optional edge count
•Retrieval of subset of properties
•Descending edge score

39. Performance Work
Demanding work load needs special tuning
• Inserting 1 million edges per second
• Excessive HLog (WAL) flushes

40. Performance Work
Batching
• Client Side Batching — bulk edge insertion
• Zen Server Side Batching — buffer edits across clients & flush
together
• Reduced HLog (WAL) flushes by orders of magnitude

41. Performance Work
Memory v.s. Performance
• Bloom filter
• reduce disk seeks
• memory cost: 1 byte per row
• Block size
• the smaller block size the better random access performance
• memory cost: bigger index size

42. Performance Work
CPU v.s. Data Size
• Encoding
• FAST_DIFF: effective in reducing data size, cpu intensive
• PREFIX: less effective in size reduction, less cpu intensive
• Compression
• SNAPPY, LZO, GZ, etc

43. Performance Work
Capability to tune storage engine per special load
Zen production setup
• Dedicated Zen cluster
• Namespace in shared Zen cluster

44. Data Consistency
Add an edge
1. CAS create the edge row and properties
2. CAS create the unique index if any
3. Create non-unique index if any
4. Create edge score index for outgoing direction
5. Create edge score index for incoming direction
6. Increment edge count for outgoing direction
7. Increment edge count for incoming direction

45. Distributed transaction or not?

46. Data Consistency
Stay on top of data inconsistencies
• Manual rollback in Zen server
• Offline jobs (Dr Zen) to scan and fix inconsistencies
• Tools to debug and fix one-off inconsistency

47. Future Work
•Dr Zen (make it more efficient)
•Other backends: MySQL, etc
•Distributed transactions
•Open source!

49. Watch the video with slide synchronization on
InfoQ.com!
http://www.infoq.com/presentations/zen-
pinterest-graph-storage-service