Given 2014-11 ApacheConEU

1. Lizard
Clustering an RDF Triplestore
Andy Seaborne
andy@a.o

2. Why Cluster?
➢ Service Resilience
○ Failures
○ Server admin and security patches
➢ Performance / scale
○ More hardware : CPU, RAM, system bus

3. Why Cluster?
➢ Alternatives
○ Full clustering
○ Master-slave (load scale only ; update issues)
○ Application visible partitioning

4. Who am I?
➢ Committer on Apache Jena
○ Deploy/operate Jena/TDB in ££job.
➢ W3C
○ Co-editor on SPARQ 1.0 and 1.1 query lang
spaces
○ RDF 1.1 (on syntax, inc. SPARQL alignment)
○ ASF’s W3C AC representative

5. Acknowledgements
➢ Apache
➢ Partial funding : InnovateUK*
➢ Users
○ For the discussion and encouragement
* Used to be the Technology Strategy Board.
UK Department for Business, Innovation & Skills

6. Outline
➢ TDB Design
➢ SPARQL Execution
➢ Lizard Design
➢ Back to SPARQL

7. TDB
➢ Custom RDF Database
○ Quads + Triples
➢ Custom index code
○ Threaded B+Trees
➢ Dictionary terms
○ NodeId – 8 bytes
○ Inline terms (numbers, dates, dateTimes)

8. TDB : Node Table

9. TDB : Indexes
➢ Indexes are covering
○ Range scans
○ All key, no value
○ No "triple table"

10. SPARQL Execution
{ ?x :p 123 . }
Convert to NodeIds
Look in POS to get all PO?, assign S to ?x
123 is an inline constant in TDB.
{ ?x :p 123 .
?x :q ?v . }
A database join
Index join (Loop+substitution)
Index join (= loop) on
:x1 :q ?v
where :x1 is the value of ?x

11. Index Implementation
➢ TDB uses threaded B+Trees for indexes
○ 8K blocks 100-way B+Tree
○ Threaded : scan only touches data blocks
SPO SPO SPO ------ ------ ------
Ptr Ptr ------ ------ ------
SPO SPO SPO SPO ------ ------
Ptr Ptr Ptr ------ ------
SPO SPO SPO SPO SPO SPO SPO SPO SPO SPO ------ ------

12. Choices
Query and Update
Indexes / B+Trees Node table / Objects
Blocks Key → Value Store
Where to introduce distribution?

13. This Does Not Work (very well)
Query and Update
B+Trees Objects
Blocks Key→Value
➢ Impedance mismatch
Distribute the storage
K->V store
Index access on query processor
○ Too much data moving about
○ Little parallelism
○ Bad cold-start

14. Distribute
Query and Update
B+Trees Objects
Blocks Key→Value
➢ Distribute the indexes
○ With modified index access
➢ Distribute the nodes
➢ Comms : Apache Thrift

15. Clustered Node Table
➢ Node Table
○ N replicas; Read R / Write W
e.g. W=N and R =1 =>
Complete copies of node table on each data server
○ Replaceable
Requirement: NodeId for naming

16. Clustered Indexes
➢ Indexes
○ Shard by subject
○ Replica shards
○ Compound access operations

17. Clustered Indexes
Index
Shard 1 Shard 2 Shard 3
Machine 1 Machine 2

18. Configuration 1

19. Configuration 2

20. Modified SPARQL Execution
➢ Different unit of index access
○ subject + several predicates
(subj, pred1, pred2, pred3, …)
➢ Different join algorithms
○ Merge join
○ Parallel hash join

21. Paul Hirst / CC-BY-SA-2.5