How Graph Databases efficiently store, manage and query connected data at scale

@mesirii
Challenging Relationships
How Graph Databases efficiently
store, manage and query
connected data at scale

@mesirii
Graph Databases try to make it easy for developers to leverage huge amounts
of connected information for everything from routing to recommendations.
Doing that poses a number of challenges on the implementation side. In this
talk we want to look at the different storage, query and consistency
approaches that are used behind the scenes. We’ll check out current and
future solutions used in Neo4j and other graph databases for addressing
global consistency, query and storage optimization, indexing and more and
see which papers and research database developers take inspirations from.

@mesirii
I know what
you're here for!
Covers made with: dev.to/rly
and from @ThePracticalDev

@mesirii
Michael Hunger
Open Sourcerer
Neo4j
@mesirii

@mesirii
Just too much to look at
but … GitHub
github.com/neo4j

@mesirii
We're hiring - Join Jarek!
Greek Blogger -> Eng Director

@mesirii
A short History of Neo4j

@mesirii
A short history
There was a CMS/DMS in Sweden
Which had two big issues
Language independent Keywords
Complex Access Control for SaaS
RDBMS failed
In Memory Graph was cool
Dot Com Bubble burst
A new star was born

@mesirii
The Relational Crossroads

@mesirii
Four NOSQL Categories
arising from the “relational crossroads”
Key Value
Column
Family
Document
Graph
Denormalise
Normalise

@mesirii
Four NOSQL Categories
arising from the “relational crossroads”
Denormalise
Normalise

@mesirii
Graph Approach
• natural model
• schema optional
• pre-compute & store connections
• local neighborhood queries
• graph based compute

@mesirii
Graph Database
Implementations

@mesirii
● ArangoDB
● DSE Graph
● Agens Graph
● IBM Graph
● JanusGraph
● Tibco GraphDB
Recent Newcomers
● Microsoft CosmosDB
● TigerGraph
● MemGraph
● AWS Neptune
● SAP HANA Graph
● Redis Graph

@mesirii
Implementation Designs
● Adjacency List
● Adjacency Matrix (compressed)
● Sparse Matrices
● Column Store
● HexaStore
● Hash Index
● Document Store
● Object Storage

@mesirii
● pre-materialize connections
● store "neighbours" with each node
● direct memory pointer
● cheap O(1) lookup, O(n) scan
● random memory access !
● Properties on Relationships
● Grouped by Type & Direction
● Neo4j
Adjacency List
Node Rel
Rel
Rel
Rel
Rel
Rel
Rel
Rel
Node
Node
Node
Node Rel
Rel
Rel
Rel
Node
Node

@mesirii
Adjacency Matrix
● matrix with nodes as
○ row and column
○ cell is relationship
○ can contain weight
○ 0 … no relationship
● matrix operations as
graph operations
● size is a problem (N^2)
● need to compress
● e.g. bitsets (SparkSee)

@mesirii
Sparse Matrix
● linear algebra
● GraphBLAS
○ research & development from Uni Texas
● efficient sparse matrices on CPU & GPU
● matrix operations (and filters) as graph
operations
● RedisGraph

@mesirii
Column Store
● sort by "natural ids"
● all properties and relationships as very wide columns
● need fixed schema

@mesirii
Hash / Hybrid Index
● Nodes and Relationships are documents
● Additional HashIndex(Source, Target) -> Linked List of Rels
● Used in ArangoDB

@mesirii
Document Store
● Store Target document identifiers in Array
● Simple Relationships without properties
● MongoDB $graphLookup operator
● Many index lookups
$graphLookup: {
from: "contacts",
startWith: "$friends",
connectToField: "name",
connectFromField: "friends”,
as: "socialNetwork"
},
{
_id: 0,
name: "Bob Smith",
friends: ["Anna Jones", "Chris Green"]
}, {
_id: 1,
name: "Anna Jones",
friends: ["Bob Smith", "Chris Green", "Joe Lee"]
}, {
_id: 2,
name: "Chris Green",
friends: ["Anna Jones", "Bob Smith"]
}

@mesirii
Hexa-Store
● Used by TripleStores
● Backed by Key-Value Store
● Store all combinations of triples
○ S-P-O
○ S-O-P
○ P-S-O
○ P-O-S
○ O-P-S
○ O-S-P
● And use prefix search for lookups/expand
● JS - GunDB, DGraph, Cayley

@mesirii
Native vs non-native DBs.

@mesirii
Native Database
• Each Database is native to it's core model
• eg. relational, column
• so optimized for that model in storage and
operations
• And non-native to other models that you put on top
• which causes lack of safety, performance,
expressiveness

The Doctor -
Jim Webber
Watch on YouTube
and more

@mesirii
A database is a database,
is a database.

@mesirii
MySQL Database Architecture

@mesirii
github.com/neo4j/neo4j

@mesirii
Neo4j
Database
Architecture

@mesirii
Beyond Database - Graph Platform

@mesirii
Architecture Components

@mesirii
File System
Record based files
Fixed Size Records
ID = Record ID
Offset = ID * Block Size
Pointer = Memory +
Offset
Nodes
Relationships
Properties

@mesirii
Page Cache
● OS Memory Mapping insufficient
● Which pages are important when (LRU-K)
● Transactional Guarantees / Isolation
● Concurrency
● Use for other types (indexes)
○ Generational Datastructures
● Seed cache

@mesirii
DB Engine
● Low Level Kernel SPI for common operations
● Only works with primitives / arrays
● Off Heap (tx, index, metadata, next: query state)
● Record Access
● Transaction Layer (Isolation)
● reusable Cursors (Prefetching)
● soon: Store Abstraction

@mesirii
Multi-*
● Multi-Graph (composability)
● Multi-Database (multi-tenancy)
● Multi-Cluster (geo-sharding)

@mesirii
Neo4j Type System (Cypher, Drivers, Browser)
Null
Missing or unknown
value
Boolean
True or false
Integer
64-bit signed integer
Float
Double precision
floating point
Spatial
different 2d and 3d
coordinate systems
Bytes
Raw octet stream
String
Unicode text
List
Ordered collection
Map
Keyed collection
Temporal
(local)date(time)
duration
Structure
Node Relationship Path

@mesirii
Why the hell - 4j?
Good
Founders were Java Developers
Easier to hire
Java has memory management (GC)
Java NIO
Portability
JVM got way faster/better of the years
Extensibility in all JVM Languages
Can utilize GraalVM
Bad
Little Access to low level system capabilities
(Cache, Memory, Network)
Need to use Unsafe
Garbage Collection (unpred. pauses)
No value types
Scala runtime behavior
C-Libraries are harder to integrate

@mesirii
(:You)-[:QUERY]->(:Data)
in a graph

@mesirii
Who bought "Chocolate" ?

@mesirii
SQL
SELECT distinct c.CompanyName
FROM customers AS c
JOIN orders AS o
ON (c.CustomerID = o.CustomerID)
JOIN order_details AS od
ON (o.OrderID = od.OrderID)
JOIN products AS p
ON (od.ProductID = p.ProductID)
WHERE p.ProductName = 'Chocolat'

@mesirii
Apache Tinkerpop 3.3.x - Gremlin
g = graph.traversal();
g.V().hasLabel('Product')
.has('productName','Chocolat')
.in('INCLUDES')
.in('ORDERED')
.values('companyName').dedup();

@mesirii
W3C SparQL
PREFIX sales_db: <http://sales.northwind.com/>
SELECT distinct ?company_name WHERE {
<sales_db:CompanyName> ?company_name .
?c <sales_db:ORDERED> ?o .
?o <sales_db:ITEMS> ?od .
?od <sales_db:INCLUDES> ?p .
?p <sales_db:ProductName> "Chocolat" .
}

@mesirii
Neo4j Cypher (openCypher)

@mesirii
(Cypher)-[:QUERIES]->(Graphs)

@mesirii
A (real) Question
Find all Actors and Movies they acted in
Whose name contains the letter "a"
Aggregate the frequency and movie titles
Filter by who acted in more than 5 movies
Return their name, birth year and movie titles
Ordered by number of movies
Limited to top 10

@mesirii
A (real) Cypher Query
MATCH (a:Person)-[:ACTED_IN]->(m:Movie)
WHERE a.name CONTAINS "a"
WITH a,
count(m) AS cnt,
collect(m) AS movies
WHERE cnt > 5
RETURN a.name, a.born,
[m IN movies | m.title] as titles
ORDER BY size(movies) DESC
LIMIT 10

@mesirii
Cypher Implementation

@mesirii
● cost based planner
○ e.g. index selectivity, db-statistics
● IDP (Iterative Dynamic Programming)
● Loads of papers on query plannig
Query Planning

@mesirii
Query Execution
Operators
(parallel)
Access Kernel SPI
Or compute
Expand vs. JOIN
Row
transformation

@mesirii
openCypher
● open-source query language spec
● implementers group
● publishes artifacts
● reference implementation
● open collaboration
● toward a new standard
○ fun with standards orgs

@mesirii
Cypher.next
● Return and construct graphs
● Composable Queries
● Regular Path Queries
● Configurable Pattern Matching
● Subqueries

@mesirii
Architecture & Data Flow
Application
Cypher Bolt Driver
Cypher Bolt Server
Neo4j
MATCH (a:Person)
WHERE a.name = 'Alice'
RETURN a.surname, a.age
{surname: 'Smith',
age: 33}
Parameterised
Cypher
Result
Stream
metadata

@mesirii
Driver Implementation
● Versioned Protocol (Handshake)
● Packstream Protocol based on MessagePack
● Asynchronous w/ sync APIs
● Uses Netty on Server
● Reactive w/ backpressure in v2 next year

@mesirii
Drivers (official)
Java
<dependency>
<groupId>org.neo4j.driver</groupId>
<artifactId>neo4j-java-driver</artifactId>
<version>X.Y.Z</version>
</dependency>
Python
pip install neo4j-driver
.NET
PM> Install-Package Neo4j.Driver
JavaScript
npm install neo4j-driver
Go
pip install neo4j-driver
C (Seabolt)
npm install neo4j-driver

@mesirii
Driver Concepts
Driver
Top-level object for all Neo4j interaction
Session
Logical context for sequence of transactions
Transaction
Unit of work
Statement Result
Stream of records plus metadata

@mesirii
Java (async)
String uri = "bolt://localhost:7687";
Driver driver = GraphDatabase.driver(uri, AuthTokens.basic("neo4j", "p4ssw0rd"));
Session session = driver.session();
session.readTransactionAsync(tx ->
tx.runAsync("MATCH (a:Country) RETURN a.name").thenCompose(cursor ->
cursor.forEachAsync(System.out::println)
)
).whenComplete((ignore, error) -> session.closeAsync());

@mesirii
Python (blocking)
uri = "bolt://localhost:7687"
driver = GraphDatabase.driver(uri, auth=("neo4j", "p4ssw0rd"))
def print_names(tx):
result = tx.run("MATCH (a:Person) RETURN a.name")
for record in result:
print(record["a.name"])
with driver.session() as session:
session.read_transaction(print_names)

@mesirii
Driver Implementation
● Versioned Protocol (Handshake)
● Packstream Protocol based on MessagePack
● Asynchronous w/ sync APIs
● Uses Netty on Server
● Reactive w/ backpressure in v2 next year
● Same architecture across languages

@mesirii
Transaction Routing
Connection
to reader
Session
Load Balancing Connection Pool
Connection
to writer
Connection
to reader
session.read_transaction(...) session.read_transaction(...)
session.write_transaction(...)driver.session() session.close()ACQUIRE
RELEASE
ACQUIRE
RELEASE
ACQUIRE
RELEASE

@mesirii
Driver Capabilities
● Pooling, Timeouts, Socket Options
● Transaction Management
● Cluster Routing, Smart Client, Bootstrap
● Routing Policies (e.g. geo or custom)
● Multi-Cluster
● Causal Consistency
● Transaction Chaining
● Transparent Retry
● Security

@mesirii
Connection Management
Driver
Connection
Pool
Server A
ConnectionConnectionConnection
ConnectionConnectionConnection
Session
SPAWNS
OWNS
OWNS
Server B
TO
TO
BORROWS

@mesirii
Server Selection Strategy
The Round Robin strategy (prior to 1.5)
continues to try all servers in turn,
leading to a severe backlog of work and
a dramatically lower overall throughput.
The Least Connected strategy
(introduced in 1.5) leads to only a
proportional drop in throughput under
the same circumstances, as the
misbehaving server is avoided.
one server starts to run slow

@mesirii
Clustering History
1. Zookeeper
2. Paxos (v1)
3. Paxos (v2)
4. Raft
"Raft is a consensus algorithm that is designed to be easy
to understand. It's equivalent to Paxos in fault-tolerance
and performance." raft.github.io

@mesirii
Causal Cluster
● Consensus via RAFT
● Fault Tolerant (CP)
● Causal Consistency with Bookmarks
● Quorum for Writes
● Otherwise Read-Only
● Rolling Upgrades
● Multi-Cluster

@mesirii
Causal Consistency with Bookmarks

@mesirii
Clustering (next)
● Analytics on Reporting Instances
● Cluster member integration with Spark
● Distributed linear Transactions
● Sharding
● Workload based sharding

@mesirii
Past Graph Compute Options
● Data Processing
○ Spark with GraphX, Flink with Gelly
○ Gremlin Graph Computer
● Dedicated Graph Processing
○ Urika, GraphLab, Giraph, Mosaic,
GPS, Signal-Collect, Gradoop
● Data Scientist Toolkit
○ igraph, NetworkX, Boost in Python, R, C

@mesirii
Pregel - Bulk Synchronous Parallel (BSP)
The map-reduce for graph compute.
Node-Centric Processing
1. Each node sends message
about it's own state
2. Each node receives messages
from neighbours
3. Updates it's own state
4. Global Compute Superstep

@mesirii
Neo4j Graph Algorithms
Turning Papers into Code

@mesirii
Actually,
There is a real book coming

@mesirii
How does it work?
Procedures
Neo4j
In Memory
Graph
Read projected
graph
Load projected
graph
Graph
Loader
Execute
algorithm
Store
results
1
2
4
3
Every operation is concurrent

@mesirii
How do you use it?
1. Call as Cypher procedure
2. Pass in specification (Label, Prop, Query) and configuration
3. ~.stream variant returns (a lot) of results
CALL algo.<name>.stream('Label','TYPE',{conf})
YIELD nodeId, score
4. non-stream variant writes results to graph; returns statistics
CALL algo.<name>('Label','TYPE',{conf})

@mesirii
Analytics / Machine Learning

@mesirii
Some "hot" topics
● Connected Feature Extraction
● Graph Embeddings
● Knowledge Graphs
● TextRank
● Translation Graphs
● Neural Network Understanding

@mesirii
Recent Deepmind Paper
https://github.com/deepmind/graph_nets

@mesirii
There is more to a
database than code

@mesirii
Making Developers Happy
● Neo4j Browser
● Neo4j Desktop
● Documentation
● Cypher-Refcard
● Online Training
● Discourse, Slack & StackOverflow
● Import / Export Tools
● Visualization
● Utility Libraries

@mesirii
No animals were harmed
in the production
of this talk.
https://www.oreilly.com/animals.csp

@mesirii
Find me for questions!
Thank you for your time!
Follow @mesirii

How Graph Databases efficiently store, manage and query connected data at scale

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to How Graph Databases efficiently store, manage and query connected data at scale

Similar to How Graph Databases efficiently store, manage and query connected data at scale (20)

More from jexp

More from jexp (20)

Recently uploaded

Recently uploaded (20)

How Graph Databases efficiently store, manage and query connected data at scale