SlideShare a Scribd company logo
1 of 22
Download to read offline
Graph at Scale 
10/22/14 
Brad Nussbaum 
CTO | MediaHound 
brad@mediahound.com | @bradnussbaum 
www.mediahound.com
MediaHound started with a search…
But Entertainment is more than just Film & TV
Entertainment Preferences Are Scattered 
Music Movies Television Books
All Content 
one platform that connects: 
All Sources All Devices 
All Brands All Artists All Users
The Entertainment Graph 
A comprehensive database that brings together movies, 
books, games, music, and TV, including the cast & crew, 
sources, reviews, categories, genres, lists and more! 
The Entertainment Graph powers meaningful recommendations, 
exciting data insights and comprehensive social discovery.
Use Cases: 
Academy Award Winners On Netflix 
MATCH (c:Collection)-[:CONTAINS]-(m:Movie) WHERE (m)- 
[:MATCHED_SOURCE]-(n:NETFLIX) AND c.name=“Academy Award 
Winners” RETURN m; 
Movies and Shows Based On Zombie Books 
MATCH (m:Media)-[:BASED_ON]-(b:Book)-[:HAS_TRAIT]-(z:Trait) 
WHERE (m.type=“Movie” OR m.type=“Show”) AND z.name=“Zombies” 
RETURN m; 
To access ‘The Movie Graph’ mini app (which uses a different model than above), from your browser, run 
:play movie graph
Entertainment Recommendations 
Collaborative Filtering: 
If Joe, Amy and Steve like Gladiator, AND Joe and Amy like Toy Story, 
THEN MediaHound recommends Toy Story to Steve. 
Graph Influencers 
Joe is an early adopter. 
Joe, Amy and Steve like several things in common. 
MediaHound recommends Amy and Steve follow Joe and Amy 
Joe discovers the next big hit and shares it on his feed 
Amy and Steve see Steve’s post and give it a listen.
We needed our graph database to perform 
under sustained user write load AND during 
heavy batch update operations. 
We needed to recommend media content in 
real-time which required many concurrent 
pattern matching operations on the graph.
We realized through trial and error that using 
the Transactional Cypher HTTP Endpoint was 
the BEST solution to control batch writes. 
POST http://localhost:7474/db/data/transaction/commit 
Accept: application/json; charset=UTF-8 
Content-Type: application/json 
{ 
"statements": [ 
{ 
"statement": "MERGE(n:User{username:"bradnussbaum"})-[:FOLLOWS]->(m:User{username:"bennussbaum"});" 
}, 
{ 
"statement": "MERGE(n:User{username:"bradnussbaum"})<-[:FOLLOWS]-(m:User{username:"bennussbaum"});" 
} 
] 
}
We ran tests using the low-level kernel and 
found that sustained transaction writes 
performed optimally between 400-2,000 nodes 
and relationships per transaction. 
Compare the difference between… 
• Writing a single relationship (33 bytes) per transaction f 
or 10k iterations compared to 
• Writing 1k relationships per transaction for 10 iterations. 
**As of 2.2, Neo4j will batch writes on the server 
http://neo4j.com/docs/stable/linux-performance-guide.html 
git clone git@github.com:neo4j-contrib/tooling.git
We used Enterprise Integration Patterns (EIP) 
to create optimal batch sizes for each 
transaction. 
• Splitters to break down larger messages 
• Aggregators to combine single CQL statements t 
ogether into a single batch transaction 
• Throttling to control concurrent requests and requests p 
er second
We frequently run 40+ concurrent write 
transactions to a 3 instance cluster for hours 
at time. 
Deadlocks can occur often with many concurrent write o 
perations. 
• Retry Transient Errors after a small period of time. 
• Use the Error Index to split failed TX statements. 
Read here to learn all the error status codes, seriously. 
http://neo4j.com/docs/stable/status-codes.html
Test write throughput on your cluster with the 
push factor you plan to use in production and 
intentionally kill your master under load. 
You need to have two load balancers: 
• One for all the instances you want performing reads 
• One for your master ONLY  (send writes here) 
Master Check: /db/manage/server/ha/master 
- Returns true|false 
Slave Check: /db/manage/server/ha/slave 
- Returns true|false 
Available Check: /db/manage/server/ha/available 
- Returns master|slave
Check your driver for transaction support. 
Embedded mode has full transaction support 
but most remote drivers do not at this time. 
This will be changing in the near future…depending on 
which driver you use. 
**Spring Data Neo4j is actively being developed to included these features as part of 2.2.
Graph at Scale 
10/22/14 
Ben Nussbaum 
Director of Engineering | MediaHound 
ben@mediahound.com | @bennussbaum 
www.mediahound.com
We built custom algorithms that needed run-time 
decision making as Neo4j Extensions 
with Spring Data Neo4j. 
• Cache abstraction with Google’s Guava to build large 
in-memory indexes of nodes and relationships. 
• Integration for jobs instructions and results to and from 
the broker. 
• Async for batch job processing. 
https://github.com/AtomRain/neo4j-extensions
We took advantage of spot processing from 
AWS to run our custom extension algoritms. 
• On-demand graph processing with as many instances 
at a time as needed (we have used up to 9). 
• Concurrent job operations per spot. 
• Cache optimizations based on Labels and context of 
the jobs.
We built a flexible job controller that enables 
concurrent job processing on spot instances 
• Large jobs are broken into smaller jobs that can be 
processed by a single spot instance. 
• Spots process unit jobs and return results. If a spot 
dies, the job stays in the queue and another spot picks 
it up. 
• Memory and CPU constraints on an instance make this 
a necessity, especially when processing 30M+ songs.
Spot instances run Neo4j in SINGLE mode 
and stay up to date using a Topic. 
Neo4j HA 
Spot 1 
SINGLE 
Job 
Result 
MQ 
Spot 2 
SINGLE 
Job 
Process 
MQ 
ESB 
SDN SDN 
TX 
Topic 
1 
2 
3 
4 
5 
1. ESB sends Jobs to MQ 
2. Spots consume job instructions, 
process and send results b 
ack to MQ 
3. ESB posts jobs results to HA 
4. On successful post, send u 
pdates to Topic 
5. Spots consume from Topic to s 
tay u to date
Batch jobs return thousands of CQL 
statements which must not be dependent on 
any statements before or after. 
• Compound statements to create nodes and 
relationships for specific sub-graphs to avoid the need 
for layering wherever possible. If not… 
• Run jobs in a linear phases (layering) to create nodes 
first then connect relationships
Graph at Scale 
10/22/14 
Q/A

More Related Content

What's hot

3.1.Performance and BigData Ecosystem
3.1.Performance and BigData Ecosystem3.1.Performance and BigData Ecosystem
3.1.Performance and BigData Ecosystem振东 刘
 
Flink Forward Berlin 2017: Jörg Schad, Till Rohrmann - Apache Flink meets Apa...
Flink Forward Berlin 2017: Jörg Schad, Till Rohrmann - Apache Flink meets Apa...Flink Forward Berlin 2017: Jörg Schad, Till Rohrmann - Apache Flink meets Apa...
Flink Forward Berlin 2017: Jörg Schad, Till Rohrmann - Apache Flink meets Apa...Flink Forward
 
Flink Forward San Francisco 2019: Apache Beam portability in the times of rea...
Flink Forward San Francisco 2019: Apache Beam portability in the times of rea...Flink Forward San Francisco 2019: Apache Beam portability in the times of rea...
Flink Forward San Francisco 2019: Apache Beam portability in the times of rea...Flink Forward
 
Flink Forward San Francisco 2019: Scaling a real-time streaming warehouse wit...
Flink Forward San Francisco 2019: Scaling a real-time streaming warehouse wit...Flink Forward San Francisco 2019: Scaling a real-time streaming warehouse wit...
Flink Forward San Francisco 2019: Scaling a real-time streaming warehouse wit...Flink Forward
 
Cowboy dating with big data
Cowboy dating with big data Cowboy dating with big data
Cowboy dating with big data b0ris_1
 
Audience counting at Scale
Audience counting at ScaleAudience counting at Scale
Audience counting at Scaleb0ris_1
 
Flink Forward San Francisco 2019: Elastic Data Processing with Apache Flink a...
Flink Forward San Francisco 2019: Elastic Data Processing with Apache Flink a...Flink Forward San Francisco 2019: Elastic Data Processing with Apache Flink a...
Flink Forward San Francisco 2019: Elastic Data Processing with Apache Flink a...Flink Forward
 
Putting Kafka Together with the Best of Google Cloud Platform
Putting Kafka Together with the Best of Google Cloud Platform Putting Kafka Together with the Best of Google Cloud Platform
Putting Kafka Together with the Best of Google Cloud Platform confluent
 
Future of Apache Flink Deployments: Containers, Kubernetes and More - Flink F...
Future of Apache Flink Deployments: Containers, Kubernetes and More - Flink F...Future of Apache Flink Deployments: Containers, Kubernetes and More - Flink F...
Future of Apache Flink Deployments: Containers, Kubernetes and More - Flink F...Till Rohrmann
 
Flink Forward Berlin 2017: Piotr Wawrzyniak - Extending Apache Flink stream p...
Flink Forward Berlin 2017: Piotr Wawrzyniak - Extending Apache Flink stream p...Flink Forward Berlin 2017: Piotr Wawrzyniak - Extending Apache Flink stream p...
Flink Forward Berlin 2017: Piotr Wawrzyniak - Extending Apache Flink stream p...Flink Forward
 
Flink Forward San Francisco 2019: Developing and operating real-time applicat...
Flink Forward San Francisco 2019: Developing and operating real-time applicat...Flink Forward San Francisco 2019: Developing and operating real-time applicat...
Flink Forward San Francisco 2019: Developing and operating real-time applicat...Flink Forward
 
Flink Forward San Francisco 2019: The Trade Desk's Year in Flink - Jonathan ...
Flink Forward San Francisco 2019: The Trade Desk's Year in Flink -  Jonathan ...Flink Forward San Francisco 2019: The Trade Desk's Year in Flink -  Jonathan ...
Flink Forward San Francisco 2019: The Trade Desk's Year in Flink - Jonathan ...Flink Forward
 
Streaming your Lyft Ride Prices - Flink Forward SF 2019
Streaming your Lyft Ride Prices - Flink Forward SF 2019Streaming your Lyft Ride Prices - Flink Forward SF 2019
Streaming your Lyft Ride Prices - Flink Forward SF 2019Thomas Weise
 
Flink Forward San Francisco 2019: Building production Flink jobs with Airstre...
Flink Forward San Francisco 2019: Building production Flink jobs with Airstre...Flink Forward San Francisco 2019: Building production Flink jobs with Airstre...
Flink Forward San Francisco 2019: Building production Flink jobs with Airstre...Flink Forward
 
Apache Beam @ GCPUG.TW Flink.TW 20161006
Apache Beam @ GCPUG.TW Flink.TW 20161006Apache Beam @ GCPUG.TW Flink.TW 20161006
Apache Beam @ GCPUG.TW Flink.TW 20161006Randy Huang
 
Flink Forward SF 2017: Bill Liu & Haohui Mai - AthenaX : Uber’s streaming pro...
Flink Forward SF 2017: Bill Liu & Haohui Mai - AthenaX : Uber’s streaming pro...Flink Forward SF 2017: Bill Liu & Haohui Mai - AthenaX : Uber’s streaming pro...
Flink Forward SF 2017: Bill Liu & Haohui Mai - AthenaX : Uber’s streaming pro...Flink Forward
 
Administrative techniques to reduce Kafka costs | Anna Kepler, Viasat
Administrative techniques to reduce Kafka costs | Anna Kepler, ViasatAdministrative techniques to reduce Kafka costs | Anna Kepler, Viasat
Administrative techniques to reduce Kafka costs | Anna Kepler, ViasatHostedbyConfluent
 

What's hot (20)

3.1.Performance and BigData Ecosystem
3.1.Performance and BigData Ecosystem3.1.Performance and BigData Ecosystem
3.1.Performance and BigData Ecosystem
 
Flink Forward Berlin 2017: Jörg Schad, Till Rohrmann - Apache Flink meets Apa...
Flink Forward Berlin 2017: Jörg Schad, Till Rohrmann - Apache Flink meets Apa...Flink Forward Berlin 2017: Jörg Schad, Till Rohrmann - Apache Flink meets Apa...
Flink Forward Berlin 2017: Jörg Schad, Till Rohrmann - Apache Flink meets Apa...
 
Flink Forward San Francisco 2019: Apache Beam portability in the times of rea...
Flink Forward San Francisco 2019: Apache Beam portability in the times of rea...Flink Forward San Francisco 2019: Apache Beam portability in the times of rea...
Flink Forward San Francisco 2019: Apache Beam portability in the times of rea...
 
Flink Forward San Francisco 2019: Scaling a real-time streaming warehouse wit...
Flink Forward San Francisco 2019: Scaling a real-time streaming warehouse wit...Flink Forward San Francisco 2019: Scaling a real-time streaming warehouse wit...
Flink Forward San Francisco 2019: Scaling a real-time streaming warehouse wit...
 
Cowboy dating with big data
Cowboy dating with big data Cowboy dating with big data
Cowboy dating with big data
 
Audience counting at Scale
Audience counting at ScaleAudience counting at Scale
Audience counting at Scale
 
Flink Forward San Francisco 2019: Elastic Data Processing with Apache Flink a...
Flink Forward San Francisco 2019: Elastic Data Processing with Apache Flink a...Flink Forward San Francisco 2019: Elastic Data Processing with Apache Flink a...
Flink Forward San Francisco 2019: Elastic Data Processing with Apache Flink a...
 
Optimized Hive replication
Optimized Hive replicationOptimized Hive replication
Optimized Hive replication
 
Putting Kafka Together with the Best of Google Cloud Platform
Putting Kafka Together with the Best of Google Cloud Platform Putting Kafka Together with the Best of Google Cloud Platform
Putting Kafka Together with the Best of Google Cloud Platform
 
Future of Apache Flink Deployments: Containers, Kubernetes and More - Flink F...
Future of Apache Flink Deployments: Containers, Kubernetes and More - Flink F...Future of Apache Flink Deployments: Containers, Kubernetes and More - Flink F...
Future of Apache Flink Deployments: Containers, Kubernetes and More - Flink F...
 
Flink Forward Berlin 2017: Piotr Wawrzyniak - Extending Apache Flink stream p...
Flink Forward Berlin 2017: Piotr Wawrzyniak - Extending Apache Flink stream p...Flink Forward Berlin 2017: Piotr Wawrzyniak - Extending Apache Flink stream p...
Flink Forward Berlin 2017: Piotr Wawrzyniak - Extending Apache Flink stream p...
 
Flink Forward San Francisco 2019: Developing and operating real-time applicat...
Flink Forward San Francisco 2019: Developing and operating real-time applicat...Flink Forward San Francisco 2019: Developing and operating real-time applicat...
Flink Forward San Francisco 2019: Developing and operating real-time applicat...
 
Flink Forward San Francisco 2019: The Trade Desk's Year in Flink - Jonathan ...
Flink Forward San Francisco 2019: The Trade Desk's Year in Flink -  Jonathan ...Flink Forward San Francisco 2019: The Trade Desk's Year in Flink -  Jonathan ...
Flink Forward San Francisco 2019: The Trade Desk's Year in Flink - Jonathan ...
 
Streaming your Lyft Ride Prices - Flink Forward SF 2019
Streaming your Lyft Ride Prices - Flink Forward SF 2019Streaming your Lyft Ride Prices - Flink Forward SF 2019
Streaming your Lyft Ride Prices - Flink Forward SF 2019
 
Flink Forward San Francisco 2019: Building production Flink jobs with Airstre...
Flink Forward San Francisco 2019: Building production Flink jobs with Airstre...Flink Forward San Francisco 2019: Building production Flink jobs with Airstre...
Flink Forward San Francisco 2019: Building production Flink jobs with Airstre...
 
Gobblin on-aws
Gobblin on-awsGobblin on-aws
Gobblin on-aws
 
Apache Beam @ GCPUG.TW Flink.TW 20161006
Apache Beam @ GCPUG.TW Flink.TW 20161006Apache Beam @ GCPUG.TW Flink.TW 20161006
Apache Beam @ GCPUG.TW Flink.TW 20161006
 
Flink Forward SF 2017: Bill Liu & Haohui Mai - AthenaX : Uber’s streaming pro...
Flink Forward SF 2017: Bill Liu & Haohui Mai - AthenaX : Uber’s streaming pro...Flink Forward SF 2017: Bill Liu & Haohui Mai - AthenaX : Uber’s streaming pro...
Flink Forward SF 2017: Bill Liu & Haohui Mai - AthenaX : Uber’s streaming pro...
 
Running Cloudbreak on Kubernetes
Running Cloudbreak on KubernetesRunning Cloudbreak on Kubernetes
Running Cloudbreak on Kubernetes
 
Administrative techniques to reduce Kafka costs | Anna Kepler, Viasat
Administrative techniques to reduce Kafka costs | Anna Kepler, ViasatAdministrative techniques to reduce Kafka costs | Anna Kepler, Viasat
Administrative techniques to reduce Kafka costs | Anna Kepler, Viasat
 

Viewers also liked

Graph all the things
Graph all the thingsGraph all the things
Graph all the thingsNeo4j
 
Neo4j Makes Graphs Easy
Neo4j Makes Graphs EasyNeo4j Makes Graphs Easy
Neo4j Makes Graphs EasyNeo4j
 
Transparency One : La (re)découverte de la chaîne d'approvisionnement
Transparency One : La (re)découverte de la chaîne d'approvisionnementTransparency One : La (re)découverte de la chaîne d'approvisionnement
Transparency One : La (re)découverte de la chaîne d'approvisionnementNeo4j
 
Graph Your Business - GraphDay JimWebber
Graph Your Business - GraphDay JimWebberGraph Your Business - GraphDay JimWebber
Graph Your Business - GraphDay JimWebberNeo4j
 
GraphConnect 2014 SF: Betting the Company on a Graph Database - Part 2
GraphConnect 2014 SF: Betting the Company on a Graph Database - Part 2GraphConnect 2014 SF: Betting the Company on a Graph Database - Part 2
GraphConnect 2014 SF: Betting the Company on a Graph Database - Part 2Neo4j
 
Neo4j Makes Graphs Easy? - GraphDay AmandaLaucher
Neo4j Makes Graphs Easy? - GraphDay AmandaLaucherNeo4j Makes Graphs Easy? - GraphDay AmandaLaucher
Neo4j Makes Graphs Easy? - GraphDay AmandaLaucherNeo4j
 
Graph your business
Graph your businessGraph your business
Graph your businessNeo4j
 
GraphTalk Frankfurt - Master Data Management bei der Bayerischen Versicherung
GraphTalk Frankfurt - Master Data Management bei der Bayerischen VersicherungGraphTalk Frankfurt - Master Data Management bei der Bayerischen Versicherung
GraphTalk Frankfurt - Master Data Management bei der Bayerischen VersicherungNeo4j
 
Metadata and Access Control
Metadata and Access ControlMetadata and Access Control
Metadata and Access ControlNeo4j
 
GraphConnect 2014 SF: The Business Graph
GraphConnect 2014 SF: The Business GraphGraphConnect 2014 SF: The Business Graph
GraphConnect 2014 SF: The Business GraphNeo4j
 
Graph Search and Discovery for your Dark Data
Graph Search and Discovery for your Dark DataGraph Search and Discovery for your Dark Data
Graph Search and Discovery for your Dark DataNeo4j
 
GraphDay Noble/Coolio
GraphDay Noble/CoolioGraphDay Noble/Coolio
GraphDay Noble/CoolioNeo4j
 
Meetup Analytics with R and Neo4j
Meetup Analytics with R and Neo4jMeetup Analytics with R and Neo4j
Meetup Analytics with R and Neo4jNeo4j
 
Graphs fun vjug2
Graphs fun vjug2Graphs fun vjug2
Graphs fun vjug2Neo4j
 
GraphTalk - Semantische Netze mit structr
GraphTalk - Semantische Netze mit structrGraphTalk - Semantische Netze mit structr
GraphTalk - Semantische Netze mit structrNeo4j
 
GraphTalk Frankfurt - Einführung in Graphdatenbanken
GraphTalk Frankfurt - Einführung in GraphdatenbankenGraphTalk Frankfurt - Einführung in Graphdatenbanken
GraphTalk Frankfurt - Einführung in GraphdatenbankenNeo4j
 
Graph all the things - PRathle
Graph all the things - PRathleGraph all the things - PRathle
Graph all the things - PRathleNeo4j
 
GraphTalks - Einführung
GraphTalks - EinführungGraphTalks - Einführung
GraphTalks - EinführungNeo4j
 
GraphTalk - Semantisches PDM bei Schleich
GraphTalk - Semantisches PDM bei Schleich GraphTalk - Semantisches PDM bei Schleich
GraphTalk - Semantisches PDM bei Schleich Neo4j
 
RDBMS to Graphs
RDBMS to GraphsRDBMS to Graphs
RDBMS to GraphsNeo4j
 

Viewers also liked (20)

Graph all the things
Graph all the thingsGraph all the things
Graph all the things
 
Neo4j Makes Graphs Easy
Neo4j Makes Graphs EasyNeo4j Makes Graphs Easy
Neo4j Makes Graphs Easy
 
Transparency One : La (re)découverte de la chaîne d'approvisionnement
Transparency One : La (re)découverte de la chaîne d'approvisionnementTransparency One : La (re)découverte de la chaîne d'approvisionnement
Transparency One : La (re)découverte de la chaîne d'approvisionnement
 
Graph Your Business - GraphDay JimWebber
Graph Your Business - GraphDay JimWebberGraph Your Business - GraphDay JimWebber
Graph Your Business - GraphDay JimWebber
 
GraphConnect 2014 SF: Betting the Company on a Graph Database - Part 2
GraphConnect 2014 SF: Betting the Company on a Graph Database - Part 2GraphConnect 2014 SF: Betting the Company on a Graph Database - Part 2
GraphConnect 2014 SF: Betting the Company on a Graph Database - Part 2
 
Neo4j Makes Graphs Easy? - GraphDay AmandaLaucher
Neo4j Makes Graphs Easy? - GraphDay AmandaLaucherNeo4j Makes Graphs Easy? - GraphDay AmandaLaucher
Neo4j Makes Graphs Easy? - GraphDay AmandaLaucher
 
Graph your business
Graph your businessGraph your business
Graph your business
 
GraphTalk Frankfurt - Master Data Management bei der Bayerischen Versicherung
GraphTalk Frankfurt - Master Data Management bei der Bayerischen VersicherungGraphTalk Frankfurt - Master Data Management bei der Bayerischen Versicherung
GraphTalk Frankfurt - Master Data Management bei der Bayerischen Versicherung
 
Metadata and Access Control
Metadata and Access ControlMetadata and Access Control
Metadata and Access Control
 
GraphConnect 2014 SF: The Business Graph
GraphConnect 2014 SF: The Business GraphGraphConnect 2014 SF: The Business Graph
GraphConnect 2014 SF: The Business Graph
 
Graph Search and Discovery for your Dark Data
Graph Search and Discovery for your Dark DataGraph Search and Discovery for your Dark Data
Graph Search and Discovery for your Dark Data
 
GraphDay Noble/Coolio
GraphDay Noble/CoolioGraphDay Noble/Coolio
GraphDay Noble/Coolio
 
Meetup Analytics with R and Neo4j
Meetup Analytics with R and Neo4jMeetup Analytics with R and Neo4j
Meetup Analytics with R and Neo4j
 
Graphs fun vjug2
Graphs fun vjug2Graphs fun vjug2
Graphs fun vjug2
 
GraphTalk - Semantische Netze mit structr
GraphTalk - Semantische Netze mit structrGraphTalk - Semantische Netze mit structr
GraphTalk - Semantische Netze mit structr
 
GraphTalk Frankfurt - Einführung in Graphdatenbanken
GraphTalk Frankfurt - Einführung in GraphdatenbankenGraphTalk Frankfurt - Einführung in Graphdatenbanken
GraphTalk Frankfurt - Einführung in Graphdatenbanken
 
Graph all the things - PRathle
Graph all the things - PRathleGraph all the things - PRathle
Graph all the things - PRathle
 
GraphTalks - Einführung
GraphTalks - EinführungGraphTalks - Einführung
GraphTalks - Einführung
 
GraphTalk - Semantisches PDM bei Schleich
GraphTalk - Semantisches PDM bei Schleich GraphTalk - Semantisches PDM bei Schleich
GraphTalk - Semantisches PDM bei Schleich
 
RDBMS to Graphs
RDBMS to GraphsRDBMS to Graphs
RDBMS to Graphs
 

Similar to GraphConnect 2014 SF: Neo4j at Scale using Enterprise Integration Patterns

Spark (Structured) Streaming vs. Kafka Streams - two stream processing platfo...
Spark (Structured) Streaming vs. Kafka Streams - two stream processing platfo...Spark (Structured) Streaming vs. Kafka Streams - two stream processing platfo...
Spark (Structured) Streaming vs. Kafka Streams - two stream processing platfo...Guido Schmutz
 
Introduction to LAVA Workload Scheduler
Introduction to LAVA Workload SchedulerIntroduction to LAVA Workload Scheduler
Introduction to LAVA Workload SchedulerNopparat Nopkuat
 
Spark (Structured) Streaming vs. Kafka Streams - two stream processing platfo...
Spark (Structured) Streaming vs. Kafka Streams - two stream processing platfo...Spark (Structured) Streaming vs. Kafka Streams - two stream processing platfo...
Spark (Structured) Streaming vs. Kafka Streams - two stream processing platfo...Guido Schmutz
 
Headaches and Breakthroughs in Building Continuous Applications
Headaches and Breakthroughs in Building Continuous ApplicationsHeadaches and Breakthroughs in Building Continuous Applications
Headaches and Breakthroughs in Building Continuous ApplicationsDatabricks
 
Spark + AI Summit 2019: Headaches and Breakthroughs in Building Continuous Ap...
Spark + AI Summit 2019: Headaches and Breakthroughs in Building Continuous Ap...Spark + AI Summit 2019: Headaches and Breakthroughs in Building Continuous Ap...
Spark + AI Summit 2019: Headaches and Breakthroughs in Building Continuous Ap...Landon Robinson
 
Flink Forward SF 2017: Malo Deniélou - No shard left behind: Dynamic work re...
Flink Forward SF 2017: Malo Deniélou -  No shard left behind: Dynamic work re...Flink Forward SF 2017: Malo Deniélou -  No shard left behind: Dynamic work re...
Flink Forward SF 2017: Malo Deniélou - No shard left behind: Dynamic work re...Flink Forward
 
Closing Keynote
Closing KeynoteClosing Keynote
Closing KeynoteNeo4j
 
Malo Denielou - No shard left behind: Dynamic work rebalancing in Apache Beam
Malo Denielou - No shard left behind: Dynamic work rebalancing in Apache BeamMalo Denielou - No shard left behind: Dynamic work rebalancing in Apache Beam
Malo Denielou - No shard left behind: Dynamic work rebalancing in Apache BeamFlink Forward
 
Hadoop and HBase experiences in perf log project
Hadoop and HBase experiences in perf log projectHadoop and HBase experiences in perf log project
Hadoop and HBase experiences in perf log projectMao Geng
 
Essential Data Engineering for Data Scientist
Essential Data Engineering for Data Scientist Essential Data Engineering for Data Scientist
Essential Data Engineering for Data Scientist SoftServe
 
Unified, Efficient, and Portable Data Processing with Apache Beam
Unified, Efficient, and Portable Data Processing with Apache BeamUnified, Efficient, and Portable Data Processing with Apache Beam
Unified, Efficient, and Portable Data Processing with Apache BeamDataWorks Summit/Hadoop Summit
 
Productionizing your Streaming Jobs
Productionizing your Streaming JobsProductionizing your Streaming Jobs
Productionizing your Streaming JobsDatabricks
 
Apache Spark Listeners: A Crash Course in Fast, Easy Monitoring
Apache Spark Listeners: A Crash Course in Fast, Easy MonitoringApache Spark Listeners: A Crash Course in Fast, Easy Monitoring
Apache Spark Listeners: A Crash Course in Fast, Easy MonitoringDatabricks
 
The Next Generation of Data Processing and Open Source
The Next Generation of Data Processing and Open SourceThe Next Generation of Data Processing and Open Source
The Next Generation of Data Processing and Open SourceDataWorks Summit/Hadoop Summit
 
NodeJS guide for beginners
NodeJS guide for beginnersNodeJS guide for beginners
NodeJS guide for beginnersEnoch Joshua
 
PipelineAI + TensorFlow AI + Spark ML + Kuberenetes + Istio + AWS SageMaker +...
PipelineAI + TensorFlow AI + Spark ML + Kuberenetes + Istio + AWS SageMaker +...PipelineAI + TensorFlow AI + Spark ML + Kuberenetes + Istio + AWS SageMaker +...
PipelineAI + TensorFlow AI + Spark ML + Kuberenetes + Istio + AWS SageMaker +...Chris Fregly
 
Building Continuous Application with Structured Streaming and Real-Time Data ...
Building Continuous Application with Structured Streaming and Real-Time Data ...Building Continuous Application with Structured Streaming and Real-Time Data ...
Building Continuous Application with Structured Streaming and Real-Time Data ...Databricks
 
Azure tales: a real world CQRS and ES Deep Dive - Andrea Saltarello
Azure tales: a real world CQRS and ES Deep Dive - Andrea SaltarelloAzure tales: a real world CQRS and ES Deep Dive - Andrea Saltarello
Azure tales: a real world CQRS and ES Deep Dive - Andrea SaltarelloITCamp
 
RUCUG: 10. Robert Morris:Жизнь в окопах виртуализационной войны
RUCUG: 10. Robert Morris:Жизнь в окопах виртуализационной войныRUCUG: 10. Robert Morris:Жизнь в окопах виртуализационной войны
RUCUG: 10. Robert Morris:Жизнь в окопах виртуализационной войныDenis Gundarev
 

Similar to GraphConnect 2014 SF: Neo4j at Scale using Enterprise Integration Patterns (20)

Spark (Structured) Streaming vs. Kafka Streams - two stream processing platfo...
Spark (Structured) Streaming vs. Kafka Streams - two stream processing platfo...Spark (Structured) Streaming vs. Kafka Streams - two stream processing platfo...
Spark (Structured) Streaming vs. Kafka Streams - two stream processing platfo...
 
Introduction to LAVA Workload Scheduler
Introduction to LAVA Workload SchedulerIntroduction to LAVA Workload Scheduler
Introduction to LAVA Workload Scheduler
 
Spark (Structured) Streaming vs. Kafka Streams - two stream processing platfo...
Spark (Structured) Streaming vs. Kafka Streams - two stream processing platfo...Spark (Structured) Streaming vs. Kafka Streams - two stream processing platfo...
Spark (Structured) Streaming vs. Kafka Streams - two stream processing platfo...
 
Headaches and Breakthroughs in Building Continuous Applications
Headaches and Breakthroughs in Building Continuous ApplicationsHeadaches and Breakthroughs in Building Continuous Applications
Headaches and Breakthroughs in Building Continuous Applications
 
Spark + AI Summit 2019: Headaches and Breakthroughs in Building Continuous Ap...
Spark + AI Summit 2019: Headaches and Breakthroughs in Building Continuous Ap...Spark + AI Summit 2019: Headaches and Breakthroughs in Building Continuous Ap...
Spark + AI Summit 2019: Headaches and Breakthroughs in Building Continuous Ap...
 
Flink Forward SF 2017: Malo Deniélou - No shard left behind: Dynamic work re...
Flink Forward SF 2017: Malo Deniélou -  No shard left behind: Dynamic work re...Flink Forward SF 2017: Malo Deniélou -  No shard left behind: Dynamic work re...
Flink Forward SF 2017: Malo Deniélou - No shard left behind: Dynamic work re...
 
Closing Keynote
Closing KeynoteClosing Keynote
Closing Keynote
 
Malo Denielou - No shard left behind: Dynamic work rebalancing in Apache Beam
Malo Denielou - No shard left behind: Dynamic work rebalancing in Apache BeamMalo Denielou - No shard left behind: Dynamic work rebalancing in Apache Beam
Malo Denielou - No shard left behind: Dynamic work rebalancing in Apache Beam
 
Hadoop and HBase experiences in perf log project
Hadoop and HBase experiences in perf log projectHadoop and HBase experiences in perf log project
Hadoop and HBase experiences in perf log project
 
Essential Data Engineering for Data Scientist
Essential Data Engineering for Data Scientist Essential Data Engineering for Data Scientist
Essential Data Engineering for Data Scientist
 
Why meteor
Why meteorWhy meteor
Why meteor
 
Unified, Efficient, and Portable Data Processing with Apache Beam
Unified, Efficient, and Portable Data Processing with Apache BeamUnified, Efficient, and Portable Data Processing with Apache Beam
Unified, Efficient, and Portable Data Processing with Apache Beam
 
Productionizing your Streaming Jobs
Productionizing your Streaming JobsProductionizing your Streaming Jobs
Productionizing your Streaming Jobs
 
Apache Spark Listeners: A Crash Course in Fast, Easy Monitoring
Apache Spark Listeners: A Crash Course in Fast, Easy MonitoringApache Spark Listeners: A Crash Course in Fast, Easy Monitoring
Apache Spark Listeners: A Crash Course in Fast, Easy Monitoring
 
The Next Generation of Data Processing and Open Source
The Next Generation of Data Processing and Open SourceThe Next Generation of Data Processing and Open Source
The Next Generation of Data Processing and Open Source
 
NodeJS guide for beginners
NodeJS guide for beginnersNodeJS guide for beginners
NodeJS guide for beginners
 
PipelineAI + TensorFlow AI + Spark ML + Kuberenetes + Istio + AWS SageMaker +...
PipelineAI + TensorFlow AI + Spark ML + Kuberenetes + Istio + AWS SageMaker +...PipelineAI + TensorFlow AI + Spark ML + Kuberenetes + Istio + AWS SageMaker +...
PipelineAI + TensorFlow AI + Spark ML + Kuberenetes + Istio + AWS SageMaker +...
 
Building Continuous Application with Structured Streaming and Real-Time Data ...
Building Continuous Application with Structured Streaming and Real-Time Data ...Building Continuous Application with Structured Streaming and Real-Time Data ...
Building Continuous Application with Structured Streaming and Real-Time Data ...
 
Azure tales: a real world CQRS and ES Deep Dive - Andrea Saltarello
Azure tales: a real world CQRS and ES Deep Dive - Andrea SaltarelloAzure tales: a real world CQRS and ES Deep Dive - Andrea Saltarello
Azure tales: a real world CQRS and ES Deep Dive - Andrea Saltarello
 
RUCUG: 10. Robert Morris:Жизнь в окопах виртуализационной войны
RUCUG: 10. Robert Morris:Жизнь в окопах виртуализационной войныRUCUG: 10. Robert Morris:Жизнь в окопах виртуализационной войны
RUCUG: 10. Robert Morris:Жизнь в окопах виртуализационной войны
 

More from Neo4j

Neo4j Graph Summit 2024 Workshop - EMEA - Breda_and_Munchen.pdf
Neo4j Graph Summit 2024 Workshop - EMEA - Breda_and_Munchen.pdfNeo4j Graph Summit 2024 Workshop - EMEA - Breda_and_Munchen.pdf
Neo4j Graph Summit 2024 Workshop - EMEA - Breda_and_Munchen.pdfNeo4j
 
Enabling GenAI Breakthroughs with Knowledge Graphs
Enabling GenAI Breakthroughs with Knowledge GraphsEnabling GenAI Breakthroughs with Knowledge Graphs
Enabling GenAI Breakthroughs with Knowledge GraphsNeo4j
 
Neo4j_Anurag Tandon_Product Vision and Roadmap.Benelux.pptx.pdf
Neo4j_Anurag Tandon_Product Vision and Roadmap.Benelux.pptx.pdfNeo4j_Anurag Tandon_Product Vision and Roadmap.Benelux.pptx.pdf
Neo4j_Anurag Tandon_Product Vision and Roadmap.Benelux.pptx.pdfNeo4j
 
Neo4j_Jesus Barrasa_The Art of the Possible with Graph.pptx.pdf
Neo4j_Jesus Barrasa_The Art of the Possible with Graph.pptx.pdfNeo4j_Jesus Barrasa_The Art of the Possible with Graph.pptx.pdf
Neo4j_Jesus Barrasa_The Art of the Possible with Graph.pptx.pdfNeo4j
 
Swift_Maintaining Critical Standards(...).pptx.pdf
Swift_Maintaining Critical Standards(...).pptx.pdfSwift_Maintaining Critical Standards(...).pptx.pdf
Swift_Maintaining Critical Standards(...).pptx.pdfNeo4j
 
Deloitte+RedCross_Talk to your data with Knowledge-enriched Generative AI.ppt...
Deloitte+RedCross_Talk to your data with Knowledge-enriched Generative AI.ppt...Deloitte+RedCross_Talk to your data with Knowledge-enriched Generative AI.ppt...
Deloitte+RedCross_Talk to your data with Knowledge-enriched Generative AI.ppt...Neo4j
 
Ingka Digital_Linked Metadata by Design.pdf
Ingka Digital_Linked Metadata by Design.pdfIngka Digital_Linked Metadata by Design.pdf
Ingka Digital_Linked Metadata by Design.pdfNeo4j
 
Discover Neo4j Aura_ The Future of Graph Database-as-a-Service Workshop_3.13.24
Discover Neo4j Aura_ The Future of Graph Database-as-a-Service Workshop_3.13.24Discover Neo4j Aura_ The Future of Graph Database-as-a-Service Workshop_3.13.24
Discover Neo4j Aura_ The Future of Graph Database-as-a-Service Workshop_3.13.24Neo4j
 
GraphSummit Copenhagen 2024 - Neo4j Vision and Roadmap.pptx
GraphSummit Copenhagen 2024 - Neo4j Vision and Roadmap.pptxGraphSummit Copenhagen 2024 - Neo4j Vision and Roadmap.pptx
GraphSummit Copenhagen 2024 - Neo4j Vision and Roadmap.pptxNeo4j
 
Emil Eifrem at GraphSummit Copenhagen 2024 - The Art of the Possible.pptx
Emil Eifrem at GraphSummit Copenhagen 2024 - The Art of the Possible.pptxEmil Eifrem at GraphSummit Copenhagen 2024 - The Art of the Possible.pptx
Emil Eifrem at GraphSummit Copenhagen 2024 - The Art of the Possible.pptxNeo4j
 
Identification of insulin-resistance genes with Knowledge Graphs topology and...
Identification of insulin-resistance genes with Knowledge Graphs topology and...Identification of insulin-resistance genes with Knowledge Graphs topology and...
Identification of insulin-resistance genes with Knowledge Graphs topology and...Neo4j
 
Novo Nordisk's journey in developing an open-source application on Neo4j
Novo Nordisk's journey in developing an open-source application on Neo4jNovo Nordisk's journey in developing an open-source application on Neo4j
Novo Nordisk's journey in developing an open-source application on Neo4jNeo4j
 
EY: Graphs as Critical Enablers for LLM-based Assistants- the Case of Custome...
EY: Graphs as Critical Enablers for LLM-based Assistants- the Case of Custome...EY: Graphs as Critical Enablers for LLM-based Assistants- the Case of Custome...
EY: Graphs as Critical Enablers for LLM-based Assistants- the Case of Custome...Neo4j
 
GraphSummit London Feb 2024 - ABK - Neo4j Product Vision and Roadmap.pptx
GraphSummit London Feb 2024 - ABK - Neo4j Product Vision and Roadmap.pptxGraphSummit London Feb 2024 - ABK - Neo4j Product Vision and Roadmap.pptx
GraphSummit London Feb 2024 - ABK - Neo4j Product Vision and Roadmap.pptxNeo4j
 
The Art of the Possible with Graph by Dr Jim Webber Neo4j.pptx
The Art of the Possible with Graph by Dr Jim Webber Neo4j.pptxThe Art of the Possible with Graph by Dr Jim Webber Neo4j.pptx
The Art of the Possible with Graph by Dr Jim Webber Neo4j.pptxNeo4j
 
KUBRICK Graphs: A journey from in vogue to success-ion
KUBRICK Graphs: A journey from in vogue to success-ionKUBRICK Graphs: A journey from in vogue to success-ion
KUBRICK Graphs: A journey from in vogue to success-ionNeo4j
 
SKY Paradigms, change and cake: the steep curve of introducing new technologies
SKY Paradigms, change and cake: the steep curve of introducing new technologiesSKY Paradigms, change and cake: the steep curve of introducing new technologies
SKY Paradigms, change and cake: the steep curve of introducing new technologiesNeo4j
 
ASTRAZENECA. Knowledge Graphs Powering a Fast-moving Global Life Sciences Org...
ASTRAZENECA. Knowledge Graphs Powering a Fast-moving Global Life Sciences Org...ASTRAZENECA. Knowledge Graphs Powering a Fast-moving Global Life Sciences Org...
ASTRAZENECA. Knowledge Graphs Powering a Fast-moving Global Life Sciences Org...Neo4j
 
IA Générative et Graphes Neo4j : RAG time !
IA Générative et Graphes Neo4j : RAG time !IA Générative et Graphes Neo4j : RAG time !
IA Générative et Graphes Neo4j : RAG time !Neo4j
 
GraphQL and Neo4j - Simple and Intelligent Modern Apps
GraphQL and Neo4j - Simple and Intelligent Modern AppsGraphQL and Neo4j - Simple and Intelligent Modern Apps
GraphQL and Neo4j - Simple and Intelligent Modern AppsNeo4j
 

More from Neo4j (20)

Neo4j Graph Summit 2024 Workshop - EMEA - Breda_and_Munchen.pdf
Neo4j Graph Summit 2024 Workshop - EMEA - Breda_and_Munchen.pdfNeo4j Graph Summit 2024 Workshop - EMEA - Breda_and_Munchen.pdf
Neo4j Graph Summit 2024 Workshop - EMEA - Breda_and_Munchen.pdf
 
Enabling GenAI Breakthroughs with Knowledge Graphs
Enabling GenAI Breakthroughs with Knowledge GraphsEnabling GenAI Breakthroughs with Knowledge Graphs
Enabling GenAI Breakthroughs with Knowledge Graphs
 
Neo4j_Anurag Tandon_Product Vision and Roadmap.Benelux.pptx.pdf
Neo4j_Anurag Tandon_Product Vision and Roadmap.Benelux.pptx.pdfNeo4j_Anurag Tandon_Product Vision and Roadmap.Benelux.pptx.pdf
Neo4j_Anurag Tandon_Product Vision and Roadmap.Benelux.pptx.pdf
 
Neo4j_Jesus Barrasa_The Art of the Possible with Graph.pptx.pdf
Neo4j_Jesus Barrasa_The Art of the Possible with Graph.pptx.pdfNeo4j_Jesus Barrasa_The Art of the Possible with Graph.pptx.pdf
Neo4j_Jesus Barrasa_The Art of the Possible with Graph.pptx.pdf
 
Swift_Maintaining Critical Standards(...).pptx.pdf
Swift_Maintaining Critical Standards(...).pptx.pdfSwift_Maintaining Critical Standards(...).pptx.pdf
Swift_Maintaining Critical Standards(...).pptx.pdf
 
Deloitte+RedCross_Talk to your data with Knowledge-enriched Generative AI.ppt...
Deloitte+RedCross_Talk to your data with Knowledge-enriched Generative AI.ppt...Deloitte+RedCross_Talk to your data with Knowledge-enriched Generative AI.ppt...
Deloitte+RedCross_Talk to your data with Knowledge-enriched Generative AI.ppt...
 
Ingka Digital_Linked Metadata by Design.pdf
Ingka Digital_Linked Metadata by Design.pdfIngka Digital_Linked Metadata by Design.pdf
Ingka Digital_Linked Metadata by Design.pdf
 
Discover Neo4j Aura_ The Future of Graph Database-as-a-Service Workshop_3.13.24
Discover Neo4j Aura_ The Future of Graph Database-as-a-Service Workshop_3.13.24Discover Neo4j Aura_ The Future of Graph Database-as-a-Service Workshop_3.13.24
Discover Neo4j Aura_ The Future of Graph Database-as-a-Service Workshop_3.13.24
 
GraphSummit Copenhagen 2024 - Neo4j Vision and Roadmap.pptx
GraphSummit Copenhagen 2024 - Neo4j Vision and Roadmap.pptxGraphSummit Copenhagen 2024 - Neo4j Vision and Roadmap.pptx
GraphSummit Copenhagen 2024 - Neo4j Vision and Roadmap.pptx
 
Emil Eifrem at GraphSummit Copenhagen 2024 - The Art of the Possible.pptx
Emil Eifrem at GraphSummit Copenhagen 2024 - The Art of the Possible.pptxEmil Eifrem at GraphSummit Copenhagen 2024 - The Art of the Possible.pptx
Emil Eifrem at GraphSummit Copenhagen 2024 - The Art of the Possible.pptx
 
Identification of insulin-resistance genes with Knowledge Graphs topology and...
Identification of insulin-resistance genes with Knowledge Graphs topology and...Identification of insulin-resistance genes with Knowledge Graphs topology and...
Identification of insulin-resistance genes with Knowledge Graphs topology and...
 
Novo Nordisk's journey in developing an open-source application on Neo4j
Novo Nordisk's journey in developing an open-source application on Neo4jNovo Nordisk's journey in developing an open-source application on Neo4j
Novo Nordisk's journey in developing an open-source application on Neo4j
 
EY: Graphs as Critical Enablers for LLM-based Assistants- the Case of Custome...
EY: Graphs as Critical Enablers for LLM-based Assistants- the Case of Custome...EY: Graphs as Critical Enablers for LLM-based Assistants- the Case of Custome...
EY: Graphs as Critical Enablers for LLM-based Assistants- the Case of Custome...
 
GraphSummit London Feb 2024 - ABK - Neo4j Product Vision and Roadmap.pptx
GraphSummit London Feb 2024 - ABK - Neo4j Product Vision and Roadmap.pptxGraphSummit London Feb 2024 - ABK - Neo4j Product Vision and Roadmap.pptx
GraphSummit London Feb 2024 - ABK - Neo4j Product Vision and Roadmap.pptx
 
The Art of the Possible with Graph by Dr Jim Webber Neo4j.pptx
The Art of the Possible with Graph by Dr Jim Webber Neo4j.pptxThe Art of the Possible with Graph by Dr Jim Webber Neo4j.pptx
The Art of the Possible with Graph by Dr Jim Webber Neo4j.pptx
 
KUBRICK Graphs: A journey from in vogue to success-ion
KUBRICK Graphs: A journey from in vogue to success-ionKUBRICK Graphs: A journey from in vogue to success-ion
KUBRICK Graphs: A journey from in vogue to success-ion
 
SKY Paradigms, change and cake: the steep curve of introducing new technologies
SKY Paradigms, change and cake: the steep curve of introducing new technologiesSKY Paradigms, change and cake: the steep curve of introducing new technologies
SKY Paradigms, change and cake: the steep curve of introducing new technologies
 
ASTRAZENECA. Knowledge Graphs Powering a Fast-moving Global Life Sciences Org...
ASTRAZENECA. Knowledge Graphs Powering a Fast-moving Global Life Sciences Org...ASTRAZENECA. Knowledge Graphs Powering a Fast-moving Global Life Sciences Org...
ASTRAZENECA. Knowledge Graphs Powering a Fast-moving Global Life Sciences Org...
 
IA Générative et Graphes Neo4j : RAG time !
IA Générative et Graphes Neo4j : RAG time !IA Générative et Graphes Neo4j : RAG time !
IA Générative et Graphes Neo4j : RAG time !
 
GraphQL and Neo4j - Simple and Intelligent Modern Apps
GraphQL and Neo4j - Simple and Intelligent Modern AppsGraphQL and Neo4j - Simple and Intelligent Modern Apps
GraphQL and Neo4j - Simple and Intelligent Modern Apps
 

Recently uploaded

Scenario Library et REX Discover industry- and role- based scenarios
Scenario Library et REX Discover industry- and role- based scenariosScenario Library et REX Discover industry- and role- based scenarios
Scenario Library et REX Discover industry- and role- based scenariosErol GIRAUDY
 
Where developers are challenged, what developers want and where DevEx is going
Where developers are challenged, what developers want and where DevEx is goingWhere developers are challenged, what developers want and where DevEx is going
Where developers are challenged, what developers want and where DevEx is goingFrancesco Corti
 
Technical SEO for Improved Accessibility WTS FEST
Technical SEO for Improved Accessibility  WTS FESTTechnical SEO for Improved Accessibility  WTS FEST
Technical SEO for Improved Accessibility WTS FESTBillieHyde
 
The Zero-ETL Approach: Enhancing Data Agility and Insight
The Zero-ETL Approach: Enhancing Data Agility and InsightThe Zero-ETL Approach: Enhancing Data Agility and Insight
The Zero-ETL Approach: Enhancing Data Agility and InsightSafe Software
 
My key hands-on projects in Quantum, and QAI
My key hands-on projects in Quantum, and QAIMy key hands-on projects in Quantum, and QAI
My key hands-on projects in Quantum, and QAIVijayananda Mohire
 
AI Workshops at Computers In Libraries 2024
AI Workshops at Computers In Libraries 2024AI Workshops at Computers In Libraries 2024
AI Workshops at Computers In Libraries 2024Brian Pichman
 
.NET 8 ChatBot with Azure OpenAI Services.pptx
.NET 8 ChatBot with Azure OpenAI Services.pptx.NET 8 ChatBot with Azure OpenAI Services.pptx
.NET 8 ChatBot with Azure OpenAI Services.pptxHansamali Gamage
 
Graphene Quantum Dots-Based Composites for Biomedical Applications
Graphene Quantum Dots-Based Composites for  Biomedical ApplicationsGraphene Quantum Dots-Based Composites for  Biomedical Applications
Graphene Quantum Dots-Based Composites for Biomedical Applicationsnooralam814309
 
Outage Analysis: March 5th/6th 2024 Meta, Comcast, and LinkedIn
Outage Analysis: March 5th/6th 2024 Meta, Comcast, and LinkedInOutage Analysis: March 5th/6th 2024 Meta, Comcast, and LinkedIn
Outage Analysis: March 5th/6th 2024 Meta, Comcast, and LinkedInThousandEyes
 
EMEA What is ThousandEyes? Webinar
EMEA What is ThousandEyes? WebinarEMEA What is ThousandEyes? Webinar
EMEA What is ThousandEyes? WebinarThousandEyes
 
LF Energy Webinar - Unveiling OpenEEMeter 4.0
LF Energy Webinar - Unveiling OpenEEMeter 4.0LF Energy Webinar - Unveiling OpenEEMeter 4.0
LF Energy Webinar - Unveiling OpenEEMeter 4.0DanBrown980551
 
Webinar: The Art of Prioritizing Your Product Roadmap by AWS Sr PM - Tech
Webinar: The Art of Prioritizing Your Product Roadmap by AWS Sr PM - TechWebinar: The Art of Prioritizing Your Product Roadmap by AWS Sr PM - Tech
Webinar: The Art of Prioritizing Your Product Roadmap by AWS Sr PM - TechProduct School
 
IT Service Management (ITSM) Best Practices for Advanced Computing
IT Service Management (ITSM) Best Practices for Advanced ComputingIT Service Management (ITSM) Best Practices for Advanced Computing
IT Service Management (ITSM) Best Practices for Advanced ComputingMAGNIntelligence
 
CyberSecurity - Computers In Libraries 2024
CyberSecurity - Computers In Libraries 2024CyberSecurity - Computers In Libraries 2024
CyberSecurity - Computers In Libraries 2024Brian Pichman
 
3 Pitfalls Everyone Should Avoid with Cloud Data
3 Pitfalls Everyone Should Avoid with Cloud Data3 Pitfalls Everyone Should Avoid with Cloud Data
3 Pitfalls Everyone Should Avoid with Cloud DataEric D. Schabell
 
How to release an Open Source Dataweave Library
How to release an Open Source Dataweave LibraryHow to release an Open Source Dataweave Library
How to release an Open Source Dataweave Libraryshyamraj55
 
UiPath Studio Web workshop series - Day 1
UiPath Studio Web workshop series  - Day 1UiPath Studio Web workshop series  - Day 1
UiPath Studio Web workshop series - Day 1DianaGray10
 
The New Cloud World Order Is FinOps (Slideshow)
The New Cloud World Order Is FinOps (Slideshow)The New Cloud World Order Is FinOps (Slideshow)
The New Cloud World Order Is FinOps (Slideshow)codyslingerland1
 
Patch notes explaining DISARM Version 1.4 update
Patch notes explaining DISARM Version 1.4 updatePatch notes explaining DISARM Version 1.4 update
Patch notes explaining DISARM Version 1.4 updateadam112203
 
UiPath Studio Web workshop series - Day 2
UiPath Studio Web workshop series - Day 2UiPath Studio Web workshop series - Day 2
UiPath Studio Web workshop series - Day 2DianaGray10
 

Recently uploaded (20)

Scenario Library et REX Discover industry- and role- based scenarios
Scenario Library et REX Discover industry- and role- based scenariosScenario Library et REX Discover industry- and role- based scenarios
Scenario Library et REX Discover industry- and role- based scenarios
 
Where developers are challenged, what developers want and where DevEx is going
Where developers are challenged, what developers want and where DevEx is goingWhere developers are challenged, what developers want and where DevEx is going
Where developers are challenged, what developers want and where DevEx is going
 
Technical SEO for Improved Accessibility WTS FEST
Technical SEO for Improved Accessibility  WTS FESTTechnical SEO for Improved Accessibility  WTS FEST
Technical SEO for Improved Accessibility WTS FEST
 
The Zero-ETL Approach: Enhancing Data Agility and Insight
The Zero-ETL Approach: Enhancing Data Agility and InsightThe Zero-ETL Approach: Enhancing Data Agility and Insight
The Zero-ETL Approach: Enhancing Data Agility and Insight
 
My key hands-on projects in Quantum, and QAI
My key hands-on projects in Quantum, and QAIMy key hands-on projects in Quantum, and QAI
My key hands-on projects in Quantum, and QAI
 
AI Workshops at Computers In Libraries 2024
AI Workshops at Computers In Libraries 2024AI Workshops at Computers In Libraries 2024
AI Workshops at Computers In Libraries 2024
 
.NET 8 ChatBot with Azure OpenAI Services.pptx
.NET 8 ChatBot with Azure OpenAI Services.pptx.NET 8 ChatBot with Azure OpenAI Services.pptx
.NET 8 ChatBot with Azure OpenAI Services.pptx
 
Graphene Quantum Dots-Based Composites for Biomedical Applications
Graphene Quantum Dots-Based Composites for  Biomedical ApplicationsGraphene Quantum Dots-Based Composites for  Biomedical Applications
Graphene Quantum Dots-Based Composites for Biomedical Applications
 
Outage Analysis: March 5th/6th 2024 Meta, Comcast, and LinkedIn
Outage Analysis: March 5th/6th 2024 Meta, Comcast, and LinkedInOutage Analysis: March 5th/6th 2024 Meta, Comcast, and LinkedIn
Outage Analysis: March 5th/6th 2024 Meta, Comcast, and LinkedIn
 
EMEA What is ThousandEyes? Webinar
EMEA What is ThousandEyes? WebinarEMEA What is ThousandEyes? Webinar
EMEA What is ThousandEyes? Webinar
 
LF Energy Webinar - Unveiling OpenEEMeter 4.0
LF Energy Webinar - Unveiling OpenEEMeter 4.0LF Energy Webinar - Unveiling OpenEEMeter 4.0
LF Energy Webinar - Unveiling OpenEEMeter 4.0
 
Webinar: The Art of Prioritizing Your Product Roadmap by AWS Sr PM - Tech
Webinar: The Art of Prioritizing Your Product Roadmap by AWS Sr PM - TechWebinar: The Art of Prioritizing Your Product Roadmap by AWS Sr PM - Tech
Webinar: The Art of Prioritizing Your Product Roadmap by AWS Sr PM - Tech
 
IT Service Management (ITSM) Best Practices for Advanced Computing
IT Service Management (ITSM) Best Practices for Advanced ComputingIT Service Management (ITSM) Best Practices for Advanced Computing
IT Service Management (ITSM) Best Practices for Advanced Computing
 
CyberSecurity - Computers In Libraries 2024
CyberSecurity - Computers In Libraries 2024CyberSecurity - Computers In Libraries 2024
CyberSecurity - Computers In Libraries 2024
 
3 Pitfalls Everyone Should Avoid with Cloud Data
3 Pitfalls Everyone Should Avoid with Cloud Data3 Pitfalls Everyone Should Avoid with Cloud Data
3 Pitfalls Everyone Should Avoid with Cloud Data
 
How to release an Open Source Dataweave Library
How to release an Open Source Dataweave LibraryHow to release an Open Source Dataweave Library
How to release an Open Source Dataweave Library
 
UiPath Studio Web workshop series - Day 1
UiPath Studio Web workshop series  - Day 1UiPath Studio Web workshop series  - Day 1
UiPath Studio Web workshop series - Day 1
 
The New Cloud World Order Is FinOps (Slideshow)
The New Cloud World Order Is FinOps (Slideshow)The New Cloud World Order Is FinOps (Slideshow)
The New Cloud World Order Is FinOps (Slideshow)
 
Patch notes explaining DISARM Version 1.4 update
Patch notes explaining DISARM Version 1.4 updatePatch notes explaining DISARM Version 1.4 update
Patch notes explaining DISARM Version 1.4 update
 
UiPath Studio Web workshop series - Day 2
UiPath Studio Web workshop series - Day 2UiPath Studio Web workshop series - Day 2
UiPath Studio Web workshop series - Day 2
 

GraphConnect 2014 SF: Neo4j at Scale using Enterprise Integration Patterns

  • 1. Graph at Scale 10/22/14 Brad Nussbaum CTO | MediaHound brad@mediahound.com | @bradnussbaum www.mediahound.com
  • 3. But Entertainment is more than just Film & TV
  • 4. Entertainment Preferences Are Scattered Music Movies Television Books
  • 5. All Content one platform that connects: All Sources All Devices All Brands All Artists All Users
  • 6. The Entertainment Graph A comprehensive database that brings together movies, books, games, music, and TV, including the cast & crew, sources, reviews, categories, genres, lists and more! The Entertainment Graph powers meaningful recommendations, exciting data insights and comprehensive social discovery.
  • 7. Use Cases: Academy Award Winners On Netflix MATCH (c:Collection)-[:CONTAINS]-(m:Movie) WHERE (m)- [:MATCHED_SOURCE]-(n:NETFLIX) AND c.name=“Academy Award Winners” RETURN m; Movies and Shows Based On Zombie Books MATCH (m:Media)-[:BASED_ON]-(b:Book)-[:HAS_TRAIT]-(z:Trait) WHERE (m.type=“Movie” OR m.type=“Show”) AND z.name=“Zombies” RETURN m; To access ‘The Movie Graph’ mini app (which uses a different model than above), from your browser, run :play movie graph
  • 8. Entertainment Recommendations Collaborative Filtering: If Joe, Amy and Steve like Gladiator, AND Joe and Amy like Toy Story, THEN MediaHound recommends Toy Story to Steve. Graph Influencers Joe is an early adopter. Joe, Amy and Steve like several things in common. MediaHound recommends Amy and Steve follow Joe and Amy Joe discovers the next big hit and shares it on his feed Amy and Steve see Steve’s post and give it a listen.
  • 9. We needed our graph database to perform under sustained user write load AND during heavy batch update operations. We needed to recommend media content in real-time which required many concurrent pattern matching operations on the graph.
  • 10. We realized through trial and error that using the Transactional Cypher HTTP Endpoint was the BEST solution to control batch writes. POST http://localhost:7474/db/data/transaction/commit Accept: application/json; charset=UTF-8 Content-Type: application/json { "statements": [ { "statement": "MERGE(n:User{username:"bradnussbaum"})-[:FOLLOWS]->(m:User{username:"bennussbaum"});" }, { "statement": "MERGE(n:User{username:"bradnussbaum"})<-[:FOLLOWS]-(m:User{username:"bennussbaum"});" } ] }
  • 11. We ran tests using the low-level kernel and found that sustained transaction writes performed optimally between 400-2,000 nodes and relationships per transaction. Compare the difference between… • Writing a single relationship (33 bytes) per transaction f or 10k iterations compared to • Writing 1k relationships per transaction for 10 iterations. **As of 2.2, Neo4j will batch writes on the server http://neo4j.com/docs/stable/linux-performance-guide.html git clone git@github.com:neo4j-contrib/tooling.git
  • 12. We used Enterprise Integration Patterns (EIP) to create optimal batch sizes for each transaction. • Splitters to break down larger messages • Aggregators to combine single CQL statements t ogether into a single batch transaction • Throttling to control concurrent requests and requests p er second
  • 13. We frequently run 40+ concurrent write transactions to a 3 instance cluster for hours at time. Deadlocks can occur often with many concurrent write o perations. • Retry Transient Errors after a small period of time. • Use the Error Index to split failed TX statements. Read here to learn all the error status codes, seriously. http://neo4j.com/docs/stable/status-codes.html
  • 14. Test write throughput on your cluster with the push factor you plan to use in production and intentionally kill your master under load. You need to have two load balancers: • One for all the instances you want performing reads • One for your master ONLY  (send writes here) Master Check: /db/manage/server/ha/master - Returns true|false Slave Check: /db/manage/server/ha/slave - Returns true|false Available Check: /db/manage/server/ha/available - Returns master|slave
  • 15. Check your driver for transaction support. Embedded mode has full transaction support but most remote drivers do not at this time. This will be changing in the near future…depending on which driver you use. **Spring Data Neo4j is actively being developed to included these features as part of 2.2.
  • 16. Graph at Scale 10/22/14 Ben Nussbaum Director of Engineering | MediaHound ben@mediahound.com | @bennussbaum www.mediahound.com
  • 17. We built custom algorithms that needed run-time decision making as Neo4j Extensions with Spring Data Neo4j. • Cache abstraction with Google’s Guava to build large in-memory indexes of nodes and relationships. • Integration for jobs instructions and results to and from the broker. • Async for batch job processing. https://github.com/AtomRain/neo4j-extensions
  • 18. We took advantage of spot processing from AWS to run our custom extension algoritms. • On-demand graph processing with as many instances at a time as needed (we have used up to 9). • Concurrent job operations per spot. • Cache optimizations based on Labels and context of the jobs.
  • 19. We built a flexible job controller that enables concurrent job processing on spot instances • Large jobs are broken into smaller jobs that can be processed by a single spot instance. • Spots process unit jobs and return results. If a spot dies, the job stays in the queue and another spot picks it up. • Memory and CPU constraints on an instance make this a necessity, especially when processing 30M+ songs.
  • 20. Spot instances run Neo4j in SINGLE mode and stay up to date using a Topic. Neo4j HA Spot 1 SINGLE Job Result MQ Spot 2 SINGLE Job Process MQ ESB SDN SDN TX Topic 1 2 3 4 5 1. ESB sends Jobs to MQ 2. Spots consume job instructions, process and send results b ack to MQ 3. ESB posts jobs results to HA 4. On successful post, send u pdates to Topic 5. Spots consume from Topic to s tay u to date
  • 21. Batch jobs return thousands of CQL statements which must not be dependent on any statements before or after. • Compound statements to create nodes and relationships for specific sub-graphs to avoid the need for layering wherever possible. If not… • Run jobs in a linear phases (layering) to create nodes first then connect relationships
  • 22. Graph at Scale 10/22/14 Q/A