GraphConnect Europe 2016 - Enterprise Data Integration with a new JDBC Driver for Neo4j 3.0 - Lorenzo Speranzoni
1. / 50
LONDON, 26 April 2016
1
LORENZO SPERANZONI | @inserpio
CEO @ LARUS Business Automation - GCE2016 BRONZE SPONSOR
GraphConnect Europe 2016
Explore the new
COUCHBASE and JDBC
integrations for Neo4j 3.0
2. / 50
LONDON, 26 April 2016GraphConnect Europe 2016
EXPLORE THE NEW COUCHBASE & JDBC
INTEGRATIONS FOR NEO4J 3.0
AGENDA
✦ Let’s say Hi ( “Ciao” !! )
✦ Neo4j - Couchbase Connector
✦ JDBC Driver with BOLT for Neo4j 3.0
2
3. / 50
LONDON, 26 April 2016GraphConnect Europe 2016
EXPLORE THE NEW COUCHBASE & JDBC
INTEGRATIONS FOR NEO4J 3.0
AGENDA
✦ Let’s say Hi ( “Ciao” !! )
✦ Neo4j - Couchbase Connector
✦ JDBC Driver with BOLT for Neo4j 3.0
3
4. / 50
LONDON, 26 April 2016GraphConnect Europe 2016
EXPLORE THE NEW COUCHBASE & JDBC
INTEGRATIONS FOR NEO4J 3.0
WELCOME
AND
THANKS FOR COMING
TO THIS LIGHTNING TALK
4
5. / 50
LONDON, 26 April 2016GraphConnect Europe 2016
EXPLORE THE NEW COUCHBASE & JDBC
INTEGRATIONS FOR NEO4J 3.0
5
[:WORKS_FOR { role: ‘CEO’ }]
Venice
[:BRONZE_SPONSOR]
Lorenzo Speranzoni
LARUS Business Automation
[:BASED_IN]
[:CERTIFIED_ON]
[:PARTNER_OF]
6. / 50
LONDON, 26 April 2016GraphConnect Europe 2016
EXPLORE THE NEW COUCHBASE & JDBC
INTEGRATIONS FOR NEO4J 3.0
6
… this is a typical day in Venice George Clooney Wedding
BY THE WAY … … NOT THIS ONE DURING …
#*@!#!!
7. / 50
LONDON, 26 April 2016GraphConnect Europe 2016
EXPLORE THE NEW COUCHBASE & JDBC
INTEGRATIONS FOR NEO4J 3.0
7
2016
Neo4j Contributors
20152011
First Spikes
( with neo4j 1.x !! )
2013
LARUS’ TRIP WITH NEO4J (quite a long and cool story)
8. / 50
LONDON, 26 April 2016GraphConnect Europe 2016
EXPLORE THE NEW COUCHBASE & JDBC
INTEGRATIONS FOR NEO4J 3.0
8
CONSULTING
[:DELIVERS]
TRAININGS
SUPPORT
[:DELIVERS]
[:DELIVERS] [:FOR]
[:ON]
[:ON]
[:ADOPTED]
9. / 50
LONDON, 26 April 2016GraphConnect Europe 2016
EXPLORE THE NEW COUCHBASE & JDBC
INTEGRATIONS FOR NEO4J 3.0
AGENDA
✦ Let’s say Hi ( “Ciao” !! )
✦ Neo4j - Couchbase Connector
✦ JDBC Driver with BOLT for Neo4j 3.0
9
10. / 50
LONDON, 26 April 2016GraphConnect Europe 2016
EXPLORE THE NEW COUCHBASE & JDBC
INTEGRATIONS FOR NEO4J 3.0
The new Neo4j - Couchbase Connector
10
11. / 50
LONDON, 26 April 2016GraphConnect Europe 2016
EXPLORE THE NEW COUCHBASE & JDBC
INTEGRATIONS FOR NEO4J 3.0
KEY ASPECTS
✦ BI-DIRECTIONAL
✦ CONFIGURABLE
11
12. / 50
LONDON, 26 April 2016GraphConnect Europe 2016
EXPLORE THE NEW COUCHBASE & JDBC
INTEGRATIONS FOR NEO4J 3.0
https://developer.spotify.com/web-api/console/get-album
12
CASESTUDY
13. / 50
LONDON, 26 April 2016GraphConnect Europe 2016
EXPLORE THE NEW COUCHBASE & JDBC
INTEGRATIONS FOR NEO4J 3.0
https://developer.spotify.com/web-api/console/get-album [ RESPONSE ]
13
CASESTUDY
14. / 50
LONDON, 26 April 2016GraphConnect Europe 2016
EXPLORE THE NEW COUCHBASE & JDBC
INTEGRATIONS FOR NEO4J 3.0
BASIC JSON 2 GRAPH CONVERSION RULES
PLEASE NOTE.
✦ Most of the following approaches come from William Lion’s (@lionwj)
MongoDB - Neo4j Connector (http://neo4j.com/blog/neo4j-doc-manager-polyglot-
persistence-mongodb/)
✦ Don’t miss William’s session “NoSQL Polyglot Persistence: Tools and Integrations” here at
GCE2016 (4:30PM), for many more examples about managing multiple databases.
14
15. / 50
LONDON, 26 April 2016GraphConnect Europe 2016
EXPLORE THE NEW COUCHBASE & JDBC
INTEGRATIONS FOR NEO4J 3.0
PRIMITIVE ATTRIBUTES …
15
album_type: “U2”
href: “https://api.spotify.com/v1/albums/586Z…”
id: “586ZRfgsIckfcKvHVcGM4V”
name: “The Joshua Tree”
popularity: 53
release_date: “1987-03-10”
release_date_precision: “day”
type: “album”
uri: “spotify:album:586ZRfgsIckfcKvHVcGM4V”
ALBUM
BECOME NODE’S ATTRIBUTES
16. / 50
LONDON, 26 April 2016GraphConnect Europe 2016
EXPLORE THE NEW COUCHBASE & JDBC
INTEGRATIONS FOR NEO4J 3.0
ARRAY OF PRIMITIVE ATTRIBUTES …
16
album_type: “U2”
available_markets: [ “MX”, “US” ]
href: “https://api.spotify.com/v1/albums/586Z…”
id: “586ZRfgsIckfcKvHVcGM4V”
name: “The Joshua Tree”
popularity: 53
release_date: “1987-03-10”
release_date_precision: “day”
type: “album”
uri: “spotify:album:586ZRfgsIckfcKvHVcGM4V”
ALBUM
STILL BECOME NODE’S ATTRIBUTES
17. / 50
LONDON, 26 April 2016GraphConnect Europe 2016
EXPLORE THE NEW COUCHBASE & JDBC
INTEGRATIONS FOR NEO4J 3.0
OBJECT ATTRIBUTES . . .
17
href: “https://api.spotify.com/v1/artists/51B…”
id: “51Blml2LZPmy7TTiAg47vQ”
name: “U2”
type: “artist”
uri: “spotify:album:51Blml2LZPmy7TTiAg47vQ”
ARTIST
BECOME CONNECTED NODES
name: “The Joshua Tree”
ALBUM
[ :ALBUM_ARTIST ]
18. / 50
LONDON, 26 April 2016GraphConnect Europe 2016
EXPLORE THE NEW COUCHBASE & JDBC
INTEGRATIONS FOR NEO4J 3.0
ARRAY OF OBJECTS ATTRIBUTES …
18
ARTIST
AGAIN BECOME CONNECTED NODES
ALBUM
disc_number: 1
duration_ms: 337973
href: “https://api.spotify.com/v1/tracks/3zq…”
id: “3zqTlJtp7KTPrNUKNahvAo”
name: “Where The Streets Have No Name”
uri: “spotify:album:3zqTlJtp7KTPrNUKNahvAo”
track_number: 1
type: “track”
ITEM
TRACKS
ITEM
2
ITEM
3ITEM
11
ITEM
4
[:TRACKS_ITEM
S]
19. / 50
LONDON, 26 April 2016GraphConnect Europe 2016
EXPLORE THE NEW COUCHBASE & JDBC
INTEGRATIONS FOR NEO4J 3.0
Neo4j - Couchbase Connector
COMPONENTS
(and a bit of noisy code)
19
20. / 50
LONDON, 26 April 2016GraphConnect Europe 2016
EXPLORE THE NEW COUCHBASE & JDBC
INTEGRATIONS FOR NEO4J 3.0
Neo4j - Couchbase Connector Components
20
21. / 50
LONDON, 26 April 2016GraphConnect Europe 2016
EXPLORE THE NEW COUCHBASE & JDBC
INTEGRATIONS FOR NEO4J 3.0
1. Couchbase Mutation Listener
public class CouchbaseEvent {
/**
* Wrapped {@link CouchbaseMessage}.
*/
private CouchbaseMessage message;
}
public class CouchbaseEventFilter {
/**
* Returns true if event is a {@link MutationMessage}.
*/
public static boolean accept(final CouchbaseEvent couchbaseEvent) {
return couchbaseEvent.getMessage() instanceof MutationMessage;
}
}
21
22. / 50
LONDON, 26 April 2016GraphConnect Europe 2016
EXPLORE THE NEW COUCHBASE & JDBC
INTEGRATIONS FOR NEO4J 3.0
1. Couchbase Mutation Listener
public class CouchbaseEventHandler implements EventHandler<CouchbaseEvent> {
/**
* Handles {@link CouchbaseEvent}s that come into the response RingBuffer.
*/
@Override
public void onEvent(final CouchbaseEvent event, final long sequence, final boolean endOfBatch) throws Exception {
if (CouchbaseEventFilter.accept(event)) {
MutationMessage mutationMessage = (MutationMessage) event.getMessage();
ByteBuf jsonDocument = mutationMessage.content();
DocumentTransformer<String> remoteNeo4jTransformer = new RemoteNeo4jTransformer();
remoteNeo4jTransformer.transform(mutationMessage.key(), “COUCHBASE_DOCUMENT", asString(jsonDocument));
}
}
}
22
23. / 50
LONDON, 26 April 2016GraphConnect Europe 2016
EXPLORE THE NEW COUCHBASE & JDBC
INTEGRATIONS FOR NEO4J 3.0
1. Couchbase Mutation Listener
public class CouchbaseMutationListener extends Thread {
/**
* Builds and starts the event consumer.
*/
public void buildDisruptor() {
Disruptor<CouchbaseEvent> disruptor = new Disruptor<CouchbaseEvent>(new CouchbaseEventFactory(), 32, Executors.newCachedThreadPool());
disruptor.handleExceptionsWith(new CouchbaseExceptionLogger());
disruptor.handleEventsWith(new CouchbaseEventHandler());
this.couchbaseEventBuffer = disruptor.start();
}
/**
* Connect this dispatcher to Couchbase.
*/
public void connect() {
this.couchbaseClusterFacade.send(new SeedNodesRequest(couchbaseNodes)).timeout(2, TimeUnit.SECONDS).toBlocking().single();
this.couchbaseClusterFacade.send(new OpenBucketRequest(this.couchbaseBucket, couchbasePassword)).timeout(2, TimeUnit.SECONDS).toBlocking().single();
}
}
23
24. / 50
LONDON, 26 April 2016GraphConnect Europe 2016
EXPLORE THE NEW COUCHBASE & JDBC
INTEGRATIONS FOR NEO4J 3.0
2. Neo4j JSON Loader
public class JsonDocument {
private String id;
private String type;
private String content;
private JsonMappingStrategy mappingStrategy;
private List<JsonObjectDescriptor> objectDescriptors;
}
public enum JsonMappingStrategy {
ATTRIBUTE_BASED(“attribute”),
DOMAIN_DRIVEN(“domain”),
STORED_PROCEDURE(“stored procedure”),
}
24
public class JsonObjectDescriptor {
private String entityName;
private List<String> uniqueKeyAttributes;
private String typeAttribute;
}
25. / 50
LONDON, 26 April 2016GraphConnect Europe 2016
EXPLORE THE NEW COUCHBASE & JDBC
INTEGRATIONS FOR NEO4J 3.0
2. Neo4j JSON Loader
@Path("/")
public class JsonLoaderRestController {
private final GraphDatabaseService graphDatabaseService;
public JsonLoaderRestController(@Context GraphDatabaseService graphDatabaseService) {
this.graphDatabaseService = graphDatabaseService;
}
@POST
@Produces({ MediaType.APPLICATION_JSON, MediaType.APPLICATION_XML })
public Response loadJSON(JAXBElement<JsonDocument> jsonDocumentWrapper) {
JsonDocument jsonDocument = jsonDocumentWrapper.getValue();
JsonLoaderService loaderService = new DefaultJsonLoaderService(this.graphDatabaseService);
QueryStatistics queryStatistics = loaderService.save(jsonDocument);
return Response.ok(queryStatistics).build();
}
}
neo4j-server.properties
org.neo4j.server.thirdparty_jaxrs_classes=it.larusba.integration.neo4j.jsonloader.rest=/jsonloader
25
26. / 50
LONDON, 26 April 2016GraphConnect Europe 2016
EXPLORE THE NEW COUCHBASE & JDBC
INTEGRATIONS FOR NEO4J 3.0
2. Neo4j JSON Loader
public class DefaultJsonLoaderService implements JsonLoaderService {
@Override
public JsonLoaderStatistics save(JsonDocument jsonDocument) throws JsonParseException, JsonMappingException, IOException {
JsonTransformer<String> jsonTransformer = JsonTransformerFactory.getInstance(jsonDocument.getMappingStrategy());
String cypher = jsonTransformer.transform(jsonDocument);
try (Transaction tx = this.graphDatabaseService.beginTx()) {
Result result = this.graphDatabaseService.execute(cypher);
jsonLoaderStatistics = new JsonLoaderStatistics(result.getQueryStatistics());
tx.success();
}
return jsonLoaderStatistics;
}
26
27. / 50
LONDON, 26 April 2016GraphConnect Europe 2016
EXPLORE THE NEW COUCHBASE & JDBC
INTEGRATIONS FOR NEO4J 3.0
3. Neo4j Mutation Listener
public class Neo4jEventListener implements TransactionEventHandler<Void> {
private GraphDatabaseService db;
public Neo4jEventListener(GraphDatabaseService db) {
this.db = db;
}
public void afterCommit(TransactionData data, Void state) {
List<JsonDocument> jsonDocuments = buildJSONDocument(data);
if (CollectionUtils.isNotEmpty(jsonDocuments)) {
CouchbaseJSONLoader.upsert(jsonDocuments);
}
}
}
27
28. / 50
LONDON, 26 April 2016GraphConnect Europe 2016
EXPLORE THE NEW COUCHBASE & JDBC
INTEGRATIONS FOR NEO4J 3.0
4. Couchbase JSON Loader
public class CouchbaseJSONLoader
public static void upsert(List<JsonDocument> jsonDocuments) {
Cluster cluster = CouchbaseCluster.create();
Bucket defaultBucket = cluster.openBucket();
for (JsonDocument jsonDocument : jsonDocuments) {
defaultBucket.upsert(jsonDocument);
}
cluster.disconnect();
}
}
28
29. / 50
LONDON, 26 April 2016GraphConnect Europe 2016
EXPLORE THE NEW COUCHBASE & JDBC
INTEGRATIONS FOR NEO4J 3.0
Neo4j - Couchbase Connector
HOW ABOUT THE DOMAIN?
WOULDN’T BE COOL TO BE ABLE TO TRAIN
THE “JSON LOADER” WITH THE BUSINESS DOMAIN
YOUR WORKING ON?
29
30. / 50
LONDON, 26 April 2016GraphConnect Europe 2016
EXPLORE THE NEW COUCHBASE & JDBC
INTEGRATIONS FOR NEO4J 3.0
HOW ABOUT THESE INTERESTING ATTRIBUTES ?
30
31. / 50
LONDON, 26 April 2016GraphConnect Europe 2016
EXPLORE THE NEW COUCHBASE & JDBC
INTEGRATIONS FOR NEO4J 3.0
31
WHY DON’T WE USE “ID” ATTRIBUTES TO UNIQUELY IDENTIFY NODES?
We could use them on MERGE statements:
MERGE (album:ALBUM {id: ‘586ZRfgsIckfcKvHVcGM4V’})
MERGE (track:TRACK {id: ‘3zqTlJtp7KTPrNUKNahvAo’})
32. / 50
LONDON, 26 April 2016GraphConnect Europe 2016
EXPLORE THE NEW COUCHBASE & JDBC
INTEGRATIONS FOR NEO4J 3.0
32
AND WHY DON’T WE USE “TYPE” ATTRIBUTES AS NODES’ LABELS?
MERGE (album:ALBUM {id: ‘586ZRfgsIckfcKvHVcGM4V’})
MERGE (track:TRACK {id: ‘3zqTlJtp7KTPrNUKNahvAo’})
33. / 50
LONDON, 26 April 2016GraphConnect Europe 2016
EXPLORE THE NEW COUCHBASE & JDBC
INTEGRATIONS FOR NEO4J 3.0
DESCRIBE YOUR DOMAIN
i.e. instruct how the JSON 2 GRAPH transformer should behave
List<JsonObjectDescriptor> jsonObjectDescriptors = new ArrayList<JsonObjectDescriptor>();
jsonObjectDescriptors.add(new JsonObjectDescriptor("Album", Arrays.asList("id"), “type”));
jsonObjectDescriptors.add(new JsonObjectDescriptor("Artists", Arrays.asList("id"), “type"));
jsonObjectDescriptors.add(new JsonObjectDescriptor("Images", Arrays.asList("url"), null));
jsonObjectDescriptors.add(new JsonObjectDescriptor("Tracks", Arrays.asList("href"), null));
jsonObjectDescriptors.add(new JsonObjectDescriptor("Items", Arrays.asList("id"), “type”));
JsonDocument jsonDocument =
new JsonDocument("U2TheJoshuaTreeAlbum", "Album", spotifyTheJoshuaTreeAlbumDocument, JsonMappingStrategy.DOMAIN_DRIVEN, jsonObjectDescriptors);
33
key type json strategy descriptors
34. / 50
LONDON, 26 April 2016GraphConnect Europe 2016
EXPLORE THE NEW COUCHBASE & JDBC
INTEGRATIONS FOR NEO4J 3.0
DOMAIN DRIVER JSON TRANSFORMER
public class DomainDrivenJsonTransformer implements JsonTransformer<String> {
@Override
public String transform(JsonDocument jsonDocument) throws JsonParseException, JsonMappingException, IOException {
34
35. / 50
LONDON, 26 April 2016GraphConnect Europe 2016
EXPLORE THE NEW COUCHBASE & JDBC
INTEGRATIONS FOR NEO4J 3.0
35
36. / 50
LONDON, 26 April 2016GraphConnect Europe 2016
EXPLORE THE NEW COUCHBASE & JDBC
INTEGRATIONS FOR NEO4J 3.0
36
“DOMAIN DRIVEN” TRANSFORMER
ALLOWS YOU TO RE-USE
“U2” and all other NODES
…
SO THAT YOU GET
A REAL BENEFIT
FROM RELATIONSHIPS
37. / 50
LONDON, 26 April 2016GraphConnect Europe 2016
EXPLORE THE NEW COUCHBASE & JDBC
INTEGRATIONS FOR NEO4J 3.0
AGENDA
✦ Let’s say Hi ( “Ciao” !! )
✦ Neo4j - Couchbase Connector
✦ JDBC Driver with BOLT for Neo4j 3.0
37
38. / 50
LONDON, 26 April 2016GraphConnect Europe 2016
EXPLORE THE NEW COUCHBASE & JDBC
INTEGRATIONS FOR NEO4J 3.0
38
The new JDBC Driver with BOLT
Neo4j 3.0
APPLICATION
JDBC
EMIL’S OPENING KEYNOTE
39. / 50
LONDON, 26 April 2016GraphConnect Europe 2016
EXPLORE THE NEW COUCHBASE & JDBC
INTEGRATIONS FOR NEO4J 3.0
Neo4j JDBC Driver: A-TEAM
39
Project Leader
HTTP Protocol
Tester
BENOIT SIMARD
STEFAN ARMBRUSTER
MICHAEL HUNGER
Dev & Project Leader
RALF BECHER
40. / 50
LONDON, 26 April 2016GraphConnect Europe 2016
EXPLORE THE NEW COUCHBASE & JDBC
INTEGRATIONS FOR NEO4J 3.0
KEY ASPECTS
✦ MAKE USE OF THE NEW B.O.L.T. PROTOCOL
✦ MODULAR DESING (bolt, http, in-memory, file)
✦ HEAVILY TESTED
✦ FOCUS ON PERFORMANCE
40
41. / 50
LONDON, 26 April 2016GraphConnect Europe 2016
EXPLORE THE NEW COUCHBASE & JDBC
INTEGRATIONS FOR NEO4J 3.0
41
1. Make use of the new BOLT protocol
Neo4j 3.0
NEO4J-JDBC
BOLT module
neo4j-java-driver
APPLICATION
https://github.com/neo4j/neo4j-java-driver
42. / 50
LONDON, 26 April 2016GraphConnect Europe 2016
EXPLORE THE NEW COUCHBASE & JDBC
INTEGRATIONS FOR NEO4J 3.0
42
2. Modular design
Neo4j 3.0
NEO4J-JDBC BOLT
HTTP
FILE
IN-MEMORY
43. / 50
LONDON, 26 April 2016GraphConnect Europe 2016
EXPLORE THE NEW COUCHBASE & JDBC
INTEGRATIONS FOR NEO4J 3.0
3. Heavily tested
✦ ~300 unit tests
✦ ~50 integration tests
43
for ~60 classes
44. / 50
LONDON, 26 April 2016GraphConnect Europe 2016
EXPLORE THE NEW COUCHBASE & JDBC
INTEGRATIONS FOR NEO4J 3.0
4. Focus on performance
✦ OLD: 30’20’’
✦ NEW: 17’11’’
44
30.000 CSV rows
WOW!
ALMOST HALF TIME
45. / 50
LONDON, 26 April 2016GraphConnect Europe 2016
EXPLORE THE NEW COUCHBASE & JDBC
INTEGRATIONS FOR NEO4J 3.0
45
INTEGRATE NEO4J WITH ALL
YOUR FAVOURITE TOOLS AND PRODUCTS
Neo4j 3.0
46. / 50
LONDON, 26 April 2016GraphConnect Europe 2016
EXPLORE THE NEW COUCHBASE & JDBC
INTEGRATIONS FOR NEO4J 3.0
Minimum Viable Snippet
// Make sure Neo4j Driver is registered
Class.forName(“it.larusba.neo4j.jdbc.bolt.BoltDriver”);
// Connect
Connection con = DriverManager.getConnection(“jdbc:bolt://localhost”);
// Query
try (Statement stmt = con.createStatement()) {
ResultSet rs = stmt.executeQuery(“MATCH (n:User) RETURN n.name”);
while (rs.next()) {
System.out.println(rs.getString(“n.name”));
}
}
con.close();
46
47. / 50
LONDON, 26 April 2016GraphConnect Europe 2016
EXPLORE THE NEW COUCHBASE & JDBC
INTEGRATIONS FOR NEO4J 3.0
47
RC1
Neo4j Contrib
M5M3 M4
Neo4j 3.0 JDBC Driver RELEASE PLAN
HTTP PROTOCOL
java.sql.CallableStatement
FULLY RETRO-
COMPATIBILITY
BOLT
PROTOCOL
early autumn
PRODUCTION
READY
TODAY AT
GCE2016
early summer
48. / 50
LONDON, 26 April 2016GraphConnect Europe 2016
EXPLORE THE NEW COUCHBASE & JDBC
INTEGRATIONS FOR NEO4J 3.0
HAVE FUN WITH US
✦ https://github.com/larusba/neo4j-jdbc
✦ https://github.com/larusba/neo4j-couchbase-connector
✦ https://github.com/larusba/neo4j-json-loader
48
49. / 50
LONDON, 26 April 2016GraphConnect Europe 2016
EXPLORE THE NEW COUCHBASE & JDBC
INTEGRATIONS FOR NEO4J 3.0
COME AND TALK WITH THESE GUYS!!!
THEY DID ALL THIS STUFF &
THEY’RE IN THE ROOM :-)
✦ Mauro Roiter (@MauroRoiter)
✦ Marco Falcier (@mfalcier)
✦ Alberto D’Este (@ziotobia)
✦ Gianmarco Laggia (@glaggia)
49
50. / 50
LONDON, 26 April 2016
50
LORENZO SPERANZONI | @inserpio
CEO @ LARUS Business Automation - GCE2016 BRONZE SPONSOR
GraphConnect Europe 2016
THANK YOU