NEW LAUNCH! How to build graph applications with SPARQL and Gremlin using Amazon Neptune - DAT342 - re:Invent 2017

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS re:INVENT
Hands on with Amazon Neptune
M i c h a e l S c h m i d t ( S r S o f t w a r e D e v e l o p m e n t E n g i n e e r , A m a z o n N e p t u n e )
D i v i j V a i d y a ( S o f t w a r e D e v e l o p m e n t E n g i n e e r , A m a z o n N e p t u n e )
N o v e m b e r 2 0 1 7
D A T 3 4 2

TUTORIAL OUTLINE
• Graph Databases
• Graph: Concept & Use Cases
• Hands on: Exploring a Data Graph
• Introduction to Amazon Neptune: Architecture & Feature Overview
• Amazon Neptune SPARQL
• Introduction to RDF & SPARQL
• Hands on: Building Applications on RDF Graphs using SPARQL
• Amazon Neptune Gremlin
• Introduction to Property Graphs & Gremlin
• Hands on: Querying and Manipulating Property Graphs using Gremlin
• Q&A

HIGHLY CONNECTED DATA
Retail Fraud DetectionRestaurant RecommendationsSocial Networks

USE C ASES FOR HI GHLY C ONNEC T ED DAT A
Social Networking
Life Sciences Network & IT OperationsFraud Detection
Recommendations Knowledge Graphs

A G RAPH DATABASE IS OPTIMIZ E D F OR E F F ICIE NT
STORAG E AND RE TRIE VAL OF HIG HL Y CONNE CTE D DATA

EXPLORING A DATA GRAPH
1. Connect to your instance via SSH,
following the handout guidelines
2. Open http://localhost:2345/ in your
browser and log in
3. Click the “Start Tutorial” link on the
start page
4. Familiarize yourself with the data
by following the instructions in
Section 1, 1.1 EXERCISE –
Exploring the data graph (~10min)

Open Source Apache TinkerPop
Gremlin Traversal Language
W3C Standard
SPARQL Query Language
R E S O U R C E D E S C R I P T I O N
F R A M E W O R K ( R D F )
P R O P E R T Y G R A P H
LEADING GRAPH MODELS AND FRAMEWORKS

AMAZON NEPTUNE
F u l l y m a n a g e d g r a p h d a t a b a s e
FAST RELIABLE OPEN
Query billions of
relationships with
millisecond latency
6 replicas of your data
across 3 AZs with full
backup and restore
Build powerful
queries easily with
Gremlin and SPARQL
Supports Apache
TinkerPop & W3C
RDF graph models
EASY

AMAZON NEPTUNE HIGH LEVEL ARCHITECTURE

AMAZON NEPTUNE PARTNERS AND ECOSYSTEM
. . . and more.
Clients
compatible with
the open
source Apache
TinkerPop
project.
Clients
compatible with
the W3C
Semantic Web
RDF and
SPARQL
standards.

Amazon Neptune SPARQL

RDF GRAPHS

URIs AS GLOBALLY UNIQUE IDENTIFIERS
URIs to identify nodes and edge labels
<https://permid.org/1-4295902158>
=> identifies the company “Netflix Inc”
organization:isIncorporatedIn1
=> identifies the relationship “is incorporated in”
<http://sws.geonames.org/6252001/>
=> identifies country “USA”
1 This is a shortcut for
<http://permid.org/ontology/organization/isIncorporatedIn>.
RDF uses XML prefix notation, where the prefix organization is a
shortcut for <http://permid.org/ontology/organization/>.

RDF GRAPH: COLLECTION OF TRIPLES (1)
# Every edge in the RDF graph is represented as
# a (subject, predicate, object) triple
organization:isIncorporatedIn
<http://sws.geonames.org/6252001/> .
subject
predicate
object

vcard:organization-name
"Netflix Inc" .
subject
predicate
object
(URI)
subject
predicate
object
(literal)
Literals are “sinks” in the graph. They do not
have any outgoing edges.
RDF supports strings and other XML datatypes
(bool, integer, dates, floats, doubles, …)

vcard:organization-name
"Netflix Inc" .
organization:hasRegisteredPhoneNumber
"13026587581" .
<http://sws.geonames.org/6252001/>
iso:countryCode
”US” .
Same URI used as both subject and object, depending on
whether we represent outgoing vs. incoming RDF edges.

THE BENEFIT OF URIs: LINKED DATA
Linking across datasets by referencing globally unique URIs
GeoNames
Wikidata
PermID
Example: PermID (re)uses <http://sws.geonames.org/6252001/>
as a global Identifier for the USA, which is an identifier rooted in GeoNames.

THE LINKED OPEN DATA CLOUD
Linking Open Data cloud diagram 2017, by Andrejs Abele, John P. McCrae, Paul Buitelaar,
Anja Jentzsch and Richard Cyganiak. http://lod-cloud.net/ (CC-BY-SA)
Example: SNOMED CT (Systematized Nomenclature of Medicine –Clinical Terms)

RDF: A GLIMPSE BEYOND
• Predefined vocabulary URIs with fixed semantics
• rdf:type, rdfs:label, rdfs:Class, rdfs:subClassOf, rdfs:domain, rdfs:range, ...
• Domain-specific vocabularies (e.g. vCard, FOAF, DC, SKOS, etc.)
• Built-in support for advanced modelling
• Lists & other containers
• From triples to quads
• Support for “named graphs” to group triples in distinguished, queryable subgraphs
• Helpful to manage logically separated (fragments of) data sets
• Various standardized serialization formats
• NTriples/NQuads, RDF/XML, Turtle

QUERYING RDF USING SPARQL (1)
?name
Variables are
prefixed with “?”.

?property
?property
?property
?node ?node
?node

?org
?phone

?org
?countryCode
?org ?country

Data from Wikidata
THE POWER OF LINKED DATA
Data from PermID
Data from GeoNames

EXTRACTING & TRANSFORMING GRAPHS

DATA MANIPULATION VIA SPARQL UPDATE
"1997-08-29"^^xsd:date
vcard:bday

SPARQL: A GLIMPSE BEYOND
• Query Language (https://www.w3.org/TR/sparql11-query/)
• Operators: “.” (aka join), OPTIONAL, UNION, MINUS, FILTER, ...
• Aggregations: GROUP BY, HAVING, SUM, MIN, ...
• Property paths for graph traversal
• Solution modifiers: LIMIT, OFFSET, ORDER BY
• SPARQL 1.1 Protocol (https://www.w3.org/TR/sparql11-protocol/)
• Standardized, HTTP based API Endpoint for querying & manipulation
• Supports different output formats via Content Negotiation

INTERACTING WITH AMAZON NEPTUNE SPARQL
• HTTP Endpoint based on SPARQL 1.1 Protocol
• W3C Standard:
https://www.w3.org/TR/sparql11-protocol/
• Use HTTP clients (curl, wget)
• Dedicated SPARQL consoles (e.g. RDF4J
console)
• Libraries for various query languages
• RDF4J (Java), dotNetRDF, pysparql
(Python), ...
• APN and third-party tooling providing GUIs on top of
Neptune SPARQL
Example: Interacting with Neptune using the RDF4J console (http://rdf4j.org)
Example: Using APN software to build, maintain, and explore
Knowledge Graphs stored in Amazon Neptune

BUILDING APPLICATIONS ON AMAZON
NEPTUNE: PARTNER NETWORK EXAMPLE
Knowledge Graph Management
• SPARQL endpoint UI
• Navigation, exploration,
visualization
• Authoring, ontology and
instance data management
Knowledge Graph Application
Development
• Rapid prototyping of end-user
oriented applications
• Web components for end-user
oriented data interaction
Knowledge Graph Middleware
• “Queries as a Service”
• Interfaces to third party
applications

AMAZON NEPTUNE SPARQL TUTORIAL
1. Connect to your instance via SSH,
following the handout guidelines
2. Open http://localhost:2345/ in your
browser and log in
3. Click the “Start Tutorial” link on the
start page
4. Complete sections 2 – 6 of the
tutorial (~30min)

10 MIN BREAK!

Amazon Neptune Gremlin

PROPERTY GRAPH
A property graph is a set of vertices and edges with respective properties (i.e. key/value pairs)
• Vertex represents entities/domains
• Edge represents directional relationship
between vertices.
• Each edge has a label that denotes the
type of relationship
• Each vertex & edge has a unique identifier
• Vertex and edges can have properties
• Properties express non-relational information about the vertices and edges

PROPERTY GRAPH & APACHE TINKERPOP™
• Structure (graph)
Data model represented by a vertex/edge/property topology
• Process (traversal)
Means by which graph is analyzed
• Apache TinkerPop
Open source graph computing framework
• Gremlin
Graph traversal language used to analyze the graph

AMAZON NEPTUNE & APACHE TINKERPOP
• Full compatibility with Tinkerpop Gremlin 3.3.0 (latest version released August 2017)
• Optimized query execution engine for Gremlin query language
• Enhanced Gremlin features
• Optional user supplied IDs
• Multiple labels for vertex

GREMLIN: GRAPH TRAVERSAL LANGUAGE
• Domain specific language for graph
• Open source under Apache v2 license
• Open source clients for multiple languages like Java, Python, .NET, Groovy etc.
• Gremlin is a language composed of steps
• A step performs an atomic operation on the data stream
• Steps are chained together to form a traversal
• The output to a previous step is input to the next step that follows

MUSIC BRAINZ DATASET
artist FROM_AREA area
work labelPUBLISHING
FROM_AREA
https://musicbrainz.org/

QUERY EXPLAINED
Show the place where the writer of song “I Miss You” lives
g.V()

QUERY EXPLAINED
hasLabel("work")

QUERY EXPLAINED
has("type", "Song").has("name", "I Miss You")

QUERY EXPLAINED
in("WRITER")

QUERY EXPLAINED
out("FROM_AREA")

QUERY EXPLAINED
Values("name")

Source: https://github.com/apache/tinkerpop/blob/master/docs/static/images/gremlin-dashboard.png
LET’S TAKE GREMLIN FOR A SPIN!

GREMLIN CONSOLE
• Interactive REPL groovy shell
• Uses WebSocket
• Preinstalled on your machines
• Start it up!

GREMLIN: SIMPLE FILTERS
Filter on values steps has(),
hasNot(),
where(),
hasLabel()
Filter on elements
steps
dedup(),
limit(),
coin()
Boolean connectives
steps
and(),
or()

GREMLIN: PROJECTION QUERY

GREMLIN: MUTATION

TRY IT OUT!

Q&A
Michael Schmidt,
Sr. Software Development Engineer - schmdtm@amazon.com
Divij Vaidya,
Software Development Engineer - divijv@amazon.com

Tom Sawyer Perspectives
Amazon Partner Network Graph
Database Browser for Amazon
Neptune

Presented by:
Patrick Madden
Principal Solutions Architect
Tom Sawyer Software

Agenda
• Introduction
• Tom Sawyer Perspectives for Amazon Neptune
• Graph Database Browser Demonstration
• Questions

Tom Sawyer Software
• Leading provider of software for building graph and
data visualization and analysis applications
• Flagship product—Tom Sawyer Perspectives
• Full-service data visualization vendor
• Well-known for having the best graph layout
technology
• Headquartered in Berkeley, California
• Offices in the United States, Australia, Germany,
Greece, Latvia, and Mexico
• Founded 1992

Customers

Tom Sawyer Perspectives

Standards-based Data Integration
Amazon Neptune

Tom Sawyer Perspectives Configurations
Graph and Data Visualization SDK Graph Database Browser
Model-Based Engineering Business Processes

Graph Database Browser for Neptune
• Easy to load data from Amazon S3 without writing Amazon
Neptune load commands
• Load data from many sources
• Automatically detects and displays labeled data types
• Query and data validation
• Browse and visualize connections
• Automated layout and interactive views
• Built-in graph algorithms: Social network analysis, Clustering,
Reachability, Shortest path analysis, Traversals and Flows

Demonstration

Q&A
Patrick Madden
pmadden@tomsawyer.com
www.tomsawyer.com

THANK YOU!

BACKUP SLIDES

SETTING UP AMAZON NEPTUNE VIA CONSOLE
https://yukon.aws.amazon.com/neptune?region=us-east-1

• Secure deployment in a VPC
• Increased availability through deployment
in two subnets in two different Availability
Zones (AZs)
• Cluster volume always spans three AZ to
provide durable storage
• See https://alpha-docs-
aws.amazon.com/neptune/latest/userguide
/get-started-prerequisites.html for VPC
setup details
AMAZON NEPTUNE: SECURE VPC DEPLOYMENT

GET
/sparql/?query=PREFIX%20dc%3A%20%3Chttp%3A%2F%2Fpurl.org%2Fdc%2Felements%2F1.1%2F%3E%20%0ASELECT%20%3Fbook%20%3Fwho%20%0A
WHERE%20%7B%20%3Fbook%20dc%3Acreator%20%3Fwho%20%7D%0A HTTP/1.1 Host: www.example User-agent: my-sparql-client/0.1
HTTP/1.1 200 OK
Date: Fri, 06 May 2005 20:55:12 GMT
...
<?xml version="1.0"?>
<sparql xmlns="http://www.w3.org/2005/sparql-results#">
<head>
<variable name="book"/>
<variable name="who"/>
</head>
<results>
<result>
<binding name="book">
<uri>http://www.example/book/book5</uri>
</binding>
<binding name="who">
<bnode>r29392923r2922</bnode></binding>
</result>
</sparql>
SPARQL Protocol

curl --cacert cert.pem -G https://demo-cluster-01:8182/sparql --data-urlencode "query=SELECT * WHERE { ?s ?p ?o } LIMIT
2"
Output:
<?xml version='1.0' encoding='UTF-8'?>
<sparql xmlns='http://www.w3.org/2005/sparql-results#'>
<head>
<variable name='s'/>
<variable name='p'/>
<variable name='o'/>
</head>
<results>
<result>
<binding name='s'>
<uri>http://www.w3.org/1999/02/22-rdf-syntax-ns#type</uri>
</binding>
<binding name='p'>
<uri>http://www.w3.org/1999/02/22-rdf-syntax-ns#type</uri>
</binding>
<binding name='o'>
<uri>http://www.w3.org/2000/01/rdf-schema#Resource</uri>
</binding>
</result>
</results>
</sparql>
SPARQL Queries via curl

NEW LAUNCH! How to build graph applications with SPARQL and Gremlin using Amazon Neptune - DAT342 - re:Invent 2017

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to NEW LAUNCH! How to build graph applications with SPARQL and Gremlin using Amazon Neptune - DAT342 - re:Invent 2017

Similar to NEW LAUNCH! How to build graph applications with SPARQL and Gremlin using Amazon Neptune - DAT342 - re:Invent 2017 (20)

More from Amazon Web Services

More from Amazon Web Services (20)

NEW LAUNCH! How to build graph applications with SPARQL and Gremlin using Amazon Neptune - DAT342 - re:Invent 2017