In this talk we'll explore powerful analytic techniques for graph data. Firstly we'll discover some of the innate properties of (social) graphs from fields like anthropology and sociology. By understanding the forces and tensions within the graph structure and applying some graph theory, we'll be able to predict how the graph will evolve over time. To test just how powerful and accurate graph theory is, we'll also be able to (retrospectively) predict World War 1 based on a social graph and a few simple mechanical rules.
Then we'll see how graph matching can be used to extract online business intelligence (for powerful retail recommendations). In turn we'll apply these powerful techniques to modelling domains in Neo4j (a graph database) and show how Neo4j can be used to drive business intelligence.
Don't worry, there won't be much maths :-)
The 7 Things I Know About Cyber Security After 25 Years | April 2024
A Little Graph Theory for the Busy Developer - Jim Webber @ GraphConnect Chicago 2013
1. A Little Graph Theory for the
Busy Developer
Jim Webber
Chief Scientist, Neo Technology
@jimwebber
2. Roadmap
• Imprisoned data
• Graph models
• Graph theory
– Local properties, global behaviors
– Predictive techniques
• Graph matching
– Real-time analytics for fun and profit
• Fin
7. Aggregate-Oriented Data
http://martinfowler.com/bliki/AggregateOrientedDatabase.html
“There is a significant downside - the whole approach works really well
when data access is aligned with the aggregates, but what if you want to
look at the data in a different way? Order entry naturally stores orders as
aggregates, but analyzing product sales cuts across the aggregate structure.
The advantage of not using an aggregate structure in the database is that it
allows you to slice and dice your data different ways for different
audiences.
This is why aggregate-oriented stores talk so much about map-reduce.”
12. Property graphs
• Property graph model:
– Nodes with properties
– Named, directed relationships with properties
– Relationships have exactly one start and end node
• Which may be the same node
36. Strong Triadic Closure
It if a node has strong relationships to two
neighbours, then these neighbours must have at
least a weak relationship between them.
[Wikipedia]
39. Weak relationships
• Relationships can have “strength” as well as
intent
– Think: weighting on a relationship in a property
graph
• Weak links play another super-important
structural role in graph theory
– They bridge neighbourhoods
41. Local Bridge Property
“If a node A in a network satisfies the Strong
Triadic Closure Property and is involved in at
least two strong relationships, then any local
bridge it is involved in must be a weak
relationship.”
[Easley and Kleinberg]
43. Graph Partitioning
• (NP) Hard problem
– Recursively remove the spanning links between
dense regions
– Or recursively merge nodes into ever larger
“subgraph” nodes
– Choose your algorithm carefully – some are better
than others for a given domain
• Can use to (almost exactly) predict the
break up of the karate club!
47. Cypher
• Declarative graph pattern matching language
– “SQL for graphs”
– Columnar results
• Supports graph matching commands and
queries
– Find me stuff like this…
– Aggregation, ordering and limit, etc.
55. Wrap in a Cypher MATCH clause
MATCH (daddy)-[:BOUGHT]->()-[:MEMBER_OF]->(nappies),
(daddy)-[:BOUGHT]->()-[:MEMBER_OF]->(beer),
(daddy)-[b:BOUGHT]->()-[:MEMBER_OF]->(console)
56. Cypher WHERE clause
MATCH (daddy)-[:BOUGHT]->()-[:MEMBER_OF]->(nappies),
(daddy)-[:BOUGHT]->()-[:MEMBER_OF]->(beer),
(daddy)-[b:BOUGHT]->()-[:MEMBER_OF]->(console)
WHERE b is null
57. Full Cypher query
START beer=node:categories(category=‘beer’),
nappies=node:categories(category=‘nappies’),
xbox=node:products(product=‘xbox 360’)
MATCH (daddy)-[:BOUGHT]->()-[:MEMBER_OF]->(beer),
(daddy)-[:BOUGHT]->()-[:MEMBER_OF]->(nappies),
(daddy)-[b?:BOUGHT]->(xbox)
WHERE b is null
RETURN distinct daddy
65. What are graphs good for?
• Recommendations
• Pharmacology
• Business intelligence
• Social computing
• Geospatial
• MDM
• Data center management
• Web of things
• Genealogy
• Time series data
• Product catalogue
• Web analytics
• Scientific computing
• Indexing your slow RDBMS
• And much more!
Vanity slide - Work for Neo Technology, the commercial backer of the Neo4j open source graph database
We’ve all been there – trudging through 3 intermediate tables just to get the data you want.
We have key-value stores, typically very highly available and scalable for simple key-value data
Column stores naturally-indexed value storesContrary to common belief – Google’s big table isn’t the world’s most famous column storeBritish museum London is: it’s got columns and it’s where we stored all the stuff we nicked from the British Empire!
Aggregates:No more impedance mismatch, just decompose to documents/ k-v / columnsThe data model is less expressive than RDBMS, but that’s OK because:RDBMS constraints don’t really match application concernsThe expressiveness is in your domain logic, right?
People talk about Codd’s relational model being mature because it was proposed in 1969 – 42 years old.Euler’s graph theory was proposed in 1736 – 275 years old.
Graphs are the most natural way to model most domains. You already know this because you draw graphs on a whiteboard, but you’ve never had the opportunity to take that down into the database before.Nodes are a bit like documents, but they’re flat at present in Neo4j.You pour data into your nodes and then connect them – easy peasy.This enables high fidelity domain modeling because this is how your domains work.And you don’t have to do this stuff in your application code – it’s right there in the databaseLet’s prove it by exploring a fun domain…
Start off sketching the domain. That’s your model done – we see this when we revisit databases months after they’re been designed and put into productionNo decomposition, ER design, normalisation/denormalisation as you need with RDBMS.
Predictive analytics
Euler reduced the problem into an abstract formFrom geography to diagrammatic through to a graph.It’s the first documented case of representing the real world as a graph.
First we need to talk about some local properties
A triadic closure is a local property of (social) graphs whereby if two nodes are connected via a path involving a third node, there is an increased likelihood that the two nodes will become directly connected in future. This is a familiar enough situation for us in a social setting whereby if we happen to be friends with two people, ultimately there's an increased chance that those people will become direct friends too, since by being our friend in the first place, it's an indication of social similarity and suitability. It’s called triadic closure, because we try to close the triangle.
We see this all the time – it’s likely that if we have two friends, that they will also become at least acquaintances and potentially friends themselves!In general, if a node A has relationships to B & C then the relationship between B&C is likely to form – especially if the existing relationships are both strong.This is an incredibly strong assertion and will not be typically upheld by all subgraphs in a graph. Nonetheless it is sufficiently commonplace (particularly in social networks) to be trusted as a predictive aid.
Sentiment plays a role in how closures form too – there is a notion of balance.
From a triadic closure perspective this is OK, but intuitively it seems odd.Cartman’s friends shouldn’t be friends with his enemies. Nor should Cartman’s enemies be friends with his friends.
This makes sense – Cartman’s friend Craig is also an enemy of Cartman’s enemy TweekTwo negative sentiments and one positive sentiment is a balanced structure – and it makes sense too since we gang up with our friends on our poor beleaguered enemy
Another balanced – and more pleasant – arrangement is for three positive sentiments, in this case mutual friends.
A starting point for a network of friends and enemiesRed links indicate enemy of relationshipBlack links indicate friend of relationshipThe Three Emperor’s league
Italy forms the with Austria and Germany – a balanced +++ triadic closureIf Italy had made only a single alliance (or enemy) it would have been unstable and another relationship would be likely to form anyway!Triple Alliance
Russia becomes hostile to Austria and Germany – a balance --+ d triadic closure becomes agnostic towards France.German-Russian Lapse
The French and Russians ally, forming a balanced --+ triadic closure with the UKFrench-Russian Alliance
The UK and France enter into the famous Entente CordialeThis produces an unbalanced ++- triadic closure with Russia, and the graph doesn’t like it.
The British and Russians form an alliance, thereby changing their previously unbalanced triadic closure into a balanced one.Other local pressures on the graph make other closures form.Italy becomes hostile to Russia, forming a balanced --+ closure with the France, and another balanced --+ closure with the UK.Germany and the UK become hostile forming a balanced --+ closure with Austria and another balanced --+ closure with ItalyBritish-Russian Alliance
That WWI can be predicted without domain knowledge by iterating a graph and applying local structural constraints is nothing short of astonishing to me.Note how the network slides into a balanced labeling — and into World War I.
In this case the string triadic closure property still holds – though it is a weak link that characterises the relationship between Stan and Cartman.Given a starting graph, we can apply this simple local principal to see how it would evolve.
In this case the string triadic closure property still holds – though it is a weak link that characterises the relationship between Stan and Cartman.Given a starting graph, we can apply this simple local principal to see how it would evolve.
A local bridge acts as a link – perhaps the only realistic link - between two otherwise distant (or separate) subgraphs.Local bridges are semantically rich – they provide conduits for information flow between otherwise independent groups.In this case DATING is a local bridge – it must also be a weak relationship according to our definition of a local bridgeIntuitively this makes sense – your girl/boyfriend is rather less important at age 8 than your regular friends, IIRC.
How do we identify local bridges? Any weak link which would cause a component of the graph to become disconnected.Being able to identify local bridges is important – in this case it’s the only know conduit to allow the girls and boys to communicate.In real life local bridges are apparent in your organisation as experts (or managers); appear as nexus in fraud cases;
Zachary in the Journal of Anthropological Research 1977Intuitively we can see “clumps” in this graph.But how do we separate them out? It’s called minimum cut.
What’s interesting is that it’s mechanical – no domain knowledge is necessary. There’s only one failure with the method Zachary chose to partition the graph: node 9 should have gone to the instructor’s club but instead went with the original president of the club (node 34).Why? Because the student was three weeks away from completing a four-year quest to obtain a black belt, which he could only do with the instructor (node 1)Other minimum cut approaches might deliver slightly different results, but on the whole it’s amazing you get such insight from an algorithm!
But is there enough information in the graph itself to predict the schism?
But is there enough information in the graph itself to predict the schism?
We can use graph matching to look for behavioural patterns in the graph too!
The insight here is that we have a typical young father who buys beer, nappies and a game console simply by reducing subgraphWe have a pattern to search for
Now we look for young fathers – implied by beer and nappies purchases – who haven’t bought a game console.
Note that Max de Marzireimplemented a functionally better Graph Search with Neo4j and some Ruby gems for language processing in a weekend!
Image: real-timeBeing able to look for all young fathers who might be tempted to buy a new game console is helpful, but not dramatically different from what we have nowIt’s much faster to process in a graph, but still a latent business activity (e.g. mailshot)But you can take the same idea and run the query in real time foran individual customer at they go through the checkout!And you can do more too – you can add in more/less dimensions to the search.Does the shop have stock?Does the young father live in the right target area (r-tree, postcode)?We can vary the number of dimensions we include to tailor the search for performance/accuracy very easily in a graph – query latency is proportional to the amount of graph searched, not data set size.