These slides support the GraphFrames: DataFrame-based graphs for Apache Spark webinar. In this webinar, the developers of the GraphFrames package will give an overview, a live demo, and a discussion of design decisions and future plans. This talk will be generally accessible, covering major improvements from GraphX and providing resources for getting started. A running example of analyzing flight delays will be used to explain the range of GraphFrame functionality: simple SQL and graph queries, motif finding, and powerful graph algorithms.

1. GraphFrames
DataFrame-based graphs for Apache® Spark™
Joseph K. Bradley
4/14/2016

2. About the speaker: Joseph Bradley
Joseph Bradley is a Software Engineerand Apache
Spark PMC member working on MLlib at
Databricks. Previously,he was a postdoc at UC
Berkeley after receiving hisPh.D. in Machine
Learning from Carnegie Mellon U.in 2013.His
research included probabilistic graphical models,
parallel sparse regression, and aggregation
mechanismsfor peergrading in MOOCs.
2

3. About the moderator: Denny Lee
Denny Lee is a Technology Evangelistwith
Databricks; he is a hands-on data sciencesengineer
with more than 15 years of experience developing
internet-scale infrastructure, data platforms, and
distributed systems for both on-premisesand cloud.
Prior to joining Databricks, Denny worked as a
SeniorDirector of Data SciencesEngineering at
Concur and was part of the incubation teamthat
builtHadoop on Windowsand Azure (currently
known as HDInsight).
3

4. We are Databricks, the company behind Apache Spark
Founded by the creators of
Apache Spark in 2013
Share of Spark code
contributed by Databricks
in 2014
75%
4
Data Value
Created Databricks on top of Spark to make big data simple.

5. …
Apache Spark Engine
Spark Core
Spark
Streaming
Spark SQL MLlib GraphX
Unified engineacross diverse workloads & environments
Scale out, fault tolerant
Python, Java, Scala, and R
APIs
Standard libraries

7. NOTABL E USERS THAT PRESENTED AT SPARK SUMMIT
2 0 1 5 SAN F RANCISCO
Source: Slide5ofSparkCommunityUpdate

8. Outline
GraphFrames overview
GraphFrames vs. GraphXand other libraries
Details for power users
Roadmap and resources
8

9. Outline
GraphFrames overview
GraphFrames vs. GraphXand other libraries
Details for power users
Roadmap and resources
9

10. Graphs
10
vertex
edge
id City State
“JFK” “New York” NY
Example: airports & flights between them
JFK
IAD
LAX
SFO
SEA
DFW
src dst delay tripID
“JFK
”
“SEA” 45 1058923

11. Apache Spark’s GraphX library
Overview
• General-purpose graph
processinglibrary
• Optimized for fast
distributedcomputing
• Library of algorithms:
PageRank, Connected
Components,etc.
11
Challenges
• No Java, PythonAPIs
• Lower-levelRDD-based
API (vs.DataFrames)
• Cannot use recent Spark
optimizations:Catalyst
query optimizer,Tungsten
memory management

12. Enter GraphFrames
Goal: DataFrame-based graphson ApacheSpark
• Simplify interactive queries
• Support motif-findingforstructural pattern search
• Benefitfrom DataFrame optimizations
Collaboration between Databricks, UC Berkeley& MIT
+ Now with community contributors!
12

13. Graphs
13
vertex
edge
id City State
“JFK” “New York” NY
Example: airports & flights between them
JFK
IAD
LAX
SFO
SEA
DFW
src dst delay tripID
“JFK
”
“SEA” 45 1058923

14. GraphFrames
“vertices” DataFrame
• 1 vertexper Row
• id: column with unique ID
“edges” DataFrame
• 1 edge per Row
• src, dst: columns using IDs from vertices.id
14
Extra columns store vertexor edge data
(a.k.a. attributes or properties).
id City State
“JFK” “New York” NY
“SEA” “Seattle” WA
src dst delay tripID
“JFK” “SEA” 45 1058923
“DFW” “SFO” -7 4100224

15. Demo:
Building a GraphFrame
15

16. 16

19. Queries
Simple queries
Motif finding
Graph algorithms
19

20. Simple queries
SQL queries on vertices & edges
E.g., what trips are most likely to have significantdelays?
20
Graph queries
• Vertex degrees
• # edgesper vertex(incoming,outgoing,total)
• Triplets
• Join vertices and edgesto get (src, edge,dst)

21. 21

24. Motif finding
24
IAD
JFK
LAX
SFO
SEA
DFW
Search for structural
patterns within a graph.
val paths: DataFrame =
g.find(“(a)-[e1]->(b);
(b)-[e2]->(c);
!(c)-[]->(a)”)

25. Motif finding
25
IAD
JFK
LAX
SFO
SEA
DFW
(b)
(a)Search for structural
patterns within a graph.
val paths: DataFrame =
g.find(“(a)-[e1]->(b);
(b)-[e2]->(c);
!(c)-[]->(a)”)

26. Motif finding
26
IAD
JFK
LAX
SFO
SEA
DFW
(b)
(a)
(c)
Search for structural
patterns within a graph.
val paths: DataFrame =
g.find(“(a)-[e1]->(b);
(b)-[e2]->(c);
!(c)-[]->(a)”)

27. Motif finding
27
IAD
JFK
LAX
SFO
SEA
DFW
(b)
(a)
(c)
Search for structural
patterns within a graph.
val paths: DataFrame =
g.find(“(a)-[e1]->(b);
(b)-[e2]->(c);
!(c)-[]->(a)”)

28. Motif finding
28
IAD
JFK
LAX
SFO
SEA
DFW
Search for structural
patterns within a graph.
val paths: DataFrame =
g.find(“(a)-[e1]->(b);
(b)-[e2]->(c);
!(c)-[]->(a)”)
(b)
(a)
(c)
Then filter using vertex &
edge data.
paths.filter(“e1.delay > 20”)

29. 29

31. Graph algorithms
Find importantvertices
• PageRank
31
Find pathsbetweensets of vertices
• Breadth-first search (BFS)
• Shortest paths
Find groupsof vertices(components,
communities)
• Connected components
• Strongly connected components
• Label Propagation Algorithm(LPA)
Other
• Triangle counting
• SVDPlusPlus

32. 32

33. Algorithm implementations
Mostly wrappers for GraphX
• PageRank
• Shortest paths
• Connected components
• Strongly connected components
• Label Propagation Algorithm (LPA)
• SVDPlusPlus
33
Some algorithms implemented
usingDataFrames
• Breadth-first search
• Triangle counting

34. Saving & loading graphs
Save & load the DataFrames.
vertices = sqlContext.read.parquet(...)
edges = sqlContext.read.parquet(...)
g = GraphFrame(vertices, edges)
g.vertices.write.parquet(...)
g.edges.write.parquet(...)
In the future...
• SQL data sources for graph formats
34

35. APIs: Scala, Java, Python
API available from all 3 languages
à First time GraphX functionality hasbeen available to
Java & Python users
2 missing items (WIP)
• Java-friendliness is currently in alpha.
• Python does not have aggregateMessages
(for implementing your own graph algorithms).
35

36. Outline
GraphFrames overview
GraphFrames vs. GraphX and other libraries
Details for power users
Roadmap and resources
36

37. 2 types of graph libraries
37
Graph algorithms Graph queries
Standard & custom algorithms
Optimized for batch processing
Motif finding
Point queries &updates
GraphFrames: Both algorithms &queries (but notpoint updates)

38. GraphFrames vs. GraphX
38
GraphFrames GraphX
Builton DataFrames RDDs
Languages Scala, Java, Python Scala
Use cases Queries & algorithms Algorithms
Vertex IDs Any type (in Catalyst) Long
Vertex/edg
e attributes
Any number of
DataFrame columns
Any type (VD, ED)
Return
types
GraphFrame or
DataFrame
Graph[VD, ED], or
RDD[Long, VD]

39. GraphX compatibility
Simple conversionsbetweenGraphFrames& GraphX.
val g: GraphFrame = ...
// Convert GraphFrame à GraphX
val gx: Graph[Row, Row] = g.toGraphX
// Convert GraphX à GraphFrame
val g2: GraphFrame = GraphFrame.fromGraphX(gx)
39
Vertex & edgeattributes
are Rows in order to
handlenon-LongIDs
Wrapping existing GraphX code: See Belief Propagation example:
https://github.com/graphframes/graphframes/blob/master/src/main/scala/org/graphframes/examples/BeliefPropagation.scala

40. Outline
GraphFrames overview
GraphFrames vs. GraphXand other libraries
Details for power users
Roadmap and resources
40

41. Scalability
Currentstatus
• DataFrame-based parts benefitfrom DataFrame scalability +
performance optimizations(Catalyst, Tungsten).
• GraphX wrappers are as fast as GraphX (+ conversion overhead).
WIP
• GraphX hasoptimizationswhich are not yet ported to GraphFrames.
• See nextslide…
41

42. WIP optimizations
Join elimination
• GraphFrame algorithms require lots
of joins.
• Not all joins are necessary
Solution:
• Vertex IDs serve as unique keys.
• Tracking keys allows Catalyst to
eliminate some joins.
42
For more info & benchmark results, see AnkurDave’s SSE 2016 talk.
https://spark-summit.org/east-2016/events/graphframes-graph-queries-in-spark-sql/
Materializedviews
• Data locality for common usecases
• Message-passing algorithms often
need “triplet view” (src,edge, dst)
Solution:
• Materialize specific views
• Analogous to GraphX’s “replicated
vertex view”

43. Implementing new algorithms
43
Method 2: Messagepassing
aggregateMessages
• Same primitive as GraphX
• Specify messages & aggregation
using DataFrame expressions
Belief propagation example code
Method 1: DataFrame &
GraphFrame operations
Motif finding
• Series of DataFrame joins
Triangle count
• DataFrame ops + motif finding
BFS
• DataFrame joins & filters

44. Outline
GraphFrames overview
GraphFrames vs. GraphXand other libraries
Details for power users
Roadmap and resources
44

45. Current status
Published
• Open source (Apache 2.0) on Github
https://github.com/graphframes/graphframes
• Spark package http://spark-
packages.org/package/graphframes/graphframes
Compatible
• Spark 1.4, 1.5, 1.6
• Databricks Community Edition
Documented
• http://graphframes.github.io/
45

46. Roadmap
• MergeWIP speed optimizations
• Java API tests & examples
• Migrate more algorithms to DataFrame-based
implementations for greater scalability
• Getcommunity feedback!
46
Contribute
• Tracking issueson Github
• Thanks to those who have
already sent pull requests!

47. Resources for learning more
User guide + API docs http://graphframes.github.io/
• Quick-start
• Overview & examples for all algorithms
• Alsoavailable as executablenotebooks:
• Scala: http://go.databricks.com/hubfs/notebooks/3-GraphFrames-User-Guide-scala.html
• Python: http://go.databricks.com/hubfs/notebooks/3-GraphFrames-User-Guide-python.html
Blog posts
• Intro: https://databricks.com/blog/2016/03/03/introducing-graphframes.html
• Flight delay analysis: https://databricks.com/blog/2016/03/16/on-time-flight-performance-
with-spark-graphframes.html
47

48. 48

49. Thank you!
Thanks to
• Denny Lee & Bill Chambers (demo)
• Tim Hunter, Xiangrui Meng, Ankur Dave &others (GraphFrames development)