Loom and Graphs in Clojure

Loom and Graphs
in Clojure
github.com/aysylu/loom
Aysylu Greenberg
@aysylu22; http://aysy.lu
LispNYC, August 13th 2013

Overview
•  Loom's Graph API
•  Graph Algorithms in Loom
•  Titanium Loom
•  Single Static Assignment (SSA) Loom

Overview
•  Graph Algorithms in Loom
•  SSA Loom

Loom's Graph API
•  Graph, Digraph, Weighted Graph

Loom's Graph API
•  FlyGraph

Loom's Graph API
•  FlyGraph
o  read-only, ad-hoc

Loom's Graph API
•  FlyGraph
o  edges from nodes + successors

Loom's Graph API
•  FlyGraph
o  edges from nodes + successors
o  nodes and edges from successors + start

Loom's Graph API
•  Uses Clojure protocols
(clojure.org/protocols)

Loom's Graph API
o  specification only, no implementation

Loom's Graph API
o  single type can implement multiple
protocols

Loom's Graph API
o  single type can implement multiple
protocols
o  interfaces: design-time choice of the type
author, protocols: can be added to a type
at runtime

Loom's Graph API
(defprotocol Graph
(add-nodes* [g nodes] "Add nodes to graph g. See add-nodes”
(add-edges* [g edges] "Add edges to graph g. See add-edges")

Loom's Graph API
(defprotocol Graph
(add-edges* [g edges] "Add edges to graph g. See add-edges”)
(remove-nodes* [g nodes] "Remove nodes from graph g. See
remove-nodes”)
(remove-edges* [g edges] "Removes edges from graph g. See
remove-edges”)
(remove-all [g] "Removes all nodes and edges from graph g")

Loom's Graph API
(defprotocol Graph
(add-edges* [g edges] "Add edges to graph g. See add-edges”
remove-nodes”)
remove-edges”)
(remove-all [g] "Removes all nodes and edges from graph g”
(nodes [g] "Return a collection of the nodes in graph g”)
(edges [g] "Edges in g. May return each edge twice in an undirected
graph")

Loom's Graph API
(defprotocol Graph
remove-nodes”)
remove-edges”)
graph”)
(has-node? [g node] "Return true when node is in g”)
(has-edge? [g n1 n2] "Return true when edge [n1 n2] is in g")

Loom's Graph API
(defprotocol Graph
remove-nodes”)
remove-edges”)
graph”)
(has-node? [g node] "Return true when node is in g”)
(has-edge? [g n1 n2] "Return true when edge [n1 n2] is in g”)
(successors [g] [g node] "Return direct successors of node, or
(partial successors g)”)
(out-degree [g node] "Return the number of direct successors of
node"))

Loom's Graph API
(defprotocol Digraph
(predecessors [g] [g node] "Return direct
predecessors of node, or (partial predecessors g)”)
(in-degree [g node] "Return the number direct
predecessors to node")

Loom's Graph API
(defprotocol Digraph
(predecessors [g] [g node] "Return direct
predecessors of node, or (partial predecessors g)”)
(in-degree [g node] "Return the number direct
predecessors to node”)
(transpose [g] "Return a graph with all edges
reversed"))

Loom's Graph API
(defprotocol WeightedGraph
(weight [g] [g n1 n2] "Return weight of edge [n1 n2]
or (partial weight g)"))

Graph Algorithms in Loom
•  DFS/BFS (+ bidirectional)

•  Topological Sort

•  Single Source Shortest Path (Dijkstra, Bellman-Ford)

•  Strongly Connected Components (Kosaraju)

•  Density (edges/nodes)

•  Loner nodes

•  Loner nodes
•  2 coloring

•  Loner nodes
•  2 coloring
•  Max-Flow (Edmonds-Karp)

•  Loner nodes
•  2 coloring
•  Max-Flow (Edmonds-Karp)
•  alg-generic requires only successors + start (where
appropriate)

Graph Algorithms: Bellman-Ford
A B C
D E
3 4
5
2
-8

CLRS Introduction to Algorithms

(defn- init-estimates
"Initializes path cost estimates and paths from source to all vertices, for
Bellman-Ford algorithm”
[graph start]
(let [nodes (disj (nodes graph) start)
path-costs {start 0}
paths {start nil}
infinities (repeat Double/POSITIVE_INFINITY)
nils (repeat nil)
init-costs (interleave nodes infinities)
init-paths (interleave nodes nils)]
[(apply assoc path-costs init-costs)
(apply assoc paths init-paths)]))

(defn- can-relax-edge?
"Test for whether we can improve the shortest path to v found so far by
going through u.”
[[u v :as edge] weight costs]
(let [vd (get costs v)
ud (get costs u)
sum (+ ud weight)]
(> vd sum)))

(defn- relax-edge
"If there's a shorter path from s to v via u, update our map of
estimated path costs and map of paths from source to vertex v”
[[u v :as edge] weight [costs paths :as estimates]]
(let [ud (get costs u)
sum (+ ud weight)]
(if (can-relax-edge? edge weight costs)
[(assoc costs v sum) (assoc paths v u)]
estimates)))

(defn- relax-edges
"Performs edge relaxation on all edges in weighted directed graph”
[g start estimates]
(->> (edges g)
(reduce (fn [estimates [u v :as edge]]
(relax-edge edge (wt g u v) estimates))
estimates)))

(defn bellman-ford
"Given a weighted, directed graph G = (V, E) with source start, the Bellman-Ford algorithm produces map of single source shortest paths and their
costs if no negative-weight cycle that is reachable from the source exits, and false otherwise, indicating that no solution exists."
[g start]
(let [initial-estimates (init-estimates g start)
;relax-edges is calculated for all edges V-1 times
[costs paths] (reduce (fn [estimates _]
(relax-edges g start estimates))
initial-estimates
(-> g nodes count dec range))
edges (edges g)]
(if (some
(fn [[u v :as edge]]
(can-relax-edge? edge (wt g u v) costs))
edges)
false
[costs
(->> (keys paths)
;remove vertices that are unreachable from source
(remove #(= Double/POSITIVE_INFINITY (get costs %)))
(reduce
(fn [final-paths v]
(assoc final-paths v
; follows the parent pointers
; to construct path from source to node v
(loop [node v
path ()]
(if node
(recur (get paths node) (cons node path))
path))))
{}))])))

(defn bellman-ford
"Given a weighted, directed graph G = (V, E) with source start,
the Bellman-Ford algorithm produces map of single source
shortest paths and their costs if no negative-weight cycle that is
reachable from the source exits, and false otherwise, indicating
that no solution exists."

[g start]
(let [initial-estimates (init-estimates g start)
;relax-edges is calculated for all edges V-1 times
[costs paths] (reduce (fn [estimates _]
(relax-edges g start estimates))
initial-estimates
(->> g (nodes) (count) (dec) (range)))
edges (edges g)]

(if (some (fn [[u v :as edge]]
(can-relax-edge? edge (wt g u v) costs))
edges)
false

[costs
(->> (keys paths)
(remove
#(= Double/POSITIVE_INFINITY (get costs %)))

[costs
(->> (keys paths)
(remove
#(= Double/POSITIVE_INFINITY (get costs %)))
(reduce
(fn [final-paths v]
(assoc final-paths v
; follows the parent pointers
; to construct path from source to node v
(loop [node v path ()]
(if node
(recur (get paths node) (cons node path))
path))))
{}))])))

Overview
•  Graph Algorithms
•  SSA Loom

Titanium Loom
•  Titanium by Clojurewerkz
(titanium.clojurewerkz.org)

Titanium Loom
•  Clojure graph library built on top of Aurelius
Titan (thinkaurelius.github.com/titan)

Titanium Loom
•  Various storage backends: Cassandra,
HBase, BerkeleyDB Java Edition

Titanium Loom
•  Various storage backends: Cassandra,
HBase, BerkeleyDB Java Edition
•  No graph visualization

Titanium Loom
(let [in-mem-graph (tg/open {"storage.backend" "inmemory"})]
(tg/transact!
(let [
a (nodes/create! {:name "Node A"})
b (nodes/create! {:name "Node B"})
c (nodes/create! {:name "Node C"})

Titanium Loom
(let [in-mem-graph (tg/open {"storage.backend" "inmemory"})]
(tg/transact!
(let [
a (nodes/create! {:name "Node A"})
b (nodes/create! {:name "Node B"})
c (nodes/create! {:name "Node C"})
e1 (edges/connect! a "edge A->B" b)
e2 (edges/connect! b "edge B->C" c)
e3 (edges/connect! c "edge C->A” a)
graph (titanium->loom in-mem-graph)])

Titanium Loom
(defn titanium->loom
"Converts titanium graph into Loom representation”
([titanium-graph &
{:keys [node-fn edge-fn weight-fn]
:or {node-fn (nodes/get-all-vertices)
edge-fn (map (juxt edges/tail-vertex
edges/head-vertex)
(edges/get-all-edges))
weight-fn (constantly 1)}}]
(let [nodes-set (set node-fn)
edges-set (set edge-fn)]

Titanium Loom
(reify
Graph
(nodes [_] nodes-set)
(edges [_] edges-set)
(has-node? [g node] (contains? (nodes g) node))
(has-edge? [g n1 n2] (contains? (edges g) [n1 n2]))
(successors [g] (partial successors g))
(successors [g node] (filter (nodes g)
(seq (nodes/connected-out-vertices node))))
(out-degree [g node] (count (successors g node)))

Titanium Loom
(reify
Graph
(nodes [_] nodes-set)
(edges [_] edges-set)
(has-node? [g node] (contains? (nodes g) node))
(has-edge? [g n1 n2] (contains? (edges g) [n1 n2]))
(successors [g node] (filter (nodes g)
(seq (nodes/connected-out-vertices node))))
Digraph
(predecessors [g] (partial predecessors g))
(predecessors [g node] (filter (nodes g)
(seq (nodes/connected-in-vertices node))))
(in-degree [g node] (count (predecessors g node)))
WeightedGraph
(weight [g] (partial weight g))
(weight [g n1 n2] (weight-fn n1 n2))))))

SSA Loom
•  Single Static Assignment (SSA) form
produced by core.async

SSA Loom
•  Single Static Assignment (SSA) form
produced by core.async
•  Generated by parse-to-state-machine
function

SSA Loom
(parse-to-state-machine
'[(if (> (+ x 1 2 y) 0)
(+ x 1)
(+ x 2))])

SSA Loom
'[(if (> (+ x 1 2 y) 0)
(+ x 1)
(+ x 2))])
[inst_4938
{:current-block 76,
:start-block 73,
:block-catches {76 nil, 75 nil, 74 nil, 73 nil},
:blocks
{76
[{:value :clojure.core.async.impl.ioc-macros/value, :id
inst_4937}
{:value inst_4937, :id inst_4938}],
75
[{:refs [clojure.core/+ x 2], :id inst_4935}
{:value inst_4935, :block 76, :id inst_4936}],
74
73
[{:refs [clojure.core/+ x 1 2 y], :id inst_4930}
{:refs [clojure.core/> inst_4930 0], :id inst_4931}
{:test inst_4931, :then-block 74, :else-block 75, :id
inst_4932}]}}]

SSA Loom
(def ssa
(->>
'[(if (> (+ x 1 2 y) 0)
(+ x 1)
(+ x 2))])
second
:blocks))

SSA Loom
{76
[{:value :clojure.core.async.impl.ioc-macros/
value, :id inst_4937}
{:value inst_4937, :id inst_4938}],
75
74
73
[{:refs [clojure.core/+ x 1 2 y], :id inst_4930}
{:refs [clojure.core/> inst_4930 0], :id
inst_4931}
{:test inst_4931, :then-block 74, :else-block
75, :id inst_4932}]}}]
(def ssa
(->>
'[(if (> (+ x 1 2 y) 0)
(+ x 1)
(+ x 2))])
second
:blocks))

SSA Loom
(view (ssa->loom ssa ssa-nodes-fn ssa-edges-fn))

SSA Loom
(defn ssa->loom
"Converts the SSA form generated by core.async into Loom
representation.”
([ssa node-fn edge-fn]
(let [nodes (delay (node-fn ssa))
edges (delay (edge-fn ssa))]

SSA Loom
{:graph (reify
Graph
(nodes [g] @nodes)
(edges [g] @edges)
(has-node? [g node] (contains? @nodes node))
(has-edge? [g n1 n2] (contains? @edges [n1 n2]))
(successors [g node]
(->> @edges
(filter (fn [[n1 n2]] (= n1 node)))
(map second)))

SSA Loom
Digraph
(predecessors [g] (partial predecessors g))
(predecessors [g node]
(->> @edges
(filter (fn [[n1 n2]] (= n2 node)))
(map first)))
(in-degree [g node] (count (predecessors g node))))
:data ssa})))

SSA Loom: Dataflow Analysis
•  For each basic block, solve system of
equations until reaching fixed point:
•  Use worklist approach

(defn dataflow-analysis
"Performs dataflow analysis. Nodes have value nil initially.”
[& {:keys [start graph join transfer]}]
(let [start (cond
(set? start) start
(coll? start) (set start)
:else #{start})]

(loop [out-values {}
[node & worklist]
(into clojure.lang.PersistentQueue/EMPTY start)
(let [in-value (join
(mapv out-values (predecessors graph node)))
out (transfer node in-value)
update? (not= out (get out-values node))
out-values (if update?
(assoc out-values node out)
out-values)
worklist (if update?
(into worklist (successors graph node))
worklist)]

(loop [out-values {}
[node & worklist]
(into clojure.lang.PersistentQueue/EMPTY start
(let [in-value (join
(mapv out-values (predecessors graph node)))
out (transfer node in-value)
update? (not= out (get out-values node))
out-values (if update?
(assoc out-values node out)
out-values)
worklist (if update?
(into worklist (successors graph node))
worklist)]
(if (seq worklist)
(recur out-values worklist)
out-values)))))

SSA Loom: Global Availability
(defn global-cse
[ssa]
(let [{graph :graph node-data :data}
(ssa->loom (:blocks ssa) ssa-nodes-fn ssa-edges-fn)
start (:start-block ssa)]
(letfn [(pure? [instr] (contains? instr :refs))
(global-cse-join
[values]
(if (seq values)
(apply set/intersection values)
#{}))
(global-cse-transfer
[node in-value]
(into in-value
(map :refs (filter pure? (node-data node)))))]

SSA Loom: Global Availability
(defn global-cse
[ssa]
(let [{graph :graph node-data :data}
(ssa->loom (:blocks ssa) ssa-nodes-fn ssa-edges-fn)
start (:start-block ssa)]
(letfn [(pure? [instr] (contains? instr :refs))
(global-cse-join
[values]
(if (seq values)
(apply set/intersection values)
#{}))
(global-cse-transfer
[node in-value]
(into in-value
(map :refs (filter pure? (node-data node)))))]
(dataflow-analysis :start start :graph graph
:join global-cse-join
:transfer global-cse-transfer))))

• Reaching definitions

• Liveness analysis (dead code elimination)

• Available expressions

• Constant propagation

• Other Applications:

o  Erdős number

o  Erdős number
o  Spread of information in systems (e.g. taint)

My Experience
• Intuitive way to implement algorithms
functionally
• Some mental overhead of transforming data
structures

Open Questions
• How general should a graph API be?

Open Questions
• How general should a graph API be?
• How feature-rich should a graph API be?

Loom and Graphs in Clojure

Recommended

Recommended

More Related Content

What's hot

What's hot (17)

Similar to Loom and Graphs in Clojure

Similar to Loom and Graphs in Clojure (20)

More from Aysylu Greenberg

More from Aysylu Greenberg (20)

Recently uploaded

Recently uploaded (20)

Loom and Graphs in Clojure