The Airbnb Dataportal is an internal data resource search engine.

1. Scaling Tribal Knowledge
CHRIS WILLIAMS / JOHN BODLEY / FEB 8, 2017 / BIG DATA APPLICATION MEETUP

2. The problem

3. tribal knowledge |ˈtrībəl ˈnäləj |
noun
Tribal knowledge is any unwritten information that is not commonly
known by others within a company

4. As Airbnb grows so do the challenges around the volume,
complexity, and obscurity of data

5. In a large and complex organization, with a sea of data
resources, users struggle to find the right data

6. Data is often siloed and lacks context

7. I’m a recovering Data Scientist who wants to democratize
data; automate common workflows, surface relevant
information, and provide context

9. Tables in our Hive data warehouse
100k

10. Data resources
Beyond the data warehouse

11. > 6,000
Superset charts and
dashboards
Data resources
Beyond the data warehouse

12. > 6,000
Superset charts and
dashboards
> 5,000
Experiments and
metrics
Data resources
Beyond the data warehouse

13. > 6,000
Superset charts and
dashboards
> 5,000
Experiments and
metrics
> 4,000
Tableau dashboards
and workbooks
Data resources
Beyond the data warehouse

14. > 6,000
Superset charts and
dashboards
> 5,000
Experiments and
metrics
> 4,000
Tableau dashboards
and workbooks
> 1,000
Knowledge posts
Data resources
Beyond the data warehouse

15. With many more data sources
and data types to love

16. With many more data sources
and data types to love

17. and most importantly…

19. > 3,000 Airbnb employees

20. Portland
San Francisco
Los Angeles
Toronto
New York
Miami
Sao Paulo
Dublin
London
Paris
Barcelona
Berlin
Milan
Copenhagen
New Delhi
Seoul
Beijing
Tokyo
Sydney
Singapore
Washington, DC
> 20
Offices around the world

21. The mandate

22. To democratize data and empower Airbnb employees to be data-
informed by aiding with data exploration, discovery, and trust

23. The concept

24. Search…

25. It should be fairly evident what we feed into the search indices

28. But are we missing something?

30. The relevancy of relationships
Nodes and relationships have equal standing
created consumedSpoke 3

31. The graph
created
consumed
associated
associated
consumed
consum
ed
created
consum
ed

32. The graph
created
consumed
associated
associated
consumed
consum
ed
created
consum
ed

33. The graph
created
associated
associated
consumed
consum
ed
created
consum
ed
consumed

34. The graph
consumed
associated
associated
consumed
consum
ed
consum
ed
created
created

35. The graph
created
consumed
associated
associated
consumed
created
consum
ed
consum
ed

36. The graph
created
consumed
associated
associated consum
ed
created
consum
ed
consumed

37. The graph
created
consumed
consumed
consum
ed
created
consum
ed
associated
associated

38. The construction

40. Databases
5
APIs
3
Airflow DAG
1

41. Databases
5
APIs
3
Airflow DAG
1
We leverage all these data resources to build a graph comprising of
nodes and relationships
The Airflow DAG is run everyday and the output is stored in Hive

43. We gather over 10,000 thumbnails from the Tableau API,
Knowledge Repo database, and Superset screenshots

44. The winding data path
Airflow
Data transfer
Python
Graph datastore
py2neo
Python Neo4j
driver
Neo4j
Graph database
GraphAware
Neo4j/Elasticsearch plugin
Elasticsearch
Search engine
Flask
Python web framework
Hive
Data warehouse

45. The winding data path
Airflow
Data transfer
Python
Graph datastore
py2neo
Python Neo4j
driver
Neo4j
Graph database
GraphAware
Neo4j/Elasticsearch plugin
Elasticsearch
Search engine
Flask
Python web framework
Hive
Data warehouse

46. The winding data path
Airflow
Data transfer
Python
Graph datastore
py2neo
Python Neo4j
driver
Neo4j
Graph database
GraphAware
Neo4j/Elasticsearch plugin
Elasticsearch
Search engine
Flask
Python web framework
Hive
Data warehouse

47. The winding data path
Airflow
Data transfer
Python
Graph datastore
py2neo
Python Neo4j
driver
Neo4j
Graph database
GraphAware
Neo4j/Elasticsearch plugin
Elasticsearch
Search engine
Flask
Python web framework
Hive
Data warehouse

48. The winding data path
Airflow
Data transfer
Python
Graph datastore
py2neo
Python Neo4j
driver
Neo4j
Graph database
GraphAware
Neo4j/Elasticsearch plugin
Elasticsearch
Search engine
Flask
Python web framework
Hive
Data warehouse

49. The winding data path
Airflow
Data transfer
Python
Graph datastore
py2neo
Python Neo4j
driver
Neo4j
Graph database
GraphAware
Neo4j/Elasticsearch plugin
Elasticsearch
Search engine
Flask
Python web framework
Hive
Data warehouse

50. The winding data path
Airflow
Data transfer
Python
Graph datastore
py2neo
Python Neo4j
driver
Neo4j
Graph database
GraphAware
Neo4j/Elasticsearch plugin
Elasticsearch
Search engine
Flask
Python web framework
Hive
Data warehouse

51. The winding data path
Airflow
Data transfer
Python
Graph datastore
py2neo
Python Neo4j
driver
Neo4j
Graph database
GraphAware
Neo4j/Elasticsearch plugin
Elasticsearch
Search engine
Flask
Python web framework
Hive
Data warehouse

52. The winding data path
Airflow
Data transfer
Python
Graph datastore
py2neo
Python Neo4j
driver
Neo4j
Graph database
GraphAware
Neo4j/Elasticsearch plugin
Elasticsearch
Search engine
Flask
Python web framework
Hive
Data warehouse

53. The winding data path
Airflow
Data transfer
Python
Graph datastore
py2neo
Python Neo4j
driver
Neo4j
Graph database
GraphAware
Neo4j/Elasticsearch plugin
Elasticsearch
Search engine
Flask
Python web framework
Hive
Data warehouse

54. The winding data path
Airflow
Data transfer
Python
Graph datastore
py2neo
Python Neo4j
driver
Neo4j
Graph database
GraphAware
Neo4j/Elasticsearch plugin
Elasticsearch
Search engine
Flask
Python web framework
Hive
Data warehouse

55. The winding data path
Airflow
Data transfer
Python
Graph datastore
py2neo
Python Neo4j
driver
Neo4j
Graph database
GraphAware
Neo4j/Elasticsearch plugin
Elasticsearch
Search engine
Flask
Python web framework
Hive
Data warehouse

56. The winding data path
Airflow
Data transfer
Python
Graph datastore
py2neo
Python Neo4j
driver
Neo4j
Graph database
GraphAware
Neo4j/Elasticsearch plugin
Elasticsearch
Search engine
Flask
Python web framework
Hive
Data warehouse

57. The winding data path
Airflow
Data transfer
Python
Graph datastore
py2neo
Python Neo4j
driver
Neo4j
Graph database
GraphAware
Neo4j/Elasticsearch plugin
Elasticsearch
Search engine
Flask
Python web framework
Hive
Data warehouse

58. The winding data path
Airflow
Data transfer
Python
Graph datastore
py2neo
Python Neo4j
driver
Neo4j
Graph database
GraphAware
Neo4j/Elasticsearch plugin
Elasticsearch
Search engine
Flask
Python web framework
Hive
Data warehouse

59. Why we choose Neo4j for our database
The main reasons

60. Logical
Given our data is
represented as a graph
it is logical to use a
graph database to
store the data
Why we choose Neo4j for our database
The main reasons

61. Logical
Given our data is
represented as a graph
it is logical to use a
graph database to
store the data
Nimble
Performance wins
when dealing with
connected data versus
relational databases
Why we choose Neo4j for our database
The main reasons

62. Logical
Given our data is
represented as a graph
it is logical to use a
graph database to
store the data
Nimble
Performance wins
when dealing with
connected data versus
relational databases
Popular
It is the world’s leading
graph database and
the community edition
is free
Why we choose Neo4j for our database
The main reasons

63. Logical
Given our data is
represented as a graph
it is logical to use a
graph database to
store the data
Nimble
Performance wins
when dealing with
connected data versus
relational databases
Popular
It is the world’s leading
graph database and
the community edition
is free
Integrative
It integrates well with
Python and
Elasticsearch
Why we choose Neo4j for our database
The main reasons

64. The Neo4j and Elasticsearch symbiotic relationship
Courtesy of two GraphAware plugins

65. The Neo4j and Elasticsearch symbiotic relationship
Courtesy of two GraphAware plugins
Neo4j plugin
Provides bi-directional integration which transparently and asynchronously replicate data from
Neo4j to Elasticsearch

66. The Neo4j and Elasticsearch symbiotic relationship
Courtesy of two GraphAware plugins
Neo4j plugin
Provides bi-directional integration which transparently and asynchronously replicate data from
Neo4j to Elasticsearch
Elasticsearch plugin
Enables Elasticsearch to consult with the Neo4j database during a search query to enrich the
search rankings by leveraging the graph topology

70. The schema

71. createds t

72. (s:Entity)-[r:CREATED]->(t:Entity)

73. :Entity
:Org
:Group :User
:Superset
:Slice:Dashboard
Node label hierarchy
:Hive
:Schema :Table

74. (:Entity:Org:User {id: ‘jane_doe’})

75. (:Entity:Hive:Table {id: ‘core_data.dim_users’})

76. (:Entity:Superset:Dashboard {id: 123})

77. Efficient data retrieval and uniqueness
Restrictions and workarounds with the Neo4j schema

78. Efficient data retrieval and uniqueness
Restrictions and workarounds with the Neo4j schema
Indexes
Neo4j provides indexes for efficient data retrieval similar to a RDMS, however they are only
defined for a single label

79. Efficient data retrieval and uniqueness
Restrictions and workarounds with the Neo4j schema
Indexes
Neo4j provides indexes for efficient data retrieval similar to a RDMS, however they are only
defined for a single label
Uniqueness Constraints
Ensures that properties are unique for all nodes for a specific single label

80. Efficient data retrieval and uniqueness
Restrictions and workarounds with the Neo4j schema
Indexes
Neo4j provides indexes for efficient data retrieval similar to a RDMS, however they are only
defined for a single label
Uniqueness Constraints
Ensures that properties are unique for all nodes for a specific single label
GraphAware UUID plugin
Transparently assigns a globally unique UUID property to newly created elements which
cannot be changed or deleted

81. (:Entity {uuid: ‘<UUID>’})

82. The web app

83. The web app

84. Designing the user experience and interface of
a data tool should not be an afterthought

85. Designing the user experience and interface of
a data tool should not be an afterthought

86. Technical data power
user; the epitome of a
tribal knowledge
holder
Daphne Data
User personas
Less data literate;
needs to keep tabs on
her team’s resources
Manager Mel
New employee or
new team; has no idea
what’s going on
Nathan New

87. Designing for data exploration, discovery, and trust
Company dataSearch
Resource details
&meta-data
User data Group data

88. Company dataSearch User data Group data
Resource details
&meta-data

89. Company dataSearch User data Group data
Resource details
&meta-data

90. Search
Resource details
&meta-data
Company dataUser data Group data

91. Search
Resource details
&meta-data
Company dataUser data Group data
Google-esque search filters

92. Search
Resource details
&meta-data
Company dataUser data Group data
Google-esque search filters
Resource details & meta-data

93. Search
Resource details
&meta-data
Company dataUser data Group data
Google-esque search filters
Resource details & meta-data
Context, context, & context

94. Search
Resource details
&meta-data
Company dataUser data Group data

95. Search
Resource details
&meta-data
Company dataUser data Group data
Description, external link, social

96. Search
Resource details
&meta-data
Company dataUser data Group data
Meta-data & consumption
Description, external link, social

97. Search
Resource details
&meta-data
Company dataUser data Group data
Surface relationships,
everything’s a link to promote
exploration
Meta-data & consumption
Description, external link, social

98. Column details & value distributions
Table lineage
Enrich meta-data on the fly
Search
Resource details
&meta-data
Company dataUser data Group data

99. Column details & value distributions
Table lineage
Enrich meta-data on the fly
Search
Resource details
&meta-data
Company dataUser data Group data

100. Search
Resource details
&meta-data
Company dataUser data Group data

101. Search
Resource details
&meta-data
Company dataUser data Group data

102. Search
Resource details
&meta-data
Company dataUser data Group data

103. Search
Resource details
&meta-data
Company dataUser data Group data

104. User details &
meta-data
Search
Resource details
&meta-data
Company dataUser data Group data

105. User details &
meta-data
What they make,
what they consume
Search
Resource details
&meta-data
Company dataUser data Group data

106. Former employees also
hold tribal knowledge
Search
Resource details
&meta-data
Company dataUser data Group data

107. Search
Resource details
&meta-data
Company dataUser data Group data

108. Group overview
Search
Resource details
&meta-data
Company dataUser data Group data

109. Group overview
Search
Resource details
&meta-data
Company dataUser data Group data
Pinterest-like curation

110. Group overview
Search
Resource details
&meta-data
Company dataUser data Group data
Basic organization functionality
Pinterest-like curation

111. Search
Resource details
&meta-data
Company dataUser data Group data
Curated + Popular content

112. Search
Resource details
&meta-data
Company dataUser data Group data
Curated + Popular content
Thumbnails for maximum context

113. Search
Resource details
&meta-data
Company dataUser data Group data
Pinning flow from resource page
Edit mode / draggable grid

114. Search
Resource details
&meta-data
Company dataUser data Group data
Pinning flow from resource page
Edit mode / draggable grid

115. ???? ??
Employees can feel disconnected
from Company-level metrics
Search
Resource details
&meta-data
Company dataUser data Group data

116. The technology stack
Application +
dependencies
DOM Testing
eslint
enzyme
mocha
chai
Application
state
Styling
khan/aphrodite

117. The challenges

118. The challenges

119. The challenges
Complex
dependencies
An umbrella data tool is
vulnerable to changes
in upstream resource
dependencies

120. The challenges
Complex
dependencies
An umbrella data tool is
vulnerable to changes
in upstream resource
dependencies
Data-dense design
Balancing simplicity and
functionality is hard;
most internal design
resources are not made
for data-rich apps

121. The challenges
Complex
dependencies
An umbrella data tool is
vulnerable to changes
in upstream resource
dependencies
Data-dense design
Balancing simplicity and
functionality is hard;
most internal design
resources are not made
for data-rich apps
Graph merging
Non-trivial Git-like
merging of (daily or real-
time) graph updates

122. The challenges
Complex
dependencies
An umbrella data tool is
vulnerable to changes
in upstream resource
dependencies
Data-dense design
Balancing simplicity and
functionality is hard;
most internal design
resources are not made
for data-rich apps
Graph flickering
Transient relationships
should not create
“flickering” artifacts
Graph merging
Non-trivial Git-like
merging of (daily or real-
time) graph updates

123. The future

124. The future

125. The future
New resource types
A/B tests, logging
schemas, SQL queries,
etc.

126. The future
New resource types
A/B tests, logging
schemas, SQL queries,
etc.
Certified content
Use certification to build
trust and enable users to
filter through a sea of
stale content

127. The future
New resource types
A/B tests, logging
schemas, SQL queries,
etc.
Certified content
Use certification to build
trust and enable users to
filter through a sea of
stale content
Alerts&
recommendations
Move from active
exploration to deliver
relevant updates and
content suggestions

128. The future
New resource types
A/B tests, logging
schemas, SQL queries,
etc.
Certified content
Use certification to build
trust and enable users to
filter through a sea of
stale content
Game-ification
Provide content
producers with a sense
of value
Alerts&
recommendations
Move from active
exploration to deliver
relevant updates and
content suggestions

129. The team

130. The Dataportal team
Analytics&Experimentation Products
John Bodley
Software Engineer
Eli Brumbaugh
Experience Designer
Jeff Feng
Product Manager
Michelle Thomas
Software Engineer
Chris Williams
Data Visualization

131. The Dataportal team
Analytics&Experimentation Products
John Bodley
Software Engineer
Eli Brumbaugh
Experience Designer
Jeff Feng
Product Manager
Michelle Thomas
Software Engineer
Chris Williams
Data Visualization

132. Thank you
john.bodley@airbnb.com
chris.williams@airbnb.com