Democratizing Data at Airbnb
•
30 likes•17,895 views
GraphConnect Europe 2017 Chris Williams & John Bodley, Airbnb
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Democratizing Data at Airbnb
Similar to Democratizing Data at Airbnb (20)
More from Neo4j
More from Neo4j (20)
Recently uploaded
Recently uploaded (20)
Democratizing Data at Airbnb
- 1. Democratizing Data at Airbnb CHRIS WILLIAMS / JOHN BODLEY / MAY 11, 2017
- 2. Airbnb connects people to unique travel experiences
- 3. The problem
- 4. tribal knowledge |ˈtrībəl ˈnäləj | noun Tribal knowledge is any unwritten information that is not commonly known by others within a company
- 5. Relying on tribal knowledge stifles productivity
- 6. As Airbnb grows so do the challenges around the volume, complexity, and obscurity of data
- 7. In a large and complex organization, with a sea of data resources, users struggle to find the right data
- 8. Data is often siloed, inaccessible, or lacks context
- 9. I’m a recovering Data Scientist who wants to democratize data, automate common workflows, surface relevant information, and provide context
- 10. Tables in our Hive data warehouse 200k
- 11. > 10,000 Superset charts and dashboards > 6,000 Experiments and metrics > 6,000 Tableau workbooks and charts > 1,500 Knowledge posts Data resources Beyond the data warehouse
- 12. With many more data sources and data types to love
- 14. > 3,500 Airbnb employees
- 15. Portland San Francisco Los Angeles Toronto New York Miami Sao Paulo Dublin London Paris Barcelona Berlin Milan Copenhagen New Delhi Seoul Beijing Tokyo Sydney Singapore Washington, DC > 20 Offices around the world
- 16. The mandate
- 17. To democratize data and empower Airbnb employees to be data- informed by aiding with data exploration, discovery, and trust
- 18. The concept
- 19. Search…
- 20. It should be fairly evident what we feed into the search indices
- 22. But are we missing something?
- 24. The relevancy of relationships Nodes and relationships have equal standing created consumedSpoke 3
- 32. The construction
- 34. Databases 6 APIs 4 Airflow DAG 1 We leverage all these data resources to build a graph in Hive comprising of nodes and relationships The workflow is run everyday though the graph is left to soak to prevent flickering
- 36. Addressing graph flickering The issue is certain types of relationships are sporadic in nature causing the graph to flicker
- 37. Persistent vs. transient relationships Persistent relationships represent a snapshot in time createdSpoke 3
- 38. Persistent vs. transient relationships Transient relationships represent events which are somewhat sporadic in nature M Tu W Th F consumedSpoke 3
- 39. The winding data path Airflow Data transfer Python Graph datastore neo4j-driver Python Neo4j driver Neo4j Graph database GraphAware Neo4j/Elasticsearch plugin Elasticsearch Search engine Flask Python web framework Hive Data warehouse
- 40. The winding data path Airflow Data transfer Python Graph datastore neo4j-driver Python Neo4j driver Neo4j Graph database GraphAware Neo4j/Elasticsearch plugin Elasticsearch Search engine Flask Python web framework Hive Data warehouse
- 41. The winding data path Airflow Data transfer Python Graph datastore neo4j-driver Python Neo4j driver Neo4j Graph database GraphAware Neo4j/Elasticsearch plugin Elasticsearch Search engine Flask Python web framework Hive Data warehouse
- 42. The winding data path Airflow Data transfer Python Graph datastore neo4j-driver Python Neo4j driver Neo4j Graph database GraphAware Neo4j/Elasticsearch plugin Elasticsearch Search engine Flask Python web framework Hive Data warehouse
- 43. The winding data path Airflow Data transfer Python Graph datastore neo4j-driver Python Neo4j driver Neo4j Graph database GraphAware Neo4j/Elasticsearch plugin Elasticsearch Search engine Flask Python web framework Hive Data warehouse
- 44. The winding data path Airflow Data transfer Python Graph datastore neo4j-driver Python Neo4j driver Neo4j Graph database GraphAware Neo4j/Elasticsearch plugin Elasticsearch Search engine Flask Python web framework Hive Data warehouse
- 45. The winding data path Airflow Data transfer Python Graph datastore neo4j-driver Python Neo4j driver Neo4j Graph database GraphAware Neo4j/Elasticsearch plugin Elasticsearch Search engine Flask Python web framework Hive Data warehouse
- 46. The winding data path Airflow Data transfer Python Graph datastore neo4j-driver Python Neo4j driver Neo4j Graph database GraphAware Neo4j/Elasticsearch plugin Elasticsearch Search engine Flask Python web framework Hive Data warehouse
- 47. The winding data path Airflow Data transfer Python Graph datastore neo4j-driver Python Neo4j driver Neo4j Graph database GraphAware Neo4j/Elasticsearch plugin Elasticsearch Search engine Flask Python web framework Hive Data warehouse
- 48. The winding data path Airflow Data transfer Python Graph datastore neo4j-driver Python Neo4j driver Neo4j Graph database GraphAware Neo4j/Elasticsearch plugin Elasticsearch Search engine Flask Python web framework Hive Data warehouse
- 49. The winding data path Airflow Data transfer Python Graph datastore neo4j-driver Python Neo4j driver Neo4j Graph database GraphAware Neo4j/Elasticsearch plugin Elasticsearch Search engine Flask Python web framework Hive Data warehouse
- 50. Logical Given our data is represented as a graph it is logical to use a graph database to store the data Nimble Performance wins when dealing with connected data versus relational databases Popular It is the world’s leading graph database and the community edition is free Integrative It integrates well with Python and Elasticsearch Why we choose Neo4j for our database The four main reasons
- 51. The Neo4j and Elasticsearch symbiotic relationship Courtesy of two GraphAware plugins Neo4j plugin Provides bi-directional integration which transparently and asynchronously replicate data from Neo4j to Elasticsearch Elasticsearch plugin Enables Elasticsearch to consult with the Neo4j database during a search query to enrich the search rankings by leveraging the graph topology
- 52. Node label hierarchy :Entity :Org :Group :User :Tableau :Workbook:Chart :Hive :Schema :Table
- 53. jane_doe (:Entity:Org:User {id: ‘jane_doe’}) (:Entity:Hive:Table {id: ‘dim_users’}) (:Entity:Tableau:Chart {id: ‘12345’}) dim_users 12345
- 54. MATCH (n:Entity:Org:User {id: ’<id>’}) USING INDEX n:User(id) RETURN n
- 55. From local to global uniqueness A mechanism to reference nodes in an abstract manner GraphAware UUID plugin Transparently assigns a globally unique UUID property to newly created elements (nodes and relationships) which cannot be changed or deleted Globally unique Enables us to uniquely identify a single node via the Entity label and UUID property which allows for parameterized queries which leads to faster query and execution times
- 56. MATCH (n:Entity {uuid: ’<uuid>’}) USING INDEX n:Entity(uuid) RETURN n
- 58. The frontend
- 59. web app
- 60. Designing the interface and user experience of a data tool should not be an afterthought
- 61. Technical data power user; the epitome of a tribal knowledge holder Daphne Data User personas Less data literate; needs to keep tabs on her team’s resources Manager Mel New employee, new team, or new to data; has no idea what’s going on Nathan New
- 62. Designing for data exploration, discovery, and trust Company dataSearch Resource details &metadata User data Group data
- 63. Company dataSearch User data Group data Resource details &metadata
- 64. Search Resource details &metadata Company dataUser data Group data Google-esque search filters Resource details & metadata Context, context, & context
- 65. Search Resource details &metadata Company dataUser data Group data Surface relationships, everything’s a link to promote exploration Metadata & consumption Description, external link, social
- 66. Column details & value distributions Table lineage Enrich metadata on the fly Search Resource details &metadata Company dataUser data Group data
- 67. Search Resource details &metadata Company dataUser data Group data
- 68. User details & metadata What they make, what they consume Search Resource details &metadata Company dataUser data Group data
- 69. Former employees also hold tribal knowledge Search Resource details &metadata Company dataUser data Group data
- 70. Group overview Search Resource details &metadata Company dataUser data Group data Thumbnails for maximum context Basic organization functionality Pinterest-like curation & suggested content
- 71. We gather over 15,000 thumbnails from Tableau, Superset, and the Knowledge Repo
- 72. Search Resource details &metadata Company dataUser data Group data Pinning flow from resource page Edit mode / draggable grid
- 73. ???? ?? Employees can feel disconnected from Company-level metrics Search Resource details &metadata Company dataUser data Group data
- 74. The technology stack Application + dependencies DOM Testing eslint enzyme mocha chai Application state Styling khan/aphrodite
- 75. The challenges
- 76. Proxy nodes Abstracting complexity where necessary while accurately modeling the data ecosystem Graph merging Non-trivial Git-like merging of graph updates Data-dense design Balancing simplicity and functionality is hard; most internal design resources are not made for data-rich apps Complex dependencies An umbrella data tool is vulnerable to changes in upstream resource dependencies The challenges
- 77. The future
- 78. Game-ification Provide content producers with a sense of value Alerts& recommendations Move from active exploration to deliver relevant updates and content suggestions Certified content Use certification to build trust and enable users to filter through a sea of stale content Network analysis Determine obsolete nodes, critical paths, lines of communication, etc. The future
- 79. The team
- 80. The Dataportal team Analytics&Experimentation Products John Bodley Software Engineer Eli Brumbaugh Experience Designer Jeff Feng Product Manager Michelle Thomas Software Engineer Chris Williams Data Visualization
- 82. Thank you
- 83. Appendix
- 84. Naturally bidirectional relationships associated Dealing with mutual relationships
- 85. Naturally bidirectional relationships associated Modeling both creates an unnecessary relationship associated
- 86. Naturally bidirectional relationships associated Most efficient solution is to use a single relationship in the many-to-one direction
- 87. CREATE TABLE nodes ( labels ARRAY<STRING>, id STRING, properties STRING )
- 89. CREATE TABLE relationships ( source STRUCT<labels:ARRAY<STRING>,id:STRING>, target STRUCT<labels:ARRAY<STRING>,id:STRING>, type STRING, properties STRING )
- 90. Efficient data retrieval Solution Create an index for every label keyed by the ID and UUID properties which in addition to index hints provides optimal node retrieval Problem Indexes provide for efficient data retrieval similar to a RDBMS primary key, however they are only defined for a single label as opposed to our tuple of hierarchical labels Restrictions and workarounds with Neo4j indexes