Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

JOIN 2019: extending LookML development

304 views

Published on

This is a talk given by Fabio Beltramini and Carl Anderson at the Looker JOIN conference (San Francisco, November 6th, 2019).

Here, we review the ecosystem of open-sourced tooling around LookML development.
We cover:

- Parsers: Interpret LookML text into a more usable/structured format for other software.

- Linters: Evaluate LookML against a set of rules, primarily to automate feedback to developers making LookML changes.

- Visualizers & Informational: Provide alternate presentations of LookML contents for informational or navigational purposes.

- Generators: Automate the writing of LookML from some other process or format.

- Testers: Evaluate LookML within the context of a connection, Looker instance, or data, primarily to automate feedback to developers making LookML changes, or, to monitor for breaking external changes.

A blog post as a companion to this talk was published the same day at:
https://medium.com/@leapingllamas/2019-lookml-open-source-state-of-the-union-d0470012fed0

Published in: Data & Analytics
  • Login to see the comments

  • Be the first to like this

JOIN 2019: extending LookML development

  1. 1. Extending LookML Development JOIN 2019
  2. 2. Fabio Beltramini Customer Success @ Looker © 2019 Looker. All rights reserved. Confidential. Carl Anderson Sr. Director, Data Science; Head BI Data Eng @ WW (The new Weight Watchers)
  3. 3. This session is for visionaries Welcome! © 2019 Looker. All rights reserved. Confidential.
  4. 4. Contents © 2019 Looker. All rights reserved. Confidential. ● Background ○ Fabio: LookML background ○ Carl: Development best practices background ● What already exists / “State of the Union” ○ rundown of existing / in-progress OSS ● What can you do next? ○ What could you contribute to existing / in-progress projects? ○ What categories of tools could you build? ○ “What would you need from Looker?”
  5. 5. But first, some background... © 2019 Looker. All rights reserved. Confidential.
  6. 6. LookML Development Highlights 20162012 New LookML introduced © 2019 Looker. All rights reserved. Confidential. YAML LookML LookML unit tests in closed beta 2019 Present Content Validator out of Beta
  7. 7. From the beginning, LookML tapped in to development best practices, and continues to develop rapidly … but we can accelerate and extend this trajectory © 2019 Looker. All rights reserved. Confidential.
  8. 8. A well-traveled path 1) Official trajectory is fast, but limited 2) What to do if my use case isn’t covered? 3) Leverage platform extensibility with open-source © 2019 Looker. All rights reserved. Confidential.
  9. 9. Extending LookML Development: The State of the Union © 2019 Looker. All rights reserved. Confidential.
  10. 10. © 2019 Looker. All rights reserved. Confidential. Generators Visualizers / Informational Parsers Linters Testing
  11. 11. Parsers Accept LookML strings/projects and easily interpret them
  12. 12. /joshtemple/lkml Python LookML Parser & Generator MIT License Josh Temple Milk Bar Node.js is a trademark of Joyent, Inc. and is used with its permission. We are not endorsed by or affiliated with Joyent. "Python" and the intertwined snake graphic are trademarks of the Python Software Foundation and are used in accordance with its policy. The Github Mark is a trademark of GitHub, Inc and is used in accordance with its policy. /fabio-looker/node-lookml-parser Node.js LookML Parser MIT License Fabio Beltramini Looker
  13. 13. Generators Generate LookML from some other input
  14. 14. ● Works with in-memory Python arrays/objects ○ In particular, with the format from its parser ● Enables Parse ⇾ Mutate ⇾ Serialize workflows lkml /joshtemple/lkml MIT License Josh Temple Milk Bar
  15. 15. ● Dynamically modify existing LookML ○ Add new key:values ○ Search for key and modify value ● Uses lkml parser LookML Tools / Updater /ww-tech/lookml-tools Apache 2.0 License Carl Anderson WW
  16. 16. LookML Tools / Updater /ww-tech/lookml-tools Apache 2.0 License Carl Anderson WW
  17. 17. ● Build up LookML objects using Python methods LookML Gen /symphonyrm/lookml-gen Apache 2.0 License Joe Schmid SymphonyRM
  18. 18. Honorable Mentions Generators that are unpublished or unlicensed, but usable or instructive Ro LookML Generator Sami Yabroudi, Ro FABS Jesse St. Charles, Frame.ai JSON ⇾ LookML Leigha Jarret, Looker EAV Generator Fabio Beltramini, Looker
  19. 19. Visualizers & Informational Understand your LookML through alternate presentations
  20. 20. ● Generates a “network diagram” of model-explore-views ● Highlight orphaned views LookML Tools / Grapher /ww-tech/lookml-tools Apache 2.0 License Carl Anderson WW
  21. 21. LookML Tools / Grapher
  22. 22. LookML Tools / Grapher
  23. 23. ● Analyzes LookML together with i__looker data ● Helps identify stale & unused fields & explores ● Reports viewed via the command line Henry /looker-open-source/henry MIT License Joseph Axisa Looker
  24. 24. Honorable Mentions Visualizers that are unpublished or unlicensed, but usable or instructive /marcellooker/model_diagram_web_app Marcell Babai, Looker /leighajarett/Looker_Explore_ERD_Generator Leigha Jarret, Looker
  25. 25. Linters Prescriptive analysis of LookML during code development
  26. 26. ● Includes 9 rules ● Output usable by CircleCI out of the box ● Written in Python, can be forked to add rules Lookmlint /WarbyParker/lookmlint Apache 2.0 License Ryan Tuck Warby Parker
  27. 27. ● Includes 12 rules ● Outputs markdown files into the repo/Looker IDE ● Written in JS, and accepts custom rules via external JS, or via in-project LISP expressions in addition to forking Look At Me Sideways /looker-open-source /look-at-me-sideways MIT License Joseph Axisa & Fabio Beltramini
  28. 28. Look At Me Sideways
  29. 29. ● Includes 10 rules ● Outputs to a CSV file, or to a table in BigQuery ● Written in Python, can be forked to add rules LookML Tools / Linter /ww-tech/lookml-tools Apache 2.0 License Carl Anderson WW
  30. 30. LookML Tools / Linter
  31. 31. Testers Test LookML against an actual instance and/or data
  32. 32. ● Runs in CI pipeline ● Queries all explores & fields to find errors Spectacles /dbanalyticsco/spectacles MIT License Dylan Baker, Josh Temple
  33. 33. Contributing
  34. 34. © 2019 Looker. All rights reserved. Confidential. Generators Visualizers / Informational Parsers Linters Testing Existing Projects
  35. 35. © 2019 Looker. All rights reserved. Confidential. Generators Visualizers / Informational Parsers Linters Testing Something New - What’s Missing? ?
  36. 36. What will you share?

×