User Guide

Getting Started

Help Center

Documentation

Community

Training

Certification

Introducing lkml, a speedy LookML parser in pure Python

joshtemple
Userlevel 4

I’m excited to debut lkml, a pure Python parser for LookML.


You can run lkml from the command line (it will output a JSON string) or import it as a Python package (it will output a nested dictionary).


@fabio at Looker already built a fantastic, open-source parser in Node, but I find that the Node dependency sometimes makes it difficult to adopt for Python-based workflows that data people are more familiar with.


I decided to embrace the challenge of hand-writing a parser in Python without any external dependencies or libraries.


I didn’t have access to the actual grammar used for LookML, so I reverse engineered it myself. Don’t worry, I’ve tested lkml on over 160K lines of public LookML I downloaded through the GitHub API and it works like a charm!


lkml is fast too. Excluding file I/O, it parses a typical file in a handful of milliseconds. Based on my tests, it’s at parity with the Node parser.


Lastly, lkml has a full unit test suite with CI.


Installation


lkml is up on pip, so the following command will install it


pip install lkml

Please also check out the project on GitHub for a more detailed description and instructions. Contributions and issues are welcome! If you build anything interesting with this parser, please send me a link!

15 replies

izzy
Userlevel 7
Badge +1

Huge, Josh! That’s commitment.


Can’t wait to see what gets built out now that the Python-speaking masses (like me 🐍 ) have easy access to this.

Y
Userlevel 2

Hi Josh. That’s a neat work. How are you using it? I remember some of my users were interested in doing somewhat reverse: generate/update LookML if the table structure changes.

joshtemple
Userlevel 4

Thanks! I released it so that others could use it to build or improve linters, syntax highlighting, CI tools, and support more Python-based “built on Looker” projects. There are a couple other open-source projects that I know of that are considering implementing it behind the scenes.


There’s no functionality to do the reverse at the moment (“dump” instead of “load”), but the project is definitely open for contributions!

Carl_Anderson
Userlevel 3

@joshtemple This is great. I’ve tested this with lookml-tools and I will be swapping out the node LookML parser with this Python lkml parser in the next release – hopefully in the next week

Carl

Carl_Anderson
Userlevel 3

lookml-tools 2.0.0 is released and it now uses lkml as its LookML parser.


R

amazing work @joshtemple I have been wanting this for a while ! works like a charm for me so far.

joshtemple
Userlevel 4

@yuriy, with version 0.2.0, lkml now supports doing the reverse: generating LookML from Python objects. Hope this is useful for your users!

JeffH
Userlevel 3

I am familiar with Looker and with Python. Why would I want to use this parser? What are some of the primary use cases that this enables? I am trying to figure out how I would use/employ this. Thanks!

LStanevich
Userlevel 2

Jeff, I gave an example of how I could envision using it here: Generating LookML from Python with lkml

joshtemple
Userlevel 4

@JeffH, a parser is a foundational building block for building open-source developer tools in Python for LookML. Having a Python representation of LookML structure allows you to do things like:



  • Write programmatic tests of your LookML (linting)

  • Map out the structure of your LookML with hierarchies and dependencies

  • Modify LookML automatically when changes are made elsewhere

  • Generate LookML from other schemas or data representations (see @LStanevich’s example)


Granted, these are all fairly “advanced” use cases, but I’m excited to see how people will build off of this parser and expand the ecosystem of open-source developer tools for Looker. Let me know if you think of other ideas!

JeffH
Userlevel 3

Thanks!

I think it would be useful to have a Database vs. LookML View complete compare, to identify missing LookML dimension fields. When you add fields to your database, they don’t automatically show up in your LookML views.

Also, it would be nice to have a function to alphabetize your dimension field and measure field sections.

I think these things would be possible with this, right?

joshtemple
Userlevel 4

Those are fantastic ideas, and yes, both should be possible with this tool!

chelsbells

Hey @joshtemple! Just started playing with this parser and am ridiculously happy to have found it. Quick question–is there a way to preserve code that’s commented out when loading/dumping?

Thanks!

joshtemple
Userlevel 4

@chelsbells , I’m so happy to hear that! lkml isn’t set up to understand white space or comments, they’re skipped over during parsing. I have an issue open for that here: https://github.com/joshtemple/lkml/issues/21, but seems to require some larger changes than I have time for currently. Feel free to dive in to the code and give it a shot! I can answer any questions you might have over at that issue.

Zoran_Stipanicev

Hi @joshtemple ,

Thank you for this great project!

I would like to update (hide) a large number of fields with preserving all the comments in the file and possibly the order of fields in the file.
I saw that you closed the issue related to comments and could not understand how to actually achieve it.
I read this page https://lkml.readthedocs.io/en/latest/advanced.html and it’s not clear.
Is there an example of what is required to preserver comments (and field order would be great).

Thanks!

Reply