Generating LookML from Python with lkml

  • 16 August 2019
  • 4 replies
  • 23 views

Userlevel 4

About a month ago, I released an open-source Python library called lkml that parses LookML into JSON or Python dictionaries. lkml is a fast, robustly tested parser that can be imported in Python or run directly from the command line.

I’m excited to announce that I’ve added additional functionality to lkml in v0.2.0 that does the reverse: generate LookML from Python objects. Now, lkml is a closed loop for working with LookML in Python: you can load LookML to a dictionary, modify it with code, and dump it back to LookML without any manual editing.


A couple ideas for how this could be used:



  • Keeping Looker in sync with external data dictionaries by automatically syncing changes to the description field in Looker

  • Programmatically generating dimensions and measures when new columns are added to a database table


Installation


You can install lkml with pip.


pip install lkml

Approach


Rather than building a highly specific object-oriented architecture where LookML fields are generated from Python classes (e.g. View, Explore, Dimension, etc.), I took a simpler approach. lkml follows a few concise rules to convert Python objects like dictionaries and lists to various chunks of LookML.


For example, a dictionary like this.


{
"dimension": {
"type": "number",
"label": "Unit Price",
"sql": "${TABLE}.price",
"name": "price"
}
}

Is converted to a LookML block like this.



The benefit of this approach is that you won’t have to wait for me to update or fix the library to support new features in LookML. lkml supports all LookML at time of writing and has been tested on over 160K lines of public LookML from GitHub.


The tradeoff of this approach is that it is possible to generate invalid LookML if you structure the input data incorrectly, so I’ve written thorough documentation about how lkml represents LookML in Python.


Check out the project on GitHub for details and documentation on how to parse and generate LookML in Python with lkml. Contributions and issues are welcome! If you build anything interesting, please send me a link!


4 replies

Userlevel 2

This is awesome. I will share it with my users. I know some of them were looking to automatically update lookml upon table schema change. Thank you!

Userlevel 2

Here’s an example of what you can quickly build with Josh’s awesome lkml library, this Python script automatically adds labels and descriptions saved in a CSV file to fields within LookML project files. Thanks Josh!

Userlevel 2

This is tremendous – we just went through the effort of generating the Python SDK for 3.1, and working fluently with the LookML model logic in Python is one of the next challenges.


@Katrien – one of the first things we’d try to use a package like this for is to create customized clones of various components, so that we could…



  • Identify an existing item as our template

  • Import its basic logic into native Python datatypes

  • Perform whatever transformations we want

  • Publish valid LookML for the new “clone”, directly into Looker

Hi Josh (and others), thanks for sharing!

From my current perspective I don’t immediately see the use-cases for this library, but I’m sure there are. Would you mind sharing one or two?


Thanks a lot!

Reply