Ro uses Looker as its BI tool. Looker is based on the idea of analysts building Looker “models” written in its LookML language; these models describe a graphical point-and-click interface that a business-side individual can use to query the specified data sets without needing to write code.
Operations, data models additions/modifications, product offerings, and medical condition offerings are scaling far too quickly at Ro for the standard Looker development model to be sustainable without having a small army of a data team to provide the labor required. Applying our general analytic tendencies to the situation, we found that much of the creation and maintenance labor required for LookML models was repetitive and error prone. Separately, LookML lacked certain features that we were having trouble doing without, particularly around documentation and the idea of view blocks. So, now we automatically generate most of our LookML code instead.
Basic Architecture
Completely independently of Looker, we code Looker views, explores, and models in our own Ro Looker templating language. A Python engine that we’ve built turns these templates into LookML and writes the LookML files to our Looker GitHub repo. A web hook pushes the LookML from GitHub to our Looker instance. The number of lines written in our Ro Looker templating language is less than 10% of the number of lines in the LookML that is generated .
Key Features
Key features of the Ro LookML code generation system include:
Automatically add dimensions to a Looker view when a new column is added to the underlying table
We couple this with our automatic pivot table generator — when Ro rolls out a new supported health condition, for example, this means that in most cases all of the fields for the new conditions appear first in our derived tables then in Looker automatically.
Reducing our custom permissions system from 15 lines per Looker field to a single word
We do permissions on a field-by-field basis instead of model-by-model.
Explore-level documentation via Sphinx
We provide documentation specific to each explore in Looker, complete with examples and technical notes. Our documentation (written in Markdown) files are wrapped in with our Looker templating, and our documentation-generation engine uses our Looker templating files to autogenerate portions of the documentation site.
New looker concept: view blocks
Groups of Looker views can appear together in several different Looker explores. For example, our “member” block contains many views with various types of member-level information; this block appears in the majority of our Looker explores. Looker does not have any existing structure that encapsulates this idea of multiple views being grouped.
Automate organization of explore views
We use some tricks to organize views in a large explore:
Remove the need to specify dimension type
We automatically deduce this from the underlying table column’s type. We also have automated levels for timestamp dimensions with different levels of time breakouts (date, day of week, month, hour, minute, etc).
Make group labels easier/automatic
In Looker currently, each dimension/measure needs to have its group label (if any) specified explicitly. With the Ro system, we can assign group labels to field name prefixes, and any field with one of these prefixes will be added to the corresponding group automatically.
Automatic parameter lists
Currently, Looker requires that you manually specify parameters. Our system can auto-generate parameter lists from unique values in a column or unique dimensions in a dimension group.
A variety of other tricks
- Ex: all measures use the first defined drillset by default
- Ex: removing the need for things like “${TABLE}”
Further reading
This article originally appeared on our blog. You can find further articles detailing of some of the above features here and here.
