Legacy Feature Notice: Old Style YAML LookML (Looker 4.4+)

[Brett_Sauve]

In Looker 4.0, we introduced the option of “new LookML”, an improved LookML syntax that addressed some of the challenges with the original, YAML-style LookML. It has many benefits that you can read about in New LookML — Why?

Instances initiated on Looker 4.4+ will only have one syntax option for new projects: the new, refreshed one. Looker instances that have run on prior Looker releases have the option to enable a legacy feature that will allow new projects to be created with the YAML-style LookML (although we don’t recommend it for most cases; it will eventually be fully deprecated).

The legacy feature will be enabled by default for instances that have run releases prior to Looker 4.4. You can read more about legacy features here. Existing projects will still function when this legacy feature is removed but you will not be able to create new YAML LookML projects.

Syntax Differences

In general new LookML parameters and their usage remain the same. There are some differences, such as:

• A few parameters have been renamed to be clearer or more consistent with other naming
• Some structural/container parameters have been removed because they didn’t provide any real benefit in the context of new LookML
• True/false has been changed to yes/no to be more user friendly

You can get a more complete list of changes in The New LookML IDE Shows New and Pre-Existing Parameters.

Enabling Old LookML with a Legacy Feature

This change is associated with the YAML-based LookML legacy feature (for more details see the Legacy Features docs page). By default, from 4.4 to 4.8 this feature is checked, which means that no change in behavior will occur. Starting in 4.10, the default is that the feature is unchecked so new projects will be in new LookML. After you transition to new LookML, un-check this legacy feature and click the Update Legacy Features button.

Example of Old (YAML) LookML

Old LookML was based on a data structure called “YAML”. It looked like this:

- view: orders
  fields:
  - dimension: id
    primary_key: true
    type: number
    sql: ${TABLE}.id

In this style of LookML indentations are used to indicate structure and must be exactly correct.

Example of New LookML

New LookML uses curly braces (the { and } symbols) to indicate structure, which means that indentations are no longer critical. It looks like this:

view: orders {
  dimension: id {
    primary_key: yes
    type: number
    sql: ${TABLE}.id ;;
  }
}

New LookML also comes with a new IDE (the place where you write LookML code), which gives help as you type to prevent errors and make it easier to know what valid options are available. The IDE is discussed more in New LookML — Deep dive into the new syntax and the new IDE.

Switching to New LookML

Due to its many benefits, Looker will eventually require all customers to use new LookML. Therefore, we encourage all customers to transition to new LookML when they have a chance. There are some steps required to make the change, which you can read about in New LookML — How to convert a “YAML” LookML project.

[chris_laumans]

In trying to migrate to the new LookML we ran into some issues, in particular related to the use of “extends”. The new LookML results in validation errors (even though everything works fine). Because of these “false positives” we are not yet able to migrate to new LookML.
I’ve had a call with Looker engineering about this, and they seem to be aware of the problems, but I have yet to hear when this issue will be solved. Any updates on this? Seems important to solve such issues before the old syntax is deprecated.

[lloydtabb]

Hi Chris,

What is the bug you are seeing? We use extends extensively in with the new LookML syntax. The nuance is that you may need to add an include statement the fields of the base view are known to the extended view.

include "users.view"

view: my_users {
   extends: [users]
 ...
}

Glad to help work through any issues you might have here.

[chris_laumans]

Hi Lloyd,

Thanks for jumping in on this. The problem we’re experiencing is related to extending models not views. I’m going to do my best to detail exactly what is happening, please let me know if something isn’t clear.

We’re getting a lot of “Could not find a field named” errors. However, in the old syntax we’re not experiencing any problems and if I test the code locally with the new syntax (ignoring the errors), everything works fine. But because of the errors, we can’t commit the code to production. Here is what we’re doing that is causing this problem (I’ll also give an example following the explanation):

• In our database we have a lot of “configuration” tables, which we join to very frequently from numerous other tables (let’s call these “transaction tables”). For example, we have a configuration table called “account” that has information about our merchants. In almost all other tables, we have an accountid on which we join to the account table.
• To keep our LookML clean, what we did is we created a “basic model” which contains all frequently used joins to tables such as account. This way, every other model can extend the basic model instead of always needing to specify the joins again.
• One thing to mention (not sure how relevant it is to the problem) is that all “transaction tables” have the same alias, since this the only way to reuse the joins in the basic model across different tables. This is specified by adding the view_name parameter to the explores in the basic model.

Here is an example of what this looks like. Here is part of the “basic model”:

explore: paymentmethod {
  extension: required
  view_name: datatable

  join: paymentmethod {
    from: paymentmethod
    sql_on: ${datatable.paymentmethodid} = ${paymentmethod.paymentmethodid} ;;
    relationship: many_to_one
    type: inner
    fields: [paymentmethodcode, paymentmethodcount]
    view_label: "Payment method"
}

explore: account {
  extension: required
  view_name: datatable

  join: account {
    from: account
    sql_on: ${datatable.accountid} = ${account.accountid} ;;
    relationship: many_to_one
    type: inner
    fields: [accountcode, accountcount]
    view_label: "Account"
  }

Here is what an example of what our other models look like when extending the basic model. Notice, that below are 2 tables with different names (a daily summary and a weekly summary) that use the same extensions.

include: "basic_model.model"
explore: conversiondailysummary {
  from: conversiondailysummary
  label: "1. Daily"

  extends: [account, paymentmethod]
}

explore: conversionweeklysummary {
  from: conversionweeklysummary
  label: "2. Weekly"

  extends: [account, paymentmethod]
}

To be able to use the same extension across multiple tables, we use the “view_name: datatable” parameter in the basic model to ensure all “transaction tables” get the same alias.

With the new LookML syntax the above code gives us the following error:

Could not find a field named "datatable.companyaccountid"
basic_model:42

We get this error for every field that originates from explores in the basic model. (76 errors in total)

However, as mentioned, everything “works” except the Validator is just producing these errors and isn’t letting us commit code.

I appreciate you looking into this

[lloydtabb]

We’re having a hard time reproducing your error, but my guess is a subtle change should fix the problem (it is the pattern I use when sharing code between models).

It seems like the errors you are getting are because basic_model.model is not a valid model and the validator is reporting error (maybe I’m wrong here, but my suggestions below should help anyway).

From the git repo (either pulled locally or from a web UI), make a new file: basic_model.base.lkml. Unfortunately there is now way to create this file within Looker itself. Once the file exists, it will show up in an “other” section in the file list.

Copy the contents of basic_model.model.lkml to basic_model.base.lkml and delete the basic_model.model.lkml file.

Change all the include references to

include "basic_model.base.lkml"

This should eliminate your model errors.

We’ll reach out separately to make sure you’ve got this working.

[chris_laumans]

Although I’m in contact with support I’m going to post here to keep this issue under attention.

We are continuing to experience this error and are unable to migrate to new LookML. I’ve followed Lloyds instructions, but that hasn’t helped unfortunately.

Zach and I talked through the issue a month ago and he was able to reproduce it when running v4.0 (which we were using at the time) but not able to reproduce it when running v4.2 or 4.4. We therefore assumed that it has been solved somewhere the release (although Zach couldn’t find a specific fix related to the issue).

We’ve now upgraded to v4.4 and I immediately tried converting to the new LookML, however unfortunately we’re still experiencing the error.

[dion1]

Hi Chris,

What are the additional errors you are getting when trying convert to New LookML?

Could not find a field named "datatable.companyaccountid" basic_model:42

[chris_laumans]

I hate to use this public forum to escalate this issue, but it’s starting to frustrate me that we’ve had this bug for more than 3 months and although we’ve have been promised fixes in at least 3 version updates none of them have actually solved the bug. My hopes were high after the 4.10 release notes seemed to mention bug fixes to our issue, but unfortunately after updating to version 4.10.13 nothing has been solved and we are still not able to upgrade our LookML to the new syntax. Can we please get this sorted out?

[lloydtabb]

Hey Chris. I use this, pattern for our internal Looker, encountered the problem and it has been fixed. Sorry for the delay, It looked just like another bug which is why we thought we had fixed it.

Fix join overrides in extends #477
Merged dmarcotte merged 1 commit into master from fix-join-overrides 19 days ago

Should be part of 4.12. We can verify for you. @Zam

[chris_laumans]

Thanks Lloyd. I do hope that fixes it, but I would appreciate if this could be double-checked, since the fixes promised in 4.8.9, 4.8.12, and 4.10 did not actually work. So I hope you understand my skepticism.

In the ticket that I have open on this topic, last month I sent support a zipped project file to reproduce the error, I hope this gives you guys enough to get to the bottom of it.

[Zam]

@chris_laumans the LookML we have was validating through the most recent fix that did not work for you, thus we don’t think it’s complete.

We emailed you on a thread where you can provide the full set of LookML which will help us test the fix holistically.

[tig]

Just added that in 4.10 and above, the default setting for the legacy feature is unchecked — so new projects will be in new LookML.