Looker Connect Training

Help Center

Documentation

Community

Cloud Certifications

Question

New LookML — Why?

  • 12 October 2016
  • 12 replies
  • 444 views
A
  • Anonymous
  • 0 replies

The IDE experience looks awesome, congrats on that.


It looks like the language decision is made. But I’m curious (and a bit surprised). Why couldn’t you do those IDE changes with YAML? It looks like the new LookML language represents the same data structures as YAML (and then the additional meaning comes from a layer above those structures). If YAML’s liberal syntax was a problem for users / your compiler, there are other standard alternatives to creating your own language.


For us, this seems to be a step away from what we love about Looker - a flat, explicit way of modeling and representing our data with the minimum of interference. A new proprietary language adds to that interference.

12 replies

chris.seymour
Rank
Userlevel 3

Hey @kallaar,


On-the-fly error checking will catch most LookML errors, but not all of them, so it is still a good idea to run the validator with the “Validate LookML” button after saving your changes. For example, an invalid field reference due to a missing join would not be surfaced in the on-the-fly checking, since it depends on interaction among LookML objects across several files. You can read more about this here: New LookML — Deep dive into the new syntax and the new IDE

K

Does this mean we need not to do “validate lookml” once after saving the view, since the syntax and other validation are realtime/on the fly?

fabio
Rank
Userlevel 7
Badge

Michael, I have just shared an unofficial LookML parser here: Parser for new LookML?

G

we are scripting in ruby, we have just one view. Sure we can try collaborating. One way I was thinking is to write out JSON with “key_$_name”, and then perform string manipulation.


if we had to write the following:


view: raw_s3_events {

 sql_table_name: default.raw_events ;;

 suggestions: no

dimension: actor_ab_segment {

 view_label: "actor"

 label: "ab_segment"

 type: string

 sql: ${TABLE}.`actor`["ab_segment"] ;;

 description: "actor=>ab_segment"

}

we could start with


view_$_raw_s3_events: {

 sql_table_name: default.raw_events ;;

 suggestions: no

dimension_$_actor_ab_segment: {

 view_label: "actor"

 label: "ab_segment"

 type: string

 sql: ${TABLE}.`actor`["ab_segment"] ;;

 description: "actor=>ab_segment"

}

And then perform a string replace.

mikhailxu
Rank
Userlevel 3

Hey @Gaurav_Shah and @john_handshake,


We don’t have any updates on this yet, we will send your feedback to our product team!

J

Hey Gaurav great to meet someone in the same boat.

we’ve been punting for now as we haven’t found time to deal with automating the conversion.

with the new releases and official deprecation perhaps it’s time for us to allocate resources though 😦

what language is your generation script written in? perhaps there are opportunities to collaborate.

G

@john_handshake were you able to move ahead of it ? facing similar problem while generating the new lookml

G

Hi @mikhailxu any updates on exposing the convertor as an api ?


generating new lookml is hard, json/yaml is easier to generate.


We have scripts that generates view based on a large schema (400 fields) and gets updated every other week.

Michael_Erasmus

@mikhailxu How about open sourcing the just the LookML parser (or at least a specification for the parser)?


We’re hoping to be able to enforce some of our own company naming conventions and best practices with a githook. Writing scripts to read a standard yaml file and walk the abstract syntax tree could make this pretty easy, which is much harder now.

mikhailxu
Rank
Userlevel 3

@john_handshake,


I believe we have efforts to expose the YAML -> LookML generator as part of our API or a script. natepickens can chime in more on the details for support timeline. I believe everything will always still work, but it may not contain the new features we will add to LookML post 4.0

mikhailxu
Rank
Userlevel 3

Hey @Maximilian_Roos,


We originally picked YAML because it was our favorite standard, there are no alternative standards because we had already picked the one we felt was the best. Unfortunately, we’ve grown beyond what it offers and need to provide something better. LookML has to evolve with the demand of what data modelers need.


Here’s some huge wins that we get with just this first release:


YAML has a lot of sharp edges, if you don’t know YAML the learning cliff is painful and brutal. We didn’t want to adopt a standard with more guard rails that sacrifice readability for error proofing. New LookML doesn’t suffer this issue.


Building a parser with YAML wasn’t sustainable. There were many ways to do the same thing in YAML(for example instantiating a collection or referencing object/parameter inheritance) We couldn’t build a rich enough IDE under these constraints, we could detect for wrong things, but are unable to suggest the right thing. We didn’t feel this was good enough and wanted the IDE to guide and help the developer as much as possible.


Finally, YAML is not a standard designed nor intended for data modeling. YAML is amazing and awesome serialization language that is super readable. YAML is fantastic for configurations, but not great for multi-file objects that build up a data model. With New LookML we’re going to keep all of the great ideas they had AND add all the ones we want.


We want LookML to be richer, more expressive, and more powerful for developers. This doesn’t exist so we simply have to build it.

J

Definitely understand some of the motivation mentioned here. We are generating (a large number of) LookML programmatically and also a bit curious about using non-standard markup language going forward.

Does looker have plan to supply tool/scripts for YAML-to-new-LookML transformation, and how long would the YAML-based syntax be supported for?


Thanks!

Reply


Terms of Service