LinkedIn
Twitter
Good performance is a key component to providing a good user experience, so we are continuously investing in improvements to the end-to-end performance of Looker. The latest enhancement that we’ve made in this area is the development of a completely new LookML compiler, now generally available across all Looker instances.
What does the LookML compiler do?
Looker’s LookML compiler, also called New Looker Runtime, is a core component of the Looker application. It is the backend engine that validates your LookML and translates your LookML into SQL queries when users are exploring data or working in dashboards. Specifically, the LookML compiler is responsible for the following:
-
Compiling a model: Parsing and loading a model into cache, which occurs every seven days or after any changes have been made.
-
Validation: Parsing LookML code and front-end content and identifying any errors.
-
Generating metadata for an Explore: Identifying and exposing views, fields, and other metadata from the LookML model to users within the Explore interface.
-
Writing SQL for a query: Identifying the “active” fields for a query and writing SQL to the database accordingly. Active fields are those that are included in a query from an Explore or dashboard.
What are the benefits of the new LookML compiler?
The New LookML compiler has feature parity with the original compiler but provides improved performance in a number of application areas. With this new compiler, some or all of the following activities should be faster:
-
LookML validation
-
Content validation
-
Explores and dashboard loads
-
SQL query generation (now using the Apache Calcite SQL writer)
Through participation in beta testing for this feature, we have observed that some customers have seen orders of magnitude performance improvements when running dashboards with the new compiler compared to running those same dashboards without the new compiler. And in aggregate across all customers using the new LookML compiler, we are seeing a reduction in query overhead timing. Below, you can see the 90th percentile for Looker query overhead by week over eight complete weeks spanning June through mid-July, 2022. The blue line represents the 90th percentile for queries using the new LookML compiler, while the red line represents the 90th percentile for queries using the old LookML compiler.
In addition to some potential performance gains, the new LookML compiler also brings with it improved LookML validation, providing you with warnings that may not have been surfaced previously. In the short term, this may result in the identification of errors in your LookML that previously had not been caught but would have resulted in errors at query time. This includes errors such as:
-
Instances where primary keys have not been defined for a view
-
Naming convention misalignment on a parameter
-
Legacy filters on data selection fields for PDTs
-
Enhanced validation for Liquid
This will help ensure that your LookML code is clean and has fewer errors impacting your front-end users.
We have extensively tested this new compiler, but if you do experience any issues, there is a a Legacy Toggle that you can turn on to enable the original LookML compiler for backwards compatibility.
What’s Next?
This project continues the evolution of Looker’s architecture to provide enterprise security, scale and robustness. Previous projects introduced Apache Calcite to generate SQL and optimize queries using Aggregate Awareness. In addition to improving application performance for all of our customers and users, this rebuild provides us with more modular, maintainable backend code and immutable-first guarantees to allow for stateless, reusable objects. This will let us continue to develop new product features and capabilities and make Looker more scalable and robust in the future.
