Looker Data Dictionary

Dawid
Participant V

Let’s talk about the Data Dictionary!

It would be good to gather feedback and requests in one place. I think it would be great addition in the future.

My humble requests and observations:

  1. I really like the UI. It’s clean and yet has a lot of information. Would prefer the list of fields to be more compact (in general I use 80% zoom anywhere in Looker)

  2. Optional display of fields from JOINs. Otherwise each explore gets big with repetitive metrics. I have two explores that are joined to everything. No need to show their fields on each page, they have their own page.

  3. Showing the explore fields first then the JOINs’. If it happens that my explore starts with N but join starts with A, the join fields will be first. I think the order should go with the importance and then alphabet. If I have a join in each explore that starts with A, then it will look like it’s the same page everywhere (until people start scrolling a lot).

  4. Ability to add descriptions that only live in Data Dictionary not LookML. There are things I want to add but not to be visible in the UI.

  5. It would be amazing to have a different view that groups items by the group first. Again it’s all to make it more compact upon first entry and then allow people to find what they need.

  6. Show dimensions group as one field not each timeframe separately

  7. Filter for hidden fields

As you can see for me the most important part is that the list is as compact as possible. Good UI and search invite people to explore, which is great but they can get overwhelmed upon first visit.

12 22 2,552
22 REPLIES 22

I like the clean interface, I was initially doubtful if there would be a huge benefit when users can look at the LookML if they are unsure but the search capability across views in explores and less intimidating view are big pluses to me and I see a lot of potential use cases.
My ideal (and likely impossible) enhancements would include:

  1. A way to see where/how often a field is used ideally in all dashboards but if not in LookML ones at least so you can ensure that you are using the same version of the metric and what impact any changes could have
  2. A way to see the lineage of a field inside the explore. I.e. the field is used in the derivation of these metrics in the lookml layer this would allow you to see the impact of changes and also if that field is still required

Also, fully agree on Dawid’s points, particularly hidden fields, and descriptions in Data Dictionary only.

moebe
Participant V

Overall I like the Data Dictionary application very much.
It is clear and well structured.
But unfortunately it is missing one, for me, decisive “feature”

If I define a measure that way:

 measure: value_usd{
    type: sum
    sql: case when currency.currency = 'USD' then ${value} else nul end;;
  }

then everything’s great.
but if I use the standard Looker syntax:

 measure: value_usd{
    type: sum
    sql: ${value} ;;
    filters: [currency.currency: "USD"]
  }

the information is lost.
That should be changed.

bens1
Participant V

Interesting! Meaning it should be ‘unfurled’ to its expanded SQL before presentation?

moebe
Participant V

Good point.
yes and no
If this should really help, filters (which could also be a case when condition) would have to be resolved.
Or the recommendation should clearly go to a case when solution instead of the filter solution.

Dawid
Participant V

I think both ways should be presented. LookML version is easier to read even for people who are not LookML developers and just want to see what filters are applied. The SQL, however, could be helpful for developers (?)

moebe
Participant V

yes, I think so too

bens1
Participant V

To me, that’s the best call. LookML can be approached from the dev stream or the business stream, so I feel being able to see side-by-side LookML and/or full SQL would allow for easier entrance from either world.

@Dawid thanks for starting this thread! my company likes the data dictionary too, our biggest asks would be:

  • clustering/hiding repetitive date dimensions (there is no need to have a new definition for a date, month, year variable)
  • a global text search feature
  • all dimension types are auto-highlighted as filters, or there is an easy way to select/deselect all dimension types

Hey There!

Dillon, PM for the Data Dictionary, here. We’re thrilled to see the discussion In broad strokes, it seems like there are a few common requests here. Please chime in if I’ve misunderstood, or if you have additional feedback!

SQL vs LookML Field Displays - Provide the LookML parameters that help define logic outside of the sql: parameter. The Data Dictionary already provides the options to view the SQL and the LookML definitions, but this loses all context when additional LookML parameters, such as filters: are introduced. This makes complete sense, we’ll think through the best way to display that information.

Compactness/Configuration - The ability to see more information on the page with compact formatting, and options around the ordering/display of tables and field groupings, for the purpose of making browsing through fields quicker. This is great feedback and something we’re working towards. Expect iterative improvements on this.

Adding Descriptions - Adding descriptions that either only live in the Data Dictionary UI, or that get pushed back to the underlying LookML. This is an interesting request that we’ve thought a lot about as well. Some of this functionality will be unlocked with the ability to save data within a data store specific to the app. We’re actively working on this and hope to introduce something in the Q3 timeframe (July-Aug).

Global Search - The ability to search for a field without specifying a Model or Explore first. We agree, this is very high on our list. The current search functionality requires users to have an understanding of Looker’s Model and Explore structure to find a field. We’re aiming to eliminate this friction, and make it quick and easy for a user to find a field without any prior knowledge of Looker’s model structure. We’re working through various implementation models as we speak. We’re hoping to land these improvements in the Q3 timeframe as well. Eventually, we’d love to tie additional metadata such as popularity or query/user count to these fields too.

Again, very helpful feedback. Please keep it coming!

@Gordon1 I’d love to dig into the data “lineage” use-case a bit more here. Do either of these capture your goals accurately? If not, would you mind providing an example?

  • When viewing fields in the Data Dictionary UI, more easily understand the definitions of derived fields. For example, if I had a derived dimension equal to profit, you currently see a LookML definition of ${revenue} - ${cost} and you’d prefer someway to see the core table/SQL definitions of the revenue and cost fields as well? Something like ${TABLE}.revenue - ${TABLE}.cost.

  • When developing a LookML model, quickly understand how a change in one field (e.g. revenue) will impact all the other fields that are derived off it (e.g. profit)?

Thanks for getting back to me @Dillon_Morrison,

  1. Exactly right, so in the first example if you looked up the definition of profit and saw ${revenue} - ${cost} you wouldn’t need to search for revenue and cost to find those definitions. Ideally this could be through unfolding each of the fields as those ‘staging’ fields are also good to know!

  2. Yes, so to use the example above, it would be great to be able to see what fields elsewhere would be changed by a redefinition of ${revenue} whether that only impacts profit, or would also impact net_profit and Profitability both in that view or at a model level.

@Gordon1 thanks so much, appreciate the clarification.

  1. This request makes total sense. I’ve added to our roadmap, and is something we’re actively thinking through now.

  2. I agree with you that this would definitely be helpful functionality, and it’s something we really want to tackle eventually. We certainly though about this for the Data Dictionary as well, but came to the conclusion that the Data Dictionary UI isn’t the “best” place to solve for this. If you’re editing fields, you’re almost always in the model, in your IDE. In our view, a better workflow would be to see how a change in one field affects any other, straight from within the IDE. We’re planning on adding the ability to insert Extensions into the IDE sometime next year. At that point, I can see a new “lineage” tool being introduced for the IDE for this use-case specifically…

@Dawid I’d love to learn more about item number 4 on your list - “Ability to add descriptions that only live in Data Dictionary not LookML. There are things I want to add but not to be visible in the UI.” Can you elaborate on your use-case a bit more here? I’m wondering if it’s either of these use-cases. Please correct me if I’m wrong 🙂

  • Are you, as a data modeler/developer, wanting to add a second definition (in addition to the LookML description)? This definition would only live in the Data Dictionary. If so, why does the core LookML description not suffice? It seems like it would be better to overwrite the core LookML description.

OR

  • Are you hoping that your non-developers (viewers, explorers, and anyone without access to LookML) are given the ability to add a second definition to a field. A definition/comment that would only live in the Data Dictionary UI. A definition/comment that you would not want in the LookML description, and would largely benefit the non-developers.Would you want any non-developer to have this capability, or would it be a specific permission set in your mind?

Dawid
Participant V

More of the first one. I think Data Dictionary should read LookML as it does but I don’t want to keep extensive descriptions in the description parameter.

The way parameter is shown in the UI of the explore makes sense, kind of. It’s a small i icon that you need to make a conscious action on to show the description. I used them if there are intricacies about the field, for example, something is excluded etc.

It doesn’t mean that the field itself is self-explanatory but that’s not the place to put big descriptions. Imagine having to hover with your mouse over each i to get this.

Data Dictionary to me should have its own separate layer where I can put long descriptions, show calculations, even attach images. If I don’t have to use Confluence for such things or other software, it would be magnificent.

Basically I would like Data Dictionary to be more than just a UI to read LookML. I want the LookML to be the source but for the Data Dictionary to enrich it much more than just parameters in the LookML.

Hope it makes sense.

@Dillon_Morrison 

I have a similar concern to @Dawid in regards to #4, it goes one step further, we don’t want to show any descriptions at all as tooltips in our dashboards. We had started using descriptions because of the data dictionary, when we realized they were in tooltips everywhere we had to comment out our descriptions and stop using the data dictionary. 

It’d be nice if we could have a “data dictionary only description” available in the LookML. Or the ability to disable description tooltips in our dashboards.

Also, one additional feature that would be great to have is a way to export the Data Dictionary to PDF and/or to a spreadsheet.

David_P1
Participant V

Adding to the thread, it would be useful to add admin controls to the Data Dictionary.

For example, as an admin, I would like to limit certain parameters in the UI, so that end-users are only presented with a simple and easy to understand list of dimensions/metrics.

I’m aware of the view options dropdown in the data dictionary, but I would like to take it a step forward and unilaterally decide which fields end-users get to see or choose from.

Also, +1 for extended definitions that live in the data dictionary, not necessarily in the tooltip description.

Dmitri
Participant II

Hi @Dawid Is it available on GIT?

We develop an extension where users should see only available models similar to data dictionary extension. We are curious what methods should be used.

Hey There!

Dillon, PM for the Data Dictionary, here. We’re thrilled to see the discussion In broad strokes, it seems like there are a few common requests here. Please chime in if I’ve misunderstood, or if you have additional feedback!

SQL vs LookML Field Displays - Provide the LookML parameters that help define logic outside of the sql: parameter. The Data Dictionary already provides the options to view the SQL and the LookML definitions, but this loses all context when additional LookML parameters, such as filters: are introduced. This makes complete sense, we’ll think through the best way to display that information.

Compactness/Configuration - The ability to see more information on the page with compact formatting, and options around the ordering/display of tables and field groupings, for the purpose of making browsing through fields quicker. This is great feedback and something we’re working towards. Expect iterative improvements on this.

Adding Descriptions - Adding descriptions that either only live in the Data Dictionary UI, or that get pushed back to the underlying LookML. This is an interesting request that we’ve thought a lot about as well. Some of this functionality will be unlocked with the ability to save data within a data store specific to the app. We’re actively working on this and hope to introduce something in the Q3 timeframe (July-Aug).

Global Search - The ability to search for a field without specifying a Model or Explore first. We agree, this is very high on our list. The current search functionality requires users to have an understanding of Looker’s Model and Explore structure to find a field. We’re aiming to eliminate this friction, and make it quick and easy for a user to find a field without any prior knowledge of Looker’s model structure. We’re working through various implementation models as we speak. We’re hoping to land these improvements in the Q3 timeframe as well. Eventually, we’d love to tie additional metadata such as popularity or query/user count to these fields too.

Again, very helpful feedback. Please keep it coming!

Any updates on the global search? I think this would be such a huge win. We are in a self serve environment with many beginners or newbies (with Looker, LookML or coding) having a need to either leverage or build new content, and they want to see a specific field but might not necessarily know which model/explore it exists in. I would also vote (as others have in this thread) the ability to see which content in Looker (look or dashboard) uses the explore or field - something similar to what Tableau offers in their Data Management add-on.

LucianaPadua
Participant II

Hello everyone, 

I’m trying to see how’s the best way to inform our Viewer users about the data and I see that the dictionary only works for users with Explore rights. We don’t want to give everyone explore rights, so i was wondering if there’s a workaround? 

Or is it not expected that a Viewer has full access to the documentation of the data? I guess the idea behind is that Viewers should just consumer what’s there on the dashboards, but even those should be explained. 

thanks for the support! 

Global Search - The ability to search for a field without specifying a Model or Explore first. We agree, this is very high on our list. The current search functionality requires users to have an understanding of Looker’s Model and Explore structure to find a field. We’re aiming to eliminate this friction, and make it quick and easy for a user to find a field without any prior knowledge of Looker’s model structure. We’re working through various implementation models as we speak. We’re hoping to land these improvements in the Q3 timeframe as well. Eventually, we’d love to tie additional metadata such as popularity or query/user count to these fields too.

Again, very helpful feedback. Please keep it coming!

@Dillon_Morrison , do you have any progress update / estimation for the global search?

It is a very high demand feature among the business users, especially who are relatively new to Looker/our content.

Thank you!

Hey Dawid, check this out - https://docs.castordoc.com/assets/dashboards

You may be able to use the Looker Dictionary to solve some of these challenges. Often times I see companies having different definitions stored in different places. Many companies who use looker end up using Castor to manage these kinds of things (along with lineage, dashboard popularity/usage, etc). 

Feel free to reach out to me if you have questions on the documentation I shared! 

Dawid
Participant V

@Dillon_Morrison  Any updates?