Question

BigQuery Job ID for Looker Query

  • 9 February 2017
  • 12 replies
  • 184 views

Hi,


We’re doing some administration of Looker on BigQuery. Is there a way to associate a Looker query with a BigQuery Job ID? We’d like to be able to get detailed BigQuery job stats and execution plan for specific Looker queries that have run.


12 replies

+1 it would be great if there were a way to do this. I know we’ve talked about a comment in the SQL and it’s a hard problem, but at our BigQuery scale this is becoming a really hard problem.

Looker team?

Userlevel 1

Hi Josh, there are no plans as of this time but our Product team is aware of the benefits of surfacing Job ID and other query stats. @joshsiegel

Userlevel 3

Actually, there is already existing functionality in Looker to handle most of the relevant use cases, but it needs to be tweaked.


Looker has a field sql_text. A hash of this field concatenated with a query timestamp would let us work across the two datasets (i.e. logs from BigQuery and logs from Looker). However, at the moment the sql text is not populated uniformly across all queries.

Userlevel 4

In a related topic: Context Comments might be relevant

Please let us know if there is an update regarding this ticket.

Userlevel 6
Badge

you can fish out the historyid (lookers ID) from the context comments which are in the bq logs.

With context comments being being disabled in Looker 7.6 for BQ, how would we go about joining Looker logs to BQ logs with no History ID available, historically we used this to join the datasets together?

Userlevel 6
Badge

Looker should really lean on the Google BQ guys to get this changed (bq cache queries incorporating comments which is silly). Having the looker historyID in the logs of bq is probably very important to many people. @izzy could you mention this to the relevant people.

Thanks!

+1

I’ve faced this issue also!

Userlevel 7
Badge +1

We didn’t realize how big of an impact this would have when we made the change with performance improvements in mind. We’ve taken steps to fix it post haste! We’re adding BigQuery context comments back in 7.10 as an optional toggle in the connection settings. It will be backported to 7.8 and 7.6 for those on older releases.


We haven’t managed to edit the BQ caching mechanism yet, but this will get back to the previously expected functionality 🎉

Any latest update on this feature request please, Looker Team?

Reply