I am sharing code example for a Google Cloud Function that leverages Looker Python SDK and the Python client for BigQuery to get the result of a query in Looker and load the result to a BigQuery table. The workflow can be set up to run automatically with a cron job using Cloud Workflow.
Potential use cases
- Workaround for System Activity ETL: Get data from Looker’s System Activity and write to BigQuery directly. These BigQuery tables can then be registered as a connection in Looker for additional LookML data modeling. Currently, Looker's System Activity stores a maximum of 100k rows, or 90 days of historical query and event data
- Transfer data from a different database to BigQuery
- Cloud Function is easy to set up, and suitable for light-weighted, on-the-fly tasks. For heavy workloads, consider using Looker's native actions (sending to Google Cloud Storage, S3) or additional ETL/ELT tools (such as GCP's Dataflow)
- Consider using Elite System Activity for more flexibility on System Activity
Let me know if you have any questions and feedback