Schedule Delivery around 1M rows of data from Looker

[vr_2020]

From Looker, I was trying to send around 1Million rows of data (data comes from Google Big Query Table to Looker and growing) with Email scheduled delivery option in form of CSV format, however there is a limitation of 

“Error: Result set for email cannot exceed 15 megabytes”

Did anyone try to schedule delivery (apart from Email and SFTP option) through Amazon S3 or Dropbox or Google Drive or Google Cloud Storage or any other options to send around 1Million rows of data from ‘Look’?  Is that possible? Please let me know

[olga1]

Hiya!

I don’t see why not 🙂 Those options are used widely in Looker especially for big files. The limitation of 15MB is related to the email service itself, not to Looker, so feel free to try any of the other options from here https://help.looker.com/hc/en-us/articles/4419764410003-Delivering-Looks-and-Explores#choosing_the_d...

You may need the permissions enabled by admin to use some of them, or the specific action (as Dropbox from your question) enabled by admin, but after that you should be good

[vr_2020]

@olga1 - Thanks! However, when I tried with Dropbox option from Looker. There is no option available like “All Results” to select in Advanced options section to send more than 5000 rows in form of CSV file. 

In SFTP,  atleast “All Results” option is available. Similar option is not available in Dropbox. So, I don’t think Dropbox option is suitable for sending more than 5000 rows

     Screenshot when SFTP option is selected

Other Option for example Amazon S3:

Amazon S3 - Even this option doesn’t tell exact maximum number of rows that can be sent out in form of CSV file as per this link Delivering Looks and Explores – Looker Help Center

Delivery options for Amazon S3 buckets

Amazon S3 buckets are a common way to store large amounts of data. You or your company will need to have created an S3 bucket with Amazon before Looker can use it.

Therefore, can you please confirm which option will be definitely (apart from Email, SFTP, Dropbox, Amazon S3) suitable to send the large number of rows of data (around 1Million)? Which will help me to narrow down the best option available in Looker as the Number of options given in third-party integrations category section are more.

[masutaka]

“Error: Result set for email cannot exceed 15 megabytes”

This is a limitation of Looker. See below:

https://cloud.google.com/looker/docs/scheduling-and-sending-dashboards#email

Emailed deliveries may not exceed 20 MB (for formats that are delivered in the email body) and 15 MB (for formats that are delivered as an attachment).

With SFTP or S3, there are no such limitations. However, there are conditions:

https://cloud.google.com/looker/docs/downloading#streaming_query_results

1. Table calculations: Table calculations cannot be streamed. Therefore, to download unlimited results for a query, you must remove table calculations from the query.

2. Database limitations: Some databases cannot stream if pivots are included in the query. For these databases, you must remove pivots to download unlimited results. Some databases can't stream any results, which means that unlimited downloads are not possible. Following are a list of databases that support streaming and a list of databases that support streaming with pivots applied:

[Nick_Lam]

Hey all, just wanted to add in case it was helpful to someone. I had an issue where users weren't able to schedule deliveries using email due to this limitation on Looks. Our users normally need things in excel and I was able to work around this by converting the Look into a dashboard and then scheduling the delivery as CSV. This allowed the delivery since the file is compressed.