Looker Connect Training

Help Center

Documentation

Community

Cloud Certifications

Question

API Paging & Limits

  • 18 October 2019
  • 4 replies
  • 768 views
Mike_Lohmeier

Hello,



We are self hosting Looker and using the multi-tenancy setup. In our production environment, we have ~500 tenants and in our test environment we have ~5000 tenants. We are seeing our test environment Looker begin to hang in the web UI on GET requests with a lot of objects for a single resource. Even worse, when using the API directly writes have begun to fail for resources with a lot of objects. The write failures start out by being slow, then cascade into 500s and finally bring the entire Looker cluster down.



This seems to be 2 root problems. First, the API doesn’t look to support paging for GET requests on unbounded resources. Is paging on GET requests in the product backlog? Is there a workaround that we can use here until it’s implemented?



Second, Looker seems to start failing on writes after a certain limit. In our case, we have ~5 million content-metadata-access records. This is caused by us having to delete other tenants from spaces that were automatically cascaded from the parent shared node. What are the limits here for Looker in terms of writes?



Thanks,


Mike

4 replies

izzy
Userlevel 7
Badge +1


Not 100% clear on the difference between a bounded/unbounded resource, but some endpoints do support pagination. all_looks(), for example, does NOT support pagination and might overload things. search_looks() DOES support pagination! Check out the docs here: https://docs.looker.com/reference/api-and-integration/api-reference/v3.1/look#search_looks .



Anywhere that there exists a search_* analog, you can use that endpoint and page the results. I hope that helps! Bringing pagination to the rest of the API would be neat.



Not sure on the limits there. re: the failures, I bet our support team (help.looker.com) would want to take a look at the logs around the time of the failures to see what those 500 errors really mean.

Mike_Lohmeier

Thanks, we’re struggling specifically with GET All Content Metadata Access which does not have a search endpoint that supports paging. Can I confirm that there is a feature request for paging in the Looker product backlog.



I will reproduce the Looker cluster crash and submit the logs and steps to reproduce to help.looker.com.



Thank you

izzy
Userlevel 7
Badge +1


Yep, I can confirm that there is a feature request logged for API pagination. (And, as I mentioned above, some endpoints already support it).

izzy
Userlevel 7
Badge +1

2 posts were split to a new topic: Google Sheets API limit?

Reply


Terms of Service