Large custom map_layers with tiles

Something that comes up occasionally is a request to add a custom map_layer to Looker that is very large. Most commonly the questions start when a user realizes that the TopoJSON file for their custom map is hundreds of MB or even GBs and that loading the whole thing into the browser at once isn’t going to work.

Luckily, there are great tools available that are low-cost or even free that make this process reasonably simple. It does require a little bit of patience and some developer tools (like the command line), but if you’ve used Github before, it’s probably well within your skillset.

For the purposes of these instructions, I’m going to assume that you already have a single file that represents your map layer. When you’re uploading the file directly to Looker, that file has to be TopoJSON. But for our purposes, we actually need it to be GeoJSON, KML, or a shapefile. If you have TopoJSON, Mapshaper is a super easy tool for transforming it to GeoJSON.

For this tutorial, I’ll use a GeoJSON file that outlines all 73,057 tracts from the 2010 U.S. Census. This GeoJSON weighs in at just over 1 Gb–definitely not something I want users to load into the browser every time they access it.

But I can take advantage of the same technology that drives Google Maps–tiles–to make the file tractable (pun intended). The basic idea is that if I’m zoomed way in, only a few tracts will be in my viewport, so I only need to load those. Then as I pan, I load the adjacent ones and so on.

If, on the other hand, I’m zoomed way out and can see big regions of the country with thousands of tracts, I don’t actually need all the detail that is in my GeoJSON file. Very rough outlines of the tracts (which weigh far less) will suffice.

Tiling the map at various zoom levels gives me all the raw materials I need to only download the pieces of the map I need right now, at the appropriate level of detail. And Mapbox provides a simple set of tools to do it.

  1. So first, go create a free account at Mapbox. Once you’ve done that, create an Access Token and save it for later. (The default public scopes work fine.)

  2. Then go into Mapbox’s Studio to upload your data. You want to create a new tileset.

  3. Once the tileset is uploaded, it’ll take a bit to process (gotta make all those tiles). After it’s done, click in and you’ll see something like this:
     

    Mapbox_Studio
  4. Now comes the trickiest (but not really that tricky) part. We need to calculate the bounding box for each tile region in our shapefile. This is because in order to know which regions to load at any given time, we need to know what the furthest SW and NE points are that encompass the region.

  5. We have a little script that’ll do this for you, which you can download here. The instructions are in the script, but you’re going to do the following:

  • Install Yarn by opening a command line and typing ( brew install yarn )
  • Run Yarn by typing yarn
  • Generate the extents.json file by typing yarn run extents /my/input/geojson.json extents.json ZCTA5. Except you’re going to replace /my/input/geojson.json with the actual location of your file and ZCTA5 with the ID field in your shapefile that uniquely identifies each region.
  1. Now that you have the extents.json file generated, the last thing you need is to put that somewhere where Looker can access it. You can upload it to an object store like S3 or Google Cloud Storage, but you’ll have to configure the access-control-allow-origin headers properly. You also can’t use Github directly, because it’ll set the wrong “Content-Type Headers.” Luckily, you can upload the file to Github or as a Github gist, and then use this nifty service to fix the headers.

  2. Now we have all the pieces we need to properly define our new map layer in Looker! Go to your model in Looker and as part of the model file, define your new map_layer as follows.

map_layer: tract {

format: "vector_tile_region"

url: "https://a.tiles.mapbox.com/v4/[YOUR_MAPBOX_TILESET_ID]/{z}/{x}/{y}.mvt?access_token=[YOUR_MAPBOX_ACCESS_TOKEN]"

feature_key: "[YOUR_MAPBOX_TILESET_NAME]"

extents_json_url: "https://rawcdn.githack.com/[YOUR_EXTENTS.JSON_FILE_LOCATION]"

min_zoom_level: [FROM_MAPBOX]

max_zoom_level: [FROM_MAPBOX]

property_key: "[THE_UNIQUE_IDENTIFIER_FOR_REGIONS_IN_YOUR_SHAPEFILE]"

}

To illustrate where you’re going to get all this info, see below:

Mapbox_Studio

And:

And that’s it! So, not a 5-second process. But it’s totally worth it ?

Explore_Fast_Facts

9 30 3,940
30 REPLIES 30

Thanks for this really awesome write up @Mintz . So I’m having some issues where due to the minimum zoom level being hardcoded/determined by Mapbox, data is not presented at all zoom levels in Looker.

image

Ideally, I’d like to be able to set the zoom level to anything in Looker (and still have data displayed), so I followed Mapbox’s instructions for adjusting zoom extent here: https://docs.mapbox.com/help/troubleshooting/adjust-tileset-zoom-extent/#transform-data-with-tippeca...

Now what I’m stuck on is this – how do I generate the bounding box for tile regions for an mbtiles filetype?

Good questions, @amm7790! So on the zooms, I think that’s mostly determined by Mapbox’s read of how much detail is in the file. Regardless of what Mapbox sets, though, I think if you set the zoom max/min to what you want when you create the map_layer in Looker, it should work well enough.

The bounding boxes that get transmitted to Mapbox are handled automatically by Looker assuming you have the extents file working right, so you shouldn’t have to worry about those steps with Tippecanoe.

I tried changing the min_zoom_level parameter to 1 in map_layer, but it did not display data in the explore viz until i reached up to zoom level 9, which is the minimum defined zoom level in Mapbox. My read on this is that the min/max zoom layer params in map_layer are only there to prevent a Looker viz in an explore from going into those “restricted” levels and showing a map, but no data?

When you say “Looker assuming you have the extents file working right, so you shouldn’t have to worry about those steps with Tippecanoe”

Do you mean I shouldn’t have to worry about creating another extents.js file for the new mbtiles filetype generated by Tippecanoe? I don’t really have a good grasp on how this new filetype works, but I would assume I’d need to generate something new for this new file.

Hi @Mintz this seems to fit a need I have perfectly! The issue that I am running into is that yarn run extents... returns that ‘Command “extents” not found.’ Is there something that I am missing? Thanks!

Hey Ryan, make sure you’ve installed yarn as outlined above, and then make sure that you’re running the command from the commandline and that you’re in the directory where the the extents.js file that you’ve downloaded is located.

Sounds like you may be in a different directory, which would explain why it can’t find the script.

Sorry, just saw this.

Hopefully you’ve been able to figure it out yourself, but I’ve never run into this exact issue with the zoom levels before. It looks from the documentation like creating the MBTiles with tippecanoe might be the right strategy to manually define the min/max zoom levels.

But as you said, you can’t create an extents file from an MBTiles file. That said, the extents file shouldn’t actually be sensitive to zoom levels, since what it’s doing is defining the far NW and SE coordinates for each region in the map layer. Those aren’t affected by the zoom levels.

So I’d try generating the extents file directly on the GeoJSON file you have, then converting it to an MBTiles file with your preferred zoom limits and uploading that to Mapbox. And see if that works.

does this mean i can directly point to the geojson files and display the same too?

Hi,
Not totally sure what you mean by “display the same too”. Can you explain a bit more about what you’re trying to do?

getbhanuin
Participant I

Hi Daniel,
any idea why am getting this error

C02WF0QNHTD8:vector-map-scripts-master mbhanu$ pwd
/Users/mbhanu/downloads/vector-map-scripts-master

(base) C02WF0QNHTD8:vector-map-scripts-master mbhanu$ ls
README.md export.js extents.js package.json scripts source yarn.lock

(base) C02WF0QNHTD8:vector-map-scripts-master mbhanu$ yarn run extents /users/mbhanu/Desktop/map_tiles.geojson extents.json store_number
yarn run v1.21.1
$ node extents.js /users/mbhanu/Desktop/map_tiles.geojson extents.json store_number
internal/modules/cjs/loader.js:969
throw err;
^

Error: Cannot find module 'geojson-stream’
Require stack:

  • /Users/mbhanu/Downloads/vector-map-scripts-master/extents.js
    at Function.Module._resolveFilename (internal/modules/cjs/loader.js:966:17)
    at Function.Module._load (internal/modules/cjs/loader.js:859:27)
    at Module.require (internal/modules/cjs/loader.js:1028:19)
    at require (internal/modules/cjs/helpers.js:72:18)
    at Object. (/Users/mbhanu/Downloads/vector-map-scripts-master/extents.js:1:21)
    at Module._compile (internal/modules/cjs/loader.js:1139:30)
    at Object.Module._extensions…js (internal/modules/cjs/loader.js:1159:10)
    at Module.load (internal/modules/cjs/loader.js:988:32)
    at Function.Module._load (internal/modules/cjs/loader.js:896:14)
    at Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:71:12) {
    code: ‘MODULE_NOT_FOUND’,
    ** requireStack: [ ‘/Users/mbhanu/Downloads/vector-map-scripts-master/extents.js’ ]**
    }
    error Command failed with exit code 1.
    info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
    (base) C02WF0QNHTD8:vector-map-scripts-master mbhanu$

but am running the command from the location were extents.js is available

Thanks
Bhanu

Hey Bhanu, I’m definitely not a yarn expert, but the first thing I’d check is to make sure that it installed correctly and that it’s up to date.

Make sure brew install yarn succeeds and then try brew update yarn

Thanks for tutorial! Very helpful for large map!

I would like to point out for Point 6 about extents.json, you can actually use cloud storage like GCS or S3, the problem is not Content-Type header, but CORS. Have to set “access-control-allow-origin” to “*” to make it work.

For GCS, I found a solution on StackOverflow: https://stackoverflow.com/a/40000115/3620517. Have to use gsutil to set the CORS on the bucket.

Hi Yuji,

Thanks for flagging this. I explained this wrong in my original post (I’ll fix). As you found, it’s fine to host the extents.json file from any CDN (including GCS and S3), though you may have to manually configure the access-control-allow-origin headers.

What I should have said is that serving it directly from Github won’t work, because Github configures the Content-Type headers incorrectly for JSON files. That’s why you have to use raw.githack.com to get around that issue.

Thanks again!

Hello.. I tried accessing the link to download the yarn script but i keep getting access denied

Okrasee
Participant I

Hi @Mintz, I got an error when putting the url in the TopoJSON URL box. This is a valid, open-source TopoJSON file

82122d45-c11a-49bb-bdaa-63b75b854361.png

Link to the github repository is here: https://github.com/OpenDataDE/State-zip-code-GeoJSON/blob/master/ny_new_york_zip_codes_geo.min.json

I searched for an answer everywhere but did not find anything. Wonder if you could help?

Hi @Okrasee, it looks like that TopoJSON is 40Mb. I’m very skeptical that’s going to work smoothly at that size.  Usually smooth performance on TopoJSON’s maxes out at about 5Mb.

That said, if you want to move forward, you’re going to need to make sure to hit the raw version of the file at https://raw.githubusercontent.com/OpenDataDE/State-zip-code-GeoJSON/master/ny_new_york_zip_codes_geo...

Echo1
Participant I

Hi @Mintz Thank you so much for sharing it with the community!

I was able to make it work 20 days ago but now I saw “Fetch data error” in Looker.

Is it because mapbox updated their api?

Thanks!

Echo

Hi @Echo1,

I’m not familiar with that error specifically. Can you paste a screenshot or more details about the exact error text you’re seeing and in what context?

Thanks,

Daniel

Echo1
Participant I
5bcaab0e-cba4-449c-a8ff-80f2ddaf1f24.png

Thank you for the reply! This is what I saw in Looker now :<

Echo1
Participant I

Hi @Echo1,

I’m not familiar with that error specifically. Can you paste a screenshot or more details about the exact error text you’re seeing and in what context?

Thanks,

Daniel

Or can you please confirm that your demo Looker dashboard is still working?

Ooooh, that’s a new one. If you know how, could you open up the JS console as you load it and see if there are any errors there?

Echo1
Participant I

Um..I’ve never used JS console in Looker before. Should I follow up these steps to setup?

https://docs.looker.com/reference/embed-sdk/embed-sdk-intro

Ah, sorry, I just meant in Chrome. If you go to View → Developer → JS Console.

Echo1
Participant I

Ah, sorry, I just meant in Chrome. If you go to View → Developer → JS Console.

Kk. I am seeing this (blurred sensitive info)

Access to fetch at 'https://rawcdn.githack.com/***/extents.json' from origin 'https://***.looker.com' has been blocked by CORS policy: No 'Access-Control-Allow-Origin' header is present on the requested resource. If an opaque response serves your needs, set the request's mode to 'no-cors' to fetch the resource with CORS disabled.

Ok, that’s helpful. I’m not sure that’s the issue that’s causing the error message, but it’s definitely interesting. If you have access to chat support, it’d be great if you could ask them so they can file a bug. If you don’t, you can send it in to support@looker.com. Something’s definitely wrong, I’m just not sure yet what it is.

Echo1
Participant I

Problem solved! Looks like there is something wrong with the 

extents_json_url: "https://rawcdn.githack.com/[YOUR_EXTENTS.JSON_FILE_LOCATION]"

I uploaded to Gitst instead of Github and regenerated the url and it worked!

Great to hear it’s solved. I still think it’s a bug though that it wouldn’t display without the extents file. Was this on a dashboard? Doesn’t look like it from your screenshot,  but just want to confirm.

@Mintz having above issue of “Failed to Fetch”, I’ve had my layers working for a while now with no issue and now was trying to update it and run into this.

Despite the error are there any plans to make this solution not this hacky? Seems looker is really behind all competitors mapping wise.

Thanks in advance!

is giving me this error in the Chrome JS console, I have my data in AWS

Access to fetch at 'https://…./looker_extents/h3_8_extents.json' from origin 'https://…..eu.looker.com' has been blocked by CORS policy: No 'Access-Control-Allow-Origin' header is present on the requested resource. If an opaque response serves your needs, set the request's mode to 'no-cors' to fetch the resource with CORS disabled.

In safari looks like in the picture

e740e1bb-c484-498d-a576-cbc56576a41c.png

Hello.. I tried accessing the link to download the yarn script but i keep getting access denied

Re-upping this comment. The link to the “extent.js” file is returning “AccessDenied”. Is this script available anywhere else? 

Thanks for the walkthrough.

I messaged the author of this article, and the extents file is now stored here:
https://github.com/dwmintz/vector-map-scripts/tree/main

Top Labels in this Space
Top Solution Authors