Question

Generating Client SDKs for the Looker API

  • 16 August 2016
  • 65 replies
  • 2094 views

Userlevel 2

Update: This article is now out of date. For the latest info on how to download or generate client SDKs for the Looker API, please see our Looker SDK Codegen repo on Github






The Looker API is a collection of “RESTful” operations that enables you to tap into the power of the Looker data platform in your own applications. The Looker API can be invoked using plain old HTTP(S) requests. Any development tool, language, or application that can make HTTP requests and ingest JSON responses should be able to use the Looker API.


Web geeks who live and breathe HTTP and/or AJAX XHR will feel at home using just the “raw” Looker API HTTP URL endpoints. GET, POST, PUT, PATCH, DELETE, oh my!


Web APIs For The ‘REST’ of Us


If writing HTTP requests is not something you or your developers do every day, accessing a REST API for the first time can be a little intimidating and disorienting. Programming with web requests often requires different idioms and patterns of behavior than traditional app development. What do you do if you’re more interested in the data results than in the journey required to get them?


What non-web devs desire is something that makes the Looker API feel like a set of functions in your language of choice: a client SDK. A client SDK is a set of functions that resides in your application that encapsulates all the HTTP goop so you can just call functions and get results.


The Looker API provides a metadata document which describes all the URL endpoints, their parameter names and types, and their response types. From that metadata you can use Swagger tools to generate client SDK libraries for your programming language.


This allows us to make the Looker data platform accessible via convenient client libraries for a wide variety of programming languages, without having to be experts in all of those languages or having to commit resources to developing and supporting all those languages. It also helps ensure consistency across all manifestations of these client SDKs which would be very difficult to manage if each SDK were hand-written.


Enough back story. Let’s generate a Looker API client SDK!


Prerequisites




  1. Login to your Looker instance as an admin to enable and configure the Looker API: https://<your_looker_endpoint>/admin/api. Make note of your Looker API Host url displayed here if you have customized it from the default.




  2. Generate API3 credentials by navigating to https://<your_looker_endpoint>/admin/users, editing your user, and clicking “New API3 Key.” You’ll need the API3 clientId and clientSecret values later to authenticate your client application.




  3. With your API configured and credentials in hand, navigate to your Looker API documentation: https://<your_looker_endpoint>:19999/api-docs/index.html. (Substitute your custom Looker API Host domain and port if applicable)

    This page is our reference for available Looker API function/endpoints specific to our Looker instance. Since this doc page is generated from the Looker API metadata, this doc page may show beta or experimental API functions that are not yet officially documented elsewhere.




  4. Make sure Python 2.7 or 3.0 is installed on your computer. At a command line / shell prompt, run python --version to see what version of Python, if any, is installed.




  5. You’ll need a Java runtime distribution installed on your computer. We recommend at least Java 8 (version 1.8.*). Run java -version to verify version installed.




  6. Make sure Maven 3.3 or later is installed on your computer. At a command line / shell prompt, run mvn --version to see what version of Maven is installed.




  7. You’ll also need to have git client tools installed on your computer. Check version with git --version




Setup




  1. Make a directory to house your Looker SDK bits: cd ~/Desktop; mkdir looker_sdk; cd looker_sdk




  2. Download your Looker API swagger.json metadata.




  3. Login to your Looker instance as admin and navigate to the admin/api panel. (https://<your-looker-endpoint>:9999/admin/api).




  4. Set “Documentation Access” to “Allow anyone to see API docs”




  5. Click the Save button.




  6. Open a new browser window and navigate to this URL: https://<your-looker-endpoint>:19999/api/3.0/swagger.json Subsitute your custom Looker API Host domain name and port if necessary.




  7. Save the contents of that swagger.json page to looker_sdk/lookerapi.json.




  8. Take a peek inside the saved file to make sure the browser saved it as JSON text, not transmogrified HTML.




  9. Return to your Looker instance Admin/API page and set the “Documentation Access” setting back to what it was before and click Save. “Require API login to see API docs” is the default setting and is required if you want to use the “Try it!” button to call API functions directly from the API doc page. (You do want this, it’s cool!)




  10. Download the Swagger-Codegen tools from github.

    Note: Swagger-Codegen is in active development. To avoid the headaches of “bleeding edge” development, it’s a good idea to pull from a stable release. At the time of this writing, we recommend using Swagger-Codegen release 2.1.6. (If you’re using Python 3.0 or later, you’ll want Swagger-Codegen release 2.2.2 instead)

    Use these steps to checkout the swagger-codegen tools into your looker_sdk/swagger-codegen directory:




  11. cd looker_sdk




  12. git clone git@github.com:swagger-api/swagger-codegen.git ./swagger-codegen




  13. cd swagger-codegen




  14. git checkout tags/v2.1.6




  15. Build the Swagger-Codegen tools




  16. cd looker_sdk/swagger-codegen




  17. mvn package




  18. Make yourself a cup of coffee. Fry up some donuts while you’re at it - this will take awhile!




  19. When the dust settles, you want to see Build Success near the end of the Maven output. If Maven or one of its subprocesses exits with an error, you need to recheck your machine configuration and recheck that you have all the prerequisites installed and operational.




Generate a Looker API Client SDK


To generate a Looker API client SDK for Python, do this:



  1. cd looker_sdk

  2. java -jar ./swagger-codegen/modules/swagger-codegen-cli/target/swagger-codegen-cli.jar generate -i lookerapi.json -l python -o python_sdk


This will generate a Looker API client SDK for Python in the looker_sdk/python_sdk directory


You can use the same command to generate Looker API client SDKs for Java, .NET, and a variety of other languages and platforms supported by the Swagger-Codegen tools. Change the language target with the -l parameter and the output directory with the -o parameter. See https://github.com/swagger-api/swagger-codegen/tree/master/modules/swagger-codegen/src/main/resources and the Swagger-Codegen documentation for the list of supported languages and platforms.


Versioning


The generated client SDK is a snapshot of the “shape” of the Looker API at a single point in time. As new stuff is added in subsequent releases of the Looker product, your existing client SDK will continue to work just fine, but your client SDK will only be aware of API stuff that existed in the Looker product release that the client SDK was generated from.


If a new Looker API feature turns up in a future Looker product release that you want to use in your app, you will need to download a new lookerapi.json file from the new Looker product instance and use that to regenerate the client SDK files.


Use the Looker API Client SDK (for Python)


Among the files generated by Swagger-Codegen for Python is a subdirectory python_sdk/swagger_client which contains all the Python class definitions we need to make Python function calls to the Looker API.


For quick-and-dirty work, you can just copy the swagger_client directory (and its subdirectories) into a subdirectory of your Python project. Note that we’re changing the directory name to looker as part of the move (so that our Python code can refer to the client SDK module as “looker” instead of as “swagger_client”)

mv looker_sdk/python_sdk/swagger_client ~/mypythonapp/looker

cd ~/mypythonapp


Here’s a quick Python test app to kickstart your Looker API development.

Fill in your Looker API Host name and port, your API3 clientId and clientSecret values:


~/mypythonapp/test_app.py


import looker

# replace with your custom Looker API Host domain and port, if applicable.
base_url = 'https://your.looker.instance:19999/api/3.0/'
client_id = 'your-API3-client-id'
client_secret = 'your-API3-client-secret'

# instantiate Auth API
unauthenticated_client = looker.ApiClient(base_url)
unauthenticated_authApi = looker.ApiAuthApi(unauthenticated_client)

# authenticate client
token = unauthenticated_authApi.login(client_id=client_id, client_secret=client_secret)
client = looker.ApiClient(base_url, 'Authorization', 'token ' + token.access_token)

# instantiate User API client
userApi = looker.UserApi(client)
me = userApi.me();

print me

# instantiate Look API client
lookApi = looker.LookApi(client)

print lookApi.all_looks()

This test app imports the Looker API Python client SDK from the looker subdirectory of the test_app.py file’s home directory. The Python code can then access the Looker API classes via the looker module symbol.


This test app authenticates to the Looker API using the clientId and clientSecret values you generated earlier on the Looker Admin/User panel. It fetches the user info for the current logged-in user using the userApi.me() API call, and fetches a list of all looks accessible to the current user using the lookApi.all_looks() API call.


Audience Participation


The Swagger-Codegen generated Python client SDK code looks like it is intended to be installed as a Python package/module. Not being a Python expert, I haven’t had much success with naive attempts to install the client SDK files as a Python module.


SO… If any of you Python alpha geeks out there would like to chime in with tips on how to install the Python client SDK as a module (and successfully reference it from a Python app) the rest of us Python n00bs would be grateful!


Proceed Carefully


This test app is safe to use because it doesn’t use any Looker APIs that change the state of the Looker instance. Many Looker APIs, however, can change or delete settings, users, models, and more from your Looker instance. Be careful! You can limit what a program can do with the Looker API by using API3 client credentials associated with a limited-rights user account, and only enable roles or permissions on that user account as needed. It’s always a good idea to do development work using a non-production Looker instance.


Disclaimers


Looker provides full support for the use and execution of the HTTP Looker API operations themselves. Officially, Looker cannot provide support for bugs or issues in Swagger generated client SDKs.


If you find something wonky when making a Looker API call through a client SDK, try performing the same Looker API call using the equivalent raw HTTP/JSON request. If the raw HTTP request also behaves oddly, that’s on us. Please contact Looker support to let us know!


Pro Tip:


Use the Looker API doc page to experiment with raw HTTP requests. Auth is taken care of for you!


If the HTTP request behaves correctly but the equivalent client SDK call is broken, that may be a bug in the generated client SDK. Let us know if you find something like this so we can check for workarounds and make sure our JSON metadata is correct. Be aware that the extent of our support for client SDK issues will be limited to recommending that you try a different release of the Swagger-Codegen tools, and suggesting you post a cry for help from fellow Looker devs here in the Looker Discourse community. We can’t help you write your application code or your homework assignments.


65 replies

Userlevel 3

I posted the swagger-generated python sdk in pip. This now works:


pip install lookerapi


import lookerapi as looker

# replace with your custom Looker API Host domain and port, if applicable.
base_url = 'https://your.looker.instance:19999/api/3.0/'
client_id = 'your-API3-client-id'
client_secret = 'your-API3-client-secret'

# instantiate Auth API
unauthenticated_client = looker.ApiClient(base_url)
unauthenticated_authApi = looker.ApiAuthApi(unauthenticated_client)

# authenticate client
token = unauthenticated_authApi.login(client_id=client_id, client_secret=client_secret)
client = looker.ApiClient(base_url, 'Authorization', 'token ' + token.access_token)

# instantiate User API client
userApi = looker.UserApi(client)
me = userApi.me();

print me

# instantiate Look API client
lookApi = looker.LookApi(client)

print lookApi.all_looks()

May I suggest : https://github.com/looker-open-source/sdk-codegen/tree/master/python it’s managed by Looker

Newest release of swagger-codegen, v2.2.0, breaks this example. v2.1.6 works fine.

Thanks for your attention here, Danny. I love that y’all are so receptive and helpful on this forum 🙂 .

Userlevel 3

Hi Jerome – Unfortunately not yet (to my knowledge). It is on my to do list in the next couple weeks.


You have an interim solution?

Userlevel 2

Nice catch! The git clone statement had the wrong output directory. It was cloning swagger-codegen into the current directory (looker_sdk) but my intent was to clone into a swagger-codegen subdir (looker_sdk/swagger-codegen) to reduce clutter in the looker_sdk dir.


Instructions updated. Thanks!


-Danny

Hi Greg,


Yes, the SDK requires python 3.7. We suggest using pyenv if you want to manage multiple python versions on your system. See https://github.com/looker-open-source/sdk-codegen/tree/master/python for more details.

Wow, I forgot to comment on this. Sorry everyone!


The notes I have on how I did this say:


git clone git@github.com:swagger-api/swagger-codegen.git
pushd swagger-codegen
git checkout tags/v2.1.6
mvn package
popd
# manually download swagger.json
java -jar ./swagger-codegen/modules/swagger-codegen-cli/target/swagger-codegen-cli.jar generate -i swagger.json -l python -o sdk
mv sdk/swagger_client sdk/pylooker

sed -i.orig 's/"https:\/\/looker-instance-hostname:19999\/api\/3.0"/None/' sdk/pylooker/configuration.py
sed -i.orig 's/swagger_client/pylooker/' sdk/pylooker/configuration.py
sed -i.orig 's/swagger_client/pylooker/' sdk/setup.py

rm sdk/pylooker/configuration.py.orig
rm sdk/setup.py.orig

cd sdk
python setup.py sdist

At that point, I’m using dir2pi to get the directory structure right for our internal, S3-based package index. You wouldn’t do that to get it to PyPI, though. The first resource I found on how to get something up to PyPI is here (scroll to slide 20). It’s been a while since I’ve uploaded something to PyPI (oops), so I can’t remember if there’s any hiccups beyond that, but hopefully this is a start.

FWIW, when creating a Python SDK v.2.3.1 for Python 3.6.4, I found I needed to change the sample usage script:


# instantiate Auth API
unauthed_client = looker.ApiClient(configuration=None)
unauthed_auth_api = looker.ApiAuthApi(unauthed_client)

# authenticate client
token = unauthed_auth_api.login(
client_id=client_id, client_secret=client_secret)
client = looker.ApiClient(
header_name='Authorization',
header_value='token ' + token.access_token)

The reason is that if I merely passed strings as positional parameters, then the configuration object wasn’t an object, but was just strings, which don’t have the configuration object’s methods, so the SDK code would break.

Thanks, this took me ages to work out!


I am stuck with 2 problem at present.


Problem 1:

In the below process step



  1. Open a new browser window and navigate to this URL: https://:19999/api/3.0/swagger.json Subsitute your custom Looker API Host domain name and port if necessary.

  2. Save the contents of that swagger.json page to looker_sdk/lookerapi.json.


I am able to browse toward the URL but that has many in list. Should I manually copy paste them and save it to lookerapi.json or only which is required. Like authentication alone ?


Problem 2: GIT


when I try to access GIT

git clone git@github.com:swagger-api/swagger-codegen.git ./swagger-codegen


I am prompted with error


Cloning into ‘swagger-codegen’…

ssh: connect to host github.com port 22: Connection refused

fatal: Could not read from remote repository.


Please make sure you have the correct access rights


Can you please help me with this ?


Thanks

Userlevel 7
Badge +1

I think that would be a question for @jax. After a quick test it looks like after setting env variables (make sure to name them properly like LOOKERSDK_BASE_URL, for example), you can just run it with no args and it will capture the environment variables.

The Java SDK Example


ApiClient lookerApiClient = new ApiClient();

lookerApiClient.setBasePath(“https://xxxxx:19999/api/3.0”);

ApiAuthApi authAPI = new ApiAuthApi(lookerApiClient);

AccessToken authenticatingToken = authAPI.login(clientid, clientkey);

lookerApiClient.addDefaultHeader(“Authorization”, "token " + authenticatingToken.getAccessToken());

DashboardApi dashboardApi = new DashboardApi(lookerApiClient);

Userlevel 2

That’s what we used – my main point was that it still took a fair amount of futzing around to get everything for the 3.1 SDK working correctly, even after starting with the official Looker tool. Nothing deeply technical, at the end of the day, but just chasing around various config issues that needed to be cleaned up, etc.


(Also, to be clear, we were using the code-gen for 3.1 very soon after it was released for any public consumption at all, and clearly labeled as “not-yet-official”. A lot of those the little issues we ran into have hopefully been addressed in the meantime.)

Userlevel 2

Aaaand…Looker 6.22 Release Notes

Hey Dan,


It looks like LookerEmbedClientExample is the Java version of the SSO script you linked to here,

but this is for generating an SSO Login URL.


To generate a Swagger SDK for Java, follow the above instructions until you get to the header “Generate a Looker API Client SDK.” After the command there is a paragraph describing how to alter the statement to get a Java SDK.



You can use the same command to generate Looker API client SDKs for Java, .NET, and a variety of other languages and platforms supported by the Swagger-Codegen tools. Change the language target with the -l parameter and the output directory with the -o parameter.



Let me know if you have any more questions.


Spencer

@dgroman1988


Ah I see, thanks for clarifying. At this time there aren’t tangible examples of using the Looker API with Java but I will let our Product/Docs team know these would be helpful.


We do have these examples in Python, but it doesn’t look like we currently have the same thing for Java.

+1 for getting this up on PyPI

sweet – i think that actually helps clarify for me some problems i’ve been having trying to put this up on our internal package index. thanks samuel!


I feel like maintenance of a public version on PyPI should be on Looker, particularly since I’m guessing different Looker customers receive versions (and any modifications to the way the SDK/API is structured) at different cadences? but then again, maybe changes will only be additive from here out. also, we might be the only ones doing this, so ¯_(ツ)_/¯

Userlevel 3

Hi Rob,


I’ll continue updating this in pip. But if you want to see the steps to publish it, it was really just having the swagger output and then the recipe from PyPI itself:

https://packaging.python.org/tutorials/packaging-projects/


Hopefully that points you in the right direction, and please share any feedback on the pip package!

Userlevel 2

Shouldn’t 4.1 under setup should say cd looker_sdk/modules/swagger-codegen instead of cd looker_sdk/swagger-codegen?


Otherwise, very helpful.


AK

This is really cool. FYI, the documentation output for PHP says that no endpoints require authorization (which is obviously not true). Maybe this is something that could be fixed in swagger.json?

@danny Here’s one doc that was spit out by swagger-codegen. Search for “Authorization” on the page:


I don’t believe I did anything above and beyond the instructions in your original post. Thanks for looking into this @danny!


Has anyone generated a python SDK for API 3.1 yet?

Reply