Question

Generating Client SDKs for the Looker API

  • 16 August 2016
  • 65 replies
  • 1779 views

Userlevel 2

Update: This article is now out of date. For the latest info on how to download or generate client SDKs for the Looker API, please see our Looker SDK Codegen repo on Github






The Looker API is a collection of “RESTful” operations that enables you to tap into the power of the Looker data platform in your own applications. The Looker API can be invoked using plain old HTTP(S) requests. Any development tool, language, or application that can make HTTP requests and ingest JSON responses should be able to use the Looker API.


Web geeks who live and breathe HTTP and/or AJAX XHR will feel at home using just the “raw” Looker API HTTP URL endpoints. GET, POST, PUT, PATCH, DELETE, oh my!


Web APIs For The ‘REST’ of Us


If writing HTTP requests is not something you or your developers do every day, accessing a REST API for the first time can be a little intimidating and disorienting. Programming with web requests often requires different idioms and patterns of behavior than traditional app development. What do you do if you’re more interested in the data results than in the journey required to get them?


What non-web devs desire is something that makes the Looker API feel like a set of functions in your language of choice: a client SDK. A client SDK is a set of functions that resides in your application that encapsulates all the HTTP goop so you can just call functions and get results.


The Looker API provides a metadata document which describes all the URL endpoints, their parameter names and types, and their response types. From that metadata you can use Swagger tools to generate client SDK libraries for your programming language.


This allows us to make the Looker data platform accessible via convenient client libraries for a wide variety of programming languages, without having to be experts in all of those languages or having to commit resources to developing and supporting all those languages. It also helps ensure consistency across all manifestations of these client SDKs which would be very difficult to manage if each SDK were hand-written.


Enough back story. Let’s generate a Looker API client SDK!


Prerequisites




  1. Login to your Looker instance as an admin to enable and configure the Looker API: https://<your_looker_endpoint>/admin/api. Make note of your Looker API Host url displayed here if you have customized it from the default.




  2. Generate API3 credentials by navigating to https://<your_looker_endpoint>/admin/users, editing your user, and clicking “New API3 Key.” You’ll need the API3 clientId and clientSecret values later to authenticate your client application.




  3. With your API configured and credentials in hand, navigate to your Looker API documentation: https://<your_looker_endpoint>:19999/api-docs/index.html. (Substitute your custom Looker API Host domain and port if applicable)

    This page is our reference for available Looker API function/endpoints specific to our Looker instance. Since this doc page is generated from the Looker API metadata, this doc page may show beta or experimental API functions that are not yet officially documented elsewhere.




  4. Make sure Python 2.7 or 3.0 is installed on your computer. At a command line / shell prompt, run python --version to see what version of Python, if any, is installed.




  5. You’ll need a Java runtime distribution installed on your computer. We recommend at least Java 8 (version 1.8.*). Run java -version to verify version installed.




  6. Make sure Maven 3.3 or later is installed on your computer. At a command line / shell prompt, run mvn --version to see what version of Maven is installed.




  7. You’ll also need to have git client tools installed on your computer. Check version with git --version




Setup




  1. Make a directory to house your Looker SDK bits: cd ~/Desktop; mkdir looker_sdk; cd looker_sdk




  2. Download your Looker API swagger.json metadata.




  3. Login to your Looker instance as admin and navigate to the admin/api panel. (https://<your-looker-endpoint>:9999/admin/api).




  4. Set “Documentation Access” to “Allow anyone to see API docs”




  5. Click the Save button.




  6. Open a new browser window and navigate to this URL: https://<your-looker-endpoint>:19999/api/3.0/swagger.json Subsitute your custom Looker API Host domain name and port if necessary.




  7. Save the contents of that swagger.json page to looker_sdk/lookerapi.json.




  8. Take a peek inside the saved file to make sure the browser saved it as JSON text, not transmogrified HTML.




  9. Return to your Looker instance Admin/API page and set the “Documentation Access” setting back to what it was before and click Save. “Require API login to see API docs” is the default setting and is required if you want to use the “Try it!” button to call API functions directly from the API doc page. (You do want this, it’s cool!)




  10. Download the Swagger-Codegen tools from github.

    Note: Swagger-Codegen is in active development. To avoid the headaches of “bleeding edge” development, it’s a good idea to pull from a stable release. At the time of this writing, we recommend using Swagger-Codegen release 2.1.6. (If you’re using Python 3.0 or later, you’ll want Swagger-Codegen release 2.2.2 instead)

    Use these steps to checkout the swagger-codegen tools into your looker_sdk/swagger-codegen directory:




  11. cd looker_sdk




  12. git clone git@github.com:swagger-api/swagger-codegen.git ./swagger-codegen




  13. cd swagger-codegen




  14. git checkout tags/v2.1.6




  15. Build the Swagger-Codegen tools




  16. cd looker_sdk/swagger-codegen




  17. mvn package




  18. Make yourself a cup of coffee. Fry up some donuts while you’re at it - this will take awhile!




  19. When the dust settles, you want to see Build Success near the end of the Maven output. If Maven or one of its subprocesses exits with an error, you need to recheck your machine configuration and recheck that you have all the prerequisites installed and operational.




Generate a Looker API Client SDK


To generate a Looker API client SDK for Python, do this:



  1. cd looker_sdk

  2. java -jar ./swagger-codegen/modules/swagger-codegen-cli/target/swagger-codegen-cli.jar generate -i lookerapi.json -l python -o python_sdk


This will generate a Looker API client SDK for Python in the looker_sdk/python_sdk directory


You can use the same command to generate Looker API client SDKs for Java, .NET, and a variety of other languages and platforms supported by the Swagger-Codegen tools. Change the language target with the -l parameter and the output directory with the -o parameter. See https://github.com/swagger-api/swagger-codegen/tree/master/modules/swagger-codegen/src/main/resources and the Swagger-Codegen documentation for the list of supported languages and platforms.


Versioning


The generated client SDK is a snapshot of the “shape” of the Looker API at a single point in time. As new stuff is added in subsequent releases of the Looker product, your existing client SDK will continue to work just fine, but your client SDK will only be aware of API stuff that existed in the Looker product release that the client SDK was generated from.


If a new Looker API feature turns up in a future Looker product release that you want to use in your app, you will need to download a new lookerapi.json file from the new Looker product instance and use that to regenerate the client SDK files.


Use the Looker API Client SDK (for Python)


Among the files generated by Swagger-Codegen for Python is a subdirectory python_sdk/swagger_client which contains all the Python class definitions we need to make Python function calls to the Looker API.


For quick-and-dirty work, you can just copy the swagger_client directory (and its subdirectories) into a subdirectory of your Python project. Note that we’re changing the directory name to looker as part of the move (so that our Python code can refer to the client SDK module as “looker” instead of as “swagger_client”)

mv looker_sdk/python_sdk/swagger_client ~/mypythonapp/looker

cd ~/mypythonapp


Here’s a quick Python test app to kickstart your Looker API development.

Fill in your Looker API Host name and port, your API3 clientId and clientSecret values:


~/mypythonapp/test_app.py


import looker

# replace with your custom Looker API Host domain and port, if applicable.
base_url = 'https://your.looker.instance:19999/api/3.0/'
client_id = 'your-API3-client-id'
client_secret = 'your-API3-client-secret'

# instantiate Auth API
unauthenticated_client = looker.ApiClient(base_url)
unauthenticated_authApi = looker.ApiAuthApi(unauthenticated_client)

# authenticate client
token = unauthenticated_authApi.login(client_id=client_id, client_secret=client_secret)
client = looker.ApiClient(base_url, 'Authorization', 'token ' + token.access_token)

# instantiate User API client
userApi = looker.UserApi(client)
me = userApi.me();

print me

# instantiate Look API client
lookApi = looker.LookApi(client)

print lookApi.all_looks()

This test app imports the Looker API Python client SDK from the looker subdirectory of the test_app.py file’s home directory. The Python code can then access the Looker API classes via the looker module symbol.


This test app authenticates to the Looker API using the clientId and clientSecret values you generated earlier on the Looker Admin/User panel. It fetches the user info for the current logged-in user using the userApi.me() API call, and fetches a list of all looks accessible to the current user using the lookApi.all_looks() API call.


Audience Participation


The Swagger-Codegen generated Python client SDK code looks like it is intended to be installed as a Python package/module. Not being a Python expert, I haven’t had much success with naive attempts to install the client SDK files as a Python module.


SO… If any of you Python alpha geeks out there would like to chime in with tips on how to install the Python client SDK as a module (and successfully reference it from a Python app) the rest of us Python n00bs would be grateful!


Proceed Carefully


This test app is safe to use because it doesn’t use any Looker APIs that change the state of the Looker instance. Many Looker APIs, however, can change or delete settings, users, models, and more from your Looker instance. Be careful! You can limit what a program can do with the Looker API by using API3 client credentials associated with a limited-rights user account, and only enable roles or permissions on that user account as needed. It’s always a good idea to do development work using a non-production Looker instance.


Disclaimers


Looker provides full support for the use and execution of the HTTP Looker API operations themselves. Officially, Looker cannot provide support for bugs or issues in Swagger generated client SDKs.


If you find something wonky when making a Looker API call through a client SDK, try performing the same Looker API call using the equivalent raw HTTP/JSON request. If the raw HTTP request also behaves oddly, that’s on us. Please contact Looker support to let us know!


Pro Tip:


Use the Looker API doc page to experiment with raw HTTP requests. Auth is taken care of for you!


If the HTTP request behaves correctly but the equivalent client SDK call is broken, that may be a bug in the generated client SDK. Let us know if you find something like this so we can check for workarounds and make sure our JSON metadata is correct. Be aware that the extent of our support for client SDK issues will be limited to recommending that you try a different release of the Swagger-Codegen tools, and suggesting you post a cry for help from fellow Looker devs here in the Looker Discourse community. We can’t help you write your application code or your homework assignments.


65 replies

Shouldn’t 4.1 under setup should say cd looker_sdk/modules/swagger-codegen instead of cd looker_sdk/swagger-codegen?


Otherwise, very helpful.


AK

Userlevel 2

Nice catch! The git clone statement had the wrong output directory. It was cloning swagger-codegen into the current directory (looker_sdk) but my intent was to clone into a swagger-codegen subdir (looker_sdk/swagger-codegen) to reduce clutter in the looker_sdk dir.


Instructions updated. Thanks!


-Danny

Newest release of swagger-codegen, v2.2.0, breaks this example. v2.1.6 works fine.

Userlevel 2

Thanks for the info! There have been issues in the swagger codgen tip recently, so we recommend sticking to the v2.1.6 version of the Swagger codegen tools for now.


There was an issue in a June/July 2016 revision of the Python swagger codegen which broke serialization of API response types that contained read-only properties. I believe the swagger codegen team has since checked in a fix for that, but I haven’t had a chance to test it or recertify the Looker API on newer releases of Swagger codegen tools.


-Danny

Userlevel 1

In Setup, paragraph 3, found it necessary during setup to be in the swagger-codegen directory:


###3. Download the Swagger-Codegen tools from github.



  1. cd looker_sdk

  2. git clone git@github.com:swagger-api/swagger-codegen.git ./swagger-codegen


  3. cd looker_sdk/swagger-codegen <-- make sure you’re in the right directory

  4. git checkout tags/v2.1.6

After cloning swagger-codegen, you may want to include a cd swagger-codegen step.

Didn’t see @dion’s post before I wrote this.

This is really cool. FYI, the documentation output for PHP says that no endpoints require authorization (which is obviously not true). Maybe this is something that could be fixed in swagger.json?

Userlevel 2

Hi rblair,


Could you point me to the PHP documentation output you refer to? Filename, line number? Thanks!


-Danny

@danny Here’s one doc that was spit out by swagger-codegen. Search for “Authorization” on the page:


Userlevel 2

Ok, thanks! Did you use a particular command line switch to get swagger to emit an API doc .md file? I haven’t seen .md output with the Java, C# or Python swagger codegen modules.


I suspect the root of this is that Swagger has a notion of authentication which doesn’t fit the Looker API auth model, so we opt out of the Swagger auth thingie. The PHP swagger generator may be taking that flag as gospel when emitting doc. I’ll have to do some testing to see if turning that flag back on will screw up the Try Me! button on the SwaggerUI api doc pages. If it does, some surgery to the api doc code may be required.


Thanks for reporting this! I’ll see what we can do.


-Danny

I don’t believe I did anything above and beyond the instructions in your original post. Thanks for looking into this @danny!

Regarding the “Audience Participation” part, I’ve gotten the generated SDK published on an internal package index. Would there be interest in getting a pre-generated Looker client up on PyPI?

Userlevel 2

Hi Samuel,


I’d love some pointers on how to get a prebuilt Python Looker SDK package published on PyPi. I’m not familiar with the Python space so any steps or tips you can provide would be a big help. Thank you!


(I’m currently working on publishing our Looker Ruby SDK client on rubygems.org)


-Danny

+1 for getting this up on PyPI

Hi There, I followed the steps and generated a Looker API client SDK for C#. Thanks for the instructions. They are really helpful.


But I don’t know what are the next steps for me to include those C# files in my project. I don’t have much experience with C#, I have to use it so that we can save a looker dashboard as a pdf file right after the data is processed in c#.


Can someone can help to write a similar example in C# like the example here in python?


Great thanks!

Wow, I forgot to comment on this. Sorry everyone!


The notes I have on how I did this say:


git clone git@github.com:swagger-api/swagger-codegen.git
pushd swagger-codegen
git checkout tags/v2.1.6
mvn package
popd
# manually download swagger.json
java -jar ./swagger-codegen/modules/swagger-codegen-cli/target/swagger-codegen-cli.jar generate -i swagger.json -l python -o sdk
mv sdk/swagger_client sdk/pylooker

sed -i.orig 's/"https:\/\/looker-instance-hostname:19999\/api\/3.0"/None/' sdk/pylooker/configuration.py
sed -i.orig 's/swagger_client/pylooker/' sdk/pylooker/configuration.py
sed -i.orig 's/swagger_client/pylooker/' sdk/setup.py

rm sdk/pylooker/configuration.py.orig
rm sdk/setup.py.orig

cd sdk
python setup.py sdist

At that point, I’m using dir2pi to get the directory structure right for our internal, S3-based package index. You wouldn’t do that to get it to PyPI, though. The first resource I found on how to get something up to PyPI is here (scroll to slide 20). It’s been a while since I’ve uploaded something to PyPI (oops), so I can’t remember if there’s any hiccups beyond that, but hopefully this is a start.

sweet – i think that actually helps clarify for me some problems i’ve been having trying to put this up on our internal package index. thanks samuel!


I feel like maintenance of a public version on PyPI should be on Looker, particularly since I’m guessing different Looker customers receive versions (and any modifications to the way the SDK/API is structured) at different cadences? but then again, maybe changes will only be additive from here out. also, we might be the only ones doing this, so ¯_(ツ)_/¯

Userlevel 2

Thanks for the tips, Samuel!

Userlevel 2

Hi Chris,


Having pre-built Looker SDKs for a variety of programming languages is definitely on my wish list.


This requires two steps:



  • Learning how to build and distribute packages in each of those languages

  • Integrating that build & distribute into our Looker release process


We’d only be able to do this for the top few languages customers use to call our API. Everyone else will still have to use the Swagger codegen process to generate a client SDK in their language, or roll-their-own SDK that wraps the REST endpoints directly.


Good news for you! Python is already in the top 5 languages used to call the Looker API. 🙂


I can’t offer any timeframes for getting this investigated and done, as I have to fit this work in around my other responsibilities. (shh! New Looker APIs in the works!) But I hear ya and I definitely want to make getting up to speed with Looker client SDKs a lot easier for you all!


-Danny

Thanks for your attention here, Danny. I love that y’all are so receptive and helpful on this forum 🙂 .

Installing as a python module is easy with pip:


$ ls
lookerapi.json python_sdk swagger-codegen swagger-codegen-cli.jar
$ cd python_sdk/
$ ls
docs README.md setup.py test tox.ini
git_push.sh requirements.txt swagger_client test-requirements.txt
$ pip install .

Note this will install the module as “swagger_client”, unless you rename the directory.

Virtual environment highly recommended.


The sdk seems to work fine with Python 3.6. I tried using the current stable version of swagger-codegen (2.2.3), but got an error, something to do with “can’t create Role” (can’t remember exactly and it has scrolled off my terminal). Using swagger-codegen v2.2.2 as specified above, no problems.


Very nice API and tutorial, many thanks!

I get the following error starting the server after I generated a nodejs package with codegen:


API Errors:

#/paths/~1dashboards~1{dashboard_id}~1prefetch/get/parameters/1: Not a valid parameter definition
#/paths/~1dashboards~1{dashboard_id}~1prefetch/get/parameters/1: Not a valid parameter definition
#: Missing required property: schema
#: Not a valid undefined definition
#/items: Additional properties not allowed: $ref
#/items: Additional properties not allowed: $ref
#/items: Additional properties not allowed: $ref
#/items: Additional properties not allowed: $ref
#/paths/~1dashboards~1{dashboard_id}~1prefetch/get/parameters/1: Missing required property: $ref

1 error and 0 warnings

It appears the dashboard_filters query arg in the dashboard prefetch request is supposed to be an array of objects which I believe is disallowed by OpenAPI spec.

After I removed this section from swagger.json and regenerated the client I could run the sample server


{
"name": "dashboard_filters",
"in": "query",
"description": "JSON encoded string of Dashboard filters that were applied to prefetch",
"required": false,
"type": "array",
"items": {
"$ref": "#/definitions/PrefetchDashboardFilterValue"
},
"collectionFormat": "csv"
}

I am stuck with 2 problem at present.


Problem 1:

In the below process step



  1. Open a new browser window and navigate to this URL: https://:19999/api/3.0/swagger.json Subsitute your custom Looker API Host domain name and port if necessary.

  2. Save the contents of that swagger.json page to looker_sdk/lookerapi.json.


I am able to browse toward the URL but that has many in list. Should I manually copy paste them and save it to lookerapi.json or only which is required. Like authentication alone ?


Problem 2: GIT


when I try to access GIT

git clone git@github.com:swagger-api/swagger-codegen.git ./swagger-codegen


I am prompted with error


Cloning into ‘swagger-codegen’…

ssh: connect to host github.com port 22: Connection refused

fatal: Could not read from remote repository.


Please make sure you have the correct access rights


Can you please help me with this ?


Thanks

Reply