Git Workflow Using Multiple Repositories Across Multiple Instances—Development, Staging, and Production

  • 23 June 2022
  • 0 replies
  • 1359 views

Userlevel 4

Looker will not be updating this content, nor guarantees that everything is up-to-date. 

Starting in Looker 7.12, you can deploy any Git commit SHA, tag, or branch to production with advanced deploy mode. This helps consolidate repositories in multi-environment developer workflows, where each environment points to a different version of a codebase. It also gives one or a few developers or administrators greater control over the changes that are deployed to production.
 

Motivation


By setting up a Git workflow across multiple Looker host instances, you can create a dedicated development environment for developers while preventing end users from accessing experimental code. All development and code review is done in a development or staging instance. Once changes pass your team's quality assurance review, they are deployed to a staging or production instance as desired.

This article describes how to set up a workflow among multiple instances and multiple Git repositories. If you have multiple instances and one Git repository, see the Git Workflow Using One Repository Across Multiple Instances — Development, Staging, and Production Help Center article.

Looker recommends using a single repository whenever possible.
 

Advanced Git Setup for Multiple Repositories
 

This setup requires multiple Git repositories and knowledge of Git commands.

You may want to set up multiple Looker hosts to achieve a development workflow that looks something like:

[1] Development > Production

[2] Development > Staging > Production

Each of these stages resides on a dedicated Looker host instance. This configuration uses a manual Git process to move code between two separate GitHub repos that can be automated or kicked off by a button.

Although this method shows a development workflow configuration [1] based on an existing single Looker project, looker_project, it could easily be adapted for [2] or any number of intermediate hosts between Development and Production as needed.
 

Overview

The general process is as follows:

  1. Create two GitHub repos, dev and prod.
  2. Associate each repo with its respective Looker instances, Development and Production, according to the Setting Up and Testing a Git Connection documentation page.
  3. Create a local Git repo that is a clone of dev, perhaps on a server owned by the approver. Add the prod GitHub repo as a second remote:
    git remote add production <<ssh_url>>
  4. Create a webhook trigger in the production GitHub repo to notify prod Looker to sync with GitHub as new code is pushed. In essence, set up a webhook on a push event to notify the Production Looker instance.
  5. Implement some kind of quality control around the second item in the migration process shown above, based on what works best for your team.

Step-by-Step Guide

 

To designate the existing devrepo and set up a prod repo on a second host, follow these steps:

  1. In the Github UI, go to the existing project, select Settings, and ensure that master is set as the default branch. If a different branch is set as default, change the default to master.

     

  2. Create a local folder and clone the looker_project:
    mkdir ~/Desktop/looker_project
    cd ~/Desktop/looker_project
    git clone git@github.com:MyOrgNameInGithub/looker_project.git
  3. Switch to the cloned project and make sure you're on the master branch:
    cd looker_project
    git checkout master
  4. If we view the Git remotes for the cloned repo, we should see origin as the original repo location:
    git remote -v    
  5. Create a new empty repo via the GitHub UI. For our example, let's name the repo looker_project_prod to house our prod project:

     

  6. Add our new empty repo as a new remote for the cloned repo, as below:
    git remote add prod git@github.com:MyOrgName/looker_project_prod.git    
    In this case, prod is simply a name we assign to the remote; it can be whatever we want. We can verify the remote repo with this command:
    git remote -v   
  7. Perform an initial push to our new remote of the master branch:
    git checkout master    
    This will ensure we're on the correct branch:
    git push prod master    
    This will push the master branch of our cloned repo to our new remote, prod. In the GitHub UI, we should see that the cloned LookML has been pushed to our new project on the master branch.

    Now, create a new empty project on the Production Looker instance:
  8. Enter Development Mode, select the Develop menu, then Manage LookML Projects:

     

  9. On the LookML Project page, click New LookML Project.

  10. Choose a name (such as looker_project_prod in our example), and select Blank Project.

     

  11. Click Create Project.

  12. Follow Looker Git setup instruction prompts to connect to the new repo. For a complete tutorial, see the Setting Up and Testing a Git Connection documentation page. Ensure that you provide the deploy key read/write access.

  13. Follow the prompts in the Looker IDE to deploy to production. Looker will pull in the code from the new prod repo to the local Looker file system.
  14. Next, configure the new repo via the Github UI to hit Looker's deploy webhook on the production host.

    In the GitHub UI, select Settings, then Webhooks, then Add webhook.

  15. Enter the URL in the following format: [instance_name.looker.com]/webhooks/projects/[project name]/deploy

    For example: https://instance_name.looker.com/webhooks/projects/looker_project_prod/deploy

    This will cause the Looker host to pull the master branch from the Git repo to its local file system, making the changes visible.

Once configured per the above, go to your Project Settings page and reset the Git connection using the Looker UI on the prod host:

In the Github UI, we can make the SSH key read-only for the prod repo by not selecting Allow write access, so that no development can happen on the Production instance.

This means that any user entering Development Mode (including during setup) will see the error below while syncing their Development Mode. This, in effect, is what we want.

Proposed Development Workflow

  • LookML development takes place on the dev Looker instance, and all code is committed and deployed via the Looker UI. This pushes the LookML to the master branch of the development repo.
  • We can enable pull requests to allow for code review before commits are merged into master, per the instructions on the Setting Up Version Control documentation page.
  • To move LookML from the dev repo to the prod repo, we just need to run the following in the cloned repo (while on the master branch):
    git pull
    git push prod master
  • When the prod repo detects the push, GitHub will hit the deploy webhook on the Looker prod instance to pull in these changes.

This topic has been closed for comments