Git Deployment Strategies with Looker

[davidtamaki]

Looker will not be updating this content, nor guarantees that everything is up-to-date. 

Introduction

The purpose of this article is to help development teams choose a Git flow strategy that best suits their team structure, delivery cadence, and code review requirements.

This article provides three flow examples, each with options for either a single instance or a multi-instance setup. The examples are presented in order of their complexity and rigor. Choose the option that achieves the right balance for your use case.

The following three workflow examples are described in this article:

1. GitHub workflow
2. GitLab workflow
3. Gitflow workflow

A fourth workflow, for multiple Git repos, is included, but this approach is not recommended.
 

Note: pushing to non-production branches

By default, Looker will deploy all changes to a repository's default production branch. The Looker documentation on Getting your changes to production describes the options for modifying the default behavior, including the following:

• You can use the Git Production Branch Name field to specify which branch from your Git repository Looker should use as the target branch into which your Looker developers' branches are merged. See the Configuring project version control settings documentation page for details.
• You can use advanced deploy mode for specifying a different commit SHA or tag name to deploy to your Looker production environment, instead of using the latest commit on the production branch. (If you want to deploy a commit from a different branch, you can use the advanced deploy mode webhook or API endpoint.) See the Advanced deploy mode documentation page for details.

1. GitHub workflow

The default Looker GitHub workflow looks like this for single and multiple Looker instances:

The deployment process is as follows. For each new feature:

1. Create a new shared branch in Looker.
2. Commit changes.
3. [Optional] If you've enabled pull requests for the project, submit a pull request for review.
4. Merge the development branch into the production branch.
5. Deploy to production. (Or, stage changes and use advanced deploy mode.)
6. If you're using a multi-instance setup, promote changes downstream for each instance (often by using the deploy webhook).

These workflows are based on the following:

• GitHub flow (Scott Chacon)
• Understanding the GitHub flow (GitHub Guides)

Other things to note:

• This option offers simple one-directional flow of the main branch.
• It's great for small teams that are moving fast and just beginning LookML development.
• Git commands:
  • Core functionality is available inside the Looker IDE and Git UI for managing PRs.
• Considerations:
  • This approach assumes anything that is deployed to the Production Mode branch has been thoroughly tested. Hot fixes are not treated any differently than a small feature branch.

2. GitLab workflow

The GitLab workflow is recommended for multiple instance setup, and it looks like this for single and multiple Looker instances:

The deployment process is as follows for a single instance setup. For each new feature:

1. Create a new shared branch in Looker.
2. Commit changes.
3. When the feature is ready for review, submit a pull request.
4. Review and approve changes.
5. Merge the development branch into the production branch.
6. Create a release branch off of the main branch (on a sprint or timed interval).
7. Perform additional tests on release branch.
8. Export or tag content with the release name.
9. Deploy the release to the production branch using advanced deploy mode.

The process differs slightly for a multi-instance setup. For each new feature:

1. Create a new shared branch in Looker.
2. Commit changes.
3. When the feature is ready for review, submit a pull request.
4. Review and approve changes.
5. Merge the development branch into the production branch.
6. Create a release branch off of the main branch (on a sprint or timed interval).
7. Deploy the release branch to the staging instance's production branch using advanced deploy mode.
8. Perform end-to-end tests on the staging instance.
9. Import, export, and tag content with the release name.
10. Deploy the release to the production instance's production branch using advanced deploy mode.

These workflows are based on the following:

• What is GitLab flow?
• Introduction to GitLab flow | GitLab

Other things to note:

• With this option, the commits flow downstream (feature branches -> main -> release n branches).
• Git commands:
  • Core functionality is available inside the Looker IDE and the Git UI for managing PRs.
  • Hot fixes
    • You merge commits directly to a release branch. Ensure that those commits are cherry-picked back to the main branch. Cherry-picking will need to be done outside of the Looker IDE.
    • Alternatively, you merge commits to the main branch and cherry-pick to the downstream branches (this also needs to be done outside of the Looker IDE).
    • Example:

      git checkout release-1.12 # checkout release branch
      git cherry-pick 5f82... # duplicate commit into release branch
      # deal with any potential merge conflicts
      git push origin release-1.12 # push commits to remote
      # use advanced deploy or hit project deploy webhook
      git tag v1.12.1 # tag the patch version
      git push origin v1.12.1 # push tag to remote              

    • Other useful commands:

      git fetch # fetch branches from remote
      git branch -r # see all remote branches
      git cherry-pick --abort # abort cherry pick             

• Considerations:
  • Looker recommends that you deploy branches with advanced deploy mode (for example, the head of a branch), rather than a specific commit SHA or tag. This will make it simpler to perform hot fixes because you can create a new branch from the Production Mode branch inside the Looker IDE. If production is deployed with a commit SHA or tag, then you may need to use Git on the command line to create a new branch (outside of the Looker IDE).
  • If you use the deployment manager, you can only select commits to deploy to production. For the same reason as described above, this could make it difficult to perform a hot fix, as there could be commits on the production branch that are ahead of the release branches.
  • Using the deploy_ref_to_production endpoint is the suggested approach to be able to quickly deploy hot fixes.
  • In a single instance setup, there are certain aspects that cannot be tested outside of Production Mode (for example, datagroups, schedules, embeds). Refer to this article to understand these limitations.

3. Gitflow workflow

This workflow offers more control, but is probably overkill for most Looker setups.

This workflow is based on the following:

• A successful Git branching model (nvie.com)
• Gitflow workflow

Other things to note:

• This flow has five distinct branch types: feature(s), develop, release(s), hot fix(es), and main. The develop and main branches are long-lived branches.
• This flow is not recommended for LookML development because the process can be difficult to integrate with a continuous deployment model. It also emphasizes specific Git commands that are not available with Looker's Git integration, which would complicate your ability to orchestrate Git management outside of Looker's LookML IDE.
• Specifically, at the time of writing, there is no way for developers to merge changes from one branch into another non-production branch from within the LookML IDE. Gitflow depends on frequent merging between branches, so this flow may add excessive complexity to your environment.

4. Multiple Git repos

This legacy approach and is not recommended.

This workflow is based on:

• Git workflow using multiple repositories across multiple instances — development, staging, and produ...

Other things to note:

• This workflow was used before the advanced deploy functionality was created to manage multiple instance deployments.
• The workflow will create one Git repository per-project per-instance. This will result in many repos (N instances * M projects), which can be a burden to manage.

Automated testing

The following are some guidelines around automated testing. These can be used with any of the previous options.

• After each commit, validate LookML logic with data tests.
• Use event-based CI flows to validate PRs. Some common examples include:
  • Validate query results and run content validation using the Looker SDK.
  • Enforce LookML style guide and linting with LAMS.
• Perform content validation before deploying to production.
  • If you're using multiple instances, perform validation in a staging instance.
  • Decide which content will be managed (the development team is responsible for updating) and which will be out of scope (content owners are responsible for updating). Generally, having a select group of shared folders to validate is a balanced approach. For any other changes to out-of-scope content, use release notes to notify the Looker user community of any upcoming changes.

Considerations

• For multi-instance setups, consider adding the deploy key as read-only on the staging and production instances to prevent any accidental development. 
• Consider configuring a deploy secret so that only authorized users can trigger a deployment.
• Consider leveraging project import to simplify the model using a hub-and-spoke framework. This allows smaller projects to validate quicker and makes it easier to share common logic to disparate teams downstream.
• For multi-instance setups, consider leveraging user attributes and constants to share common values such as connection names, database schema names, or dataset names. See this article for an example.
• If you deploy the project with webhooks or API, and do not use the deployment manager GUI, there is no visible indication for the production deploy hash in the LookML IDE. Consider adding a comment in the project manifest file for a quick visual indicator to confirm which version has been deployed.

Choosing a deployment mechanism

In addition to the workflow options discussed previously, there are multiple different mechanisms to choose from to deploy to production. The following table shows a summary of the different approaches:

Mechanism	Easy to revert?	Easy hot fix?	GUI?	Automated?
Deployment manager via commits	Yes	No	Yes	No
Git webhook via commits	Yes	No	No	Yes
Git webhook via branches	Yes	Yes	No	Yes
API / SDK via commits	Yes	No	No	Yes with CI/CD workflows
API / SDK via branches	Yes	Yes	No	Yes with CI/CD workflows

Additional resources

• Refer to the deploy_looker.py example in the Looker multistage development framework Help Center article for an example of utilizing the deploy_ref_to_production endpoint using the Looker Python SDK.
• For a multi-instance set up, it's recommended to use advanced deploy mode like this:
  • Disable advanced deploy mode in the development instance. Advanced deploy mode is by default off in the regular Git integration. With advanced deploy mode off, Looker automatically merges commits and deploys them to the instance.
  • Enable advanced deploy mode on the staging and production instances. With advanced deploy mode on, the Looker instance requires an admin to select the commit to be deployed to the production branch. Advanced deploy mode helps to prevent accidental merges.

• If you choose to use release branches, Looks and dashboards can be exported using the Looker API (via Gazer or Looker Deployer), tagged with the release branch name, and stored in version control for backup and history.
• You can specify which branch should be used as the default merge target by updating the branch in the Git Production Branch Name field of your project's Project Settings. Alternatively, use the update_project endpoint.