Question

Consolidating LookML Projects Guide

  • 21 August 2020
  • 0 replies
  • 188 views

You may find that once disparate LookML projects are now related, and therefore want to consolidate them into a single LookML project. Simply copying all files from one project to another can result in duplicate files and validation errors, as filenames and LookML objects could be the same across projects.


This guide will walk you through how to consolidate 2+ LookML projects into a new project. The end goal is to create a new project with explicitly defined include paths and no validation errors.


TL;DR:



  • Consider using project import if the main goal is to leverage existing LookML into another project

  • If consolidating more than 2 projects together, start with consolidating 2 projects into a new project. Repeat the consolidation tasks to remaining projects into the new project

  • Use folders to easily modify includes to ensure no validation errors. If folders are not being used in a project, use this tool to create default folders

  • Content does not reference the LookML project. Therefore, as long as the query parameters (Models, Explore, View, Fields) remain consistent across projects, the content will not be altered in any way

  • Coordinate with fellow developers to schedule a time in advance, be cognisant of any business impact / schedules that may be impacted during momentary downtime


1. Setup projects locally



  • Clone existing projects locally

  • Create a new empty git repository. This will be used as the new consolidated project


$ git clone git@github.com:org_name/project_a.git
$ git clone git@github.com:org_name/project_b.git
$ mkdir project_consolidation
$ cd project_consolidation/
$ git init
$ git remote add origin git@github.com:org_name/new_project.git

2. Consolidate files to the new project


Copy existing files into the new project; this can be done programmatically or manually.


Note - Projects can only contain 1 manifest file, and it must be located at the root level. If each existing project has a manifest file defined, the logic will need to be consolidated into a single file.


Choose a file structure for the new project that fits your requirements. As there can be filename and view name duplication across projects, the easiest solution to consolidate is to create subdirectories for each existing project. This will avoid any issues arising from duplicate filenames and including unnecessary files.


A simple example below of a folders for models, views, and dashboards:


**project_a:**

models/
- master_a.model.lkml
views/
- foo.view.lkml
- bar.view.lkml
dashboards/
- hello_world.dashboard.lookml

**project_b:**
models/
- master_b.model.lkml
views/
- foo.view.lkml
- bar.view.lkml
- baz.view.lkml

**new_project:**
models/
- master_a.model.lkml
- master_b.model.lkml
views/
project_a/
- foo.view.lkml
- bar.view.lkml
project_b/
- foo.view.lkml
- bar.view.lkml
- baz.view.lkml
dashboards/
project_a/
- hello_world.dashboard.lookml


Modify the include statement scope to the new folder structure. Absolute paths will be easier to manage than relative paths. This could be automated with regex, in any IDE to find/replace include statements, or even within the Looker IDE. To do this on the CLI, the following example will take the output from find on any lkml files and run the perl command to add /project_a to the include path:


find . -type f -name '*.lkml' -exec perl -pi -e 's/(include.*:.*")(?!\/\/)(.*)(\/.*\.model.*"|\/.*\.explore.*"|\/.*\.view.*"|\/.*\.dashboard.*"|\/.*\.lkml")(.*)/\1\2\/project_a\3\4/g' {} +

The Looker IDE also supports regular expression search:


**master_a.model.lkml**
include: "/views/**/*.view" -> include: "/views/project_a/**/*.view"
include: "/dashboards/*.dashboard" -> include: "/dashboards/project_a/*.dashboard"

3. Push changes to git:


Create a dev branch, add all the folders/files, and push to the new git repository:


$ git checkout -b consolidation
$ git add .
$ git commit -m "consolidating project a and b"
$ git push origin consolidation

4. Create new LookML project


In Looker, create a new LookML project (blank project). Configure git to connect to the new repository, set up the deploy key, and switch to the ‘consolidation’ branch.


You should see model configuration errors for duplicate model names. Rename the models in the original projects (project_a and project_b), e.g.: ‘master_a’ -> ‘master_a_old’


Note - If LookML Dashboards are being used, all model references will also need to be renamed.


We will now modify the model configurations for each project so that:



  1. The new project can use the original model names

  2. The original projects can use the renamed names,e.g. ‘master_a_old’


Go to ‘Manage LookML Projects’ to configure the models (further details on configuration errors can be found here).


Fix the errors in the original projects. It is not necessary to to modify the model configuration on the new project.


For “LookML model file doesn’t exist yet”:



  • Click “Configure” and change the project name to the new one and click “Save”

  • This will move the model configuration over to the new project


For “Configuration required for use”:



  • Click “Configure” and “Save”

  • This will update the model configuration on the old project to use the new model name


5. Validate all projects and content


All projects should now be functional. Confirm that include statements are scoped correctly, the content validator does not return any errors, and all projects have no validation errors. Once confirmed, commit and deploy all to production. After validating the new project, the original projects can be deleted.


Further cleaning can then be done to rearrange files; e.g. moving all view files into a single folder. This should be done after the initial consolidation process, once it is confirmed that everything works as expected.


0 replies

Be the first to reply!

Reply