You may find that once disparate LookML projects are now related, and therefore want to consolidate them into a single LookML project. Simply copying all files from one project to another can result in duplicate files and validation errors, as filenames and LookML objects could be the same across projects.
This guide will walk you through how to consolidate 2+ LookML projects into a new project. The end goal is to create a new project with explicitly defined include paths and no validation errors.
TL;DR:
$ git clone git@github.com:org_name/project_a.git
$ git clone git@github.com:org_name/project_b.git
$ mkdir project_consolidation
$ cd project_consolidation/
$ git init
$ git remote add origin git@github.com:org_name/new_project.git
Copy existing files into the new project; this can be done programmatically or manually.
Note - Projects can only contain 1 manifest file, and it must be located at the root level. If each existing project has a manifest file defined, the logic will need to be consolidated into a single file.
Choose a file structure for the new project that fits your requirements. As there can be filename and view name duplication across projects, the easiest solution to consolidate is to create subdirectories for each existing project. This will avoid any issues arising from duplicate filenames and including unnecessary files.
A simple example below of a folders for models, views, and dashboards:
**project_a:**
models/
- master_a.model.lkml
views/
- foo.view.lkml
- bar.view.lkml
dashboards/
- hello_world.dashboard.lookml
**project_b:**
models/
- master_b.model.lkml
views/
- foo.view.lkml
- bar.view.lkml
- baz.view.lkml
**new_project:**
models/
- master_a.model.lkml
- master_b.model.lkml
views/
project_a/
- foo.view.lkml
- bar.view.lkml
project_b/
- foo.view.lkml
- bar.view.lkml
- baz.view.lkml
dashboards/
project_a/
- hello_world.dashboard.lookml
Modify the include statement scope to the new folder structure. Absolute paths will be easier to manage than relative paths. This could be automated with regex, in any IDE to find/replace include statements, or even within the Looker IDE. To do this on the CLI, the following example will take the output from find
on any lkml files and run the perl
command to add /project_a
to the include path:
find . -type f -name '*.lkml' -exec perl -pi -e 's/(include.*:.*")(?!\/\/)(.*)(\/.*\.model.*"|\/.*\.explore.*"|\/.*\.view.*"|\/.*\.dashboard.*"|\/.*\.lkml")(.*)/\1\2\/project_a\3\4/g' {} +
The Looker IDE also supports regular expression search:
**master_a.model.lkml**
include: "/views/**/*.view" -> include: "/views/project_a/**/*.view"
include: "/dashboards/*.dashboard" -> include: "/dashboards/project_a/*.dashboard"
Create a dev branch, add all the folders/files, and push to the new git repository:
$ git checkout -b consolidation
$ git add .
$ git commit -m "consolidating project a and b"
$ git push origin consolidation
In Looker, create a new LookML project (blank project). Configure git to connect to the new repository, set up the deploy key, and switch to the ‘consolidation’ branch.
You should see model configuration errors for duplicate model names. Rename the models in the original projects (project_a and project_b), e.g.: ‘master_a’ -> ‘master_a_old’
Note - If LookML Dashboards are being used, all model references will also need to be renamed.
We will now modify the model configurations for each project so that:
Go to ‘Manage LookML Projects’ to configure the models (further details on configuration errors can be found here).
Fix the errors in the original projects. It is not necessary to to modify the model configuration on the new project.
For “LookML model file doesn’t exist yet”:
For “Configuration required for use”:
All projects should now be functional. Confirm that include statements are scoped correctly, the content validator does not return any errors, and all projects have no validation errors. Once confirmed, commit and deploy all to production. After validating the new project, the original projects can be deleted.
Further cleaning can then be done to rearrange files; e.g. moving all view files into a single folder. This should be done after the initial consolidation process, once it is confirmed that everything works as expected.
For “LookML model file doesn’t exist yet”
This does not work. I have a project with a lookml model file that works in dev mode. When I switch to production, it returns this message however the “Configure” settings all look correct. I cannot find documentation on how to fix this