How do we confirm that upgrading to AES-256 GCM encryption succeeded?

We’ve followed the instructions outlined here. The encryption migration process returned success, we were able to successfully start the Looker instance, so it looks like it worked, however the database appears to still be unencrypted. Looking through the logs during migration we see that a value `crypt_canary_gcm` was inserted into the `SETTINGS` table, but when we query the Looker DB, we don’t see that value present. 

Is there anything else we should be looking at to confirm that the  migration to GCM was successful? Is there anything else we should be doing to enable DB encryption?

0 6 1,385
6 REPLIES 6

Hi Jim, 

Greetings!

This is Mukul and I am a TSE at Google (Looker DCL support). Sorry for delay on getting back on this. 

Please feel free to let me know if the issue still persists and we can work on this to resolve this.

Best, Mukul

We received a response from Looker Support that I’m sharing here in case it’s helpful to anyone else.

I think the easiest way to tell, is when Looker is starting up it tells you which encryption scheme is being initialized, look for a line that says:

Encryption initialized: LEGACY[AES-128-CBC Legacy]
is what it would look like for legacy, and if it were GCM is would say that instead of legacy.

The setting table does have a value under the "crypt_canary" field but I'm not sure you could tell just by looking at it.

I would recommend checking out the "encryption_key" table, if that has entries it is GCM, if not then it is not.

We used an encrypted canary to validate that the correct CMK is in use. You can
use this canary to determine which Encryption mode is active.

For instances using Legacy Encryption, the Looker DB SETTING table will
contain a CRYPT_CANARY with a base64 value.

For instances using GCM Encruption, the SETTING table will contain a
CRYPT_CANARY_GCM with a base64 value.

So you can do:

SELECT key,value FROM setting WHERE key LIKE 'crypt_canary%';
For Legacy Encryption you'll see something like:

+--------------+------------------------------------------------------------------------+
| key | value |
+--------------+------------------------------------------------------------------------+
| crypt_canary |
"" |
+--------------+------------------------------------------------------------------------+
1 row in set (0.001 sec)
For GCM Encryption you'll see something like:

+------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| key | value |
+------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| crypt_canary_gcm |
""
|
+------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

I will add - there is no such thing as an unencrypted Looker database. Either the database is 128bit AES encrypted (legacy) or 256bit GCM encrypted. Looker will fail to start if the database were not encrypted.

We are encountering a problem with this. We believed we were on GCM already, but when trying to upgrade to 22.8.39 we hit this failure to start looker:

$ ./looker start

Starting Looker: Version 22.8.39-eb57ab..............................................

Error starting Looker. See log file for details.

Error starting Looker: Error starting Looker: This Looker instance must be migrated to GCM encryption using 'migrate_encryption'.
uri:classloader:/helltool/lib/helltool/utils/crypt.class:67:in `setup_with_mode'
uri:classloader:/helltool/lib/roles/common/crypt_runner.class:70:in `setup_crypt'
uri:classloader:/helltool/lib/roles/common/crypt_runner.class:43:in `start'
uri:classloader:/helltool/lib/roles/common/runner.class:49:in `pre_roles_start'
uri:classloader:/helltool/lib/runner.class:68:in `start'
uri:classloader:/helltool/lib/helltool/cli.class:85:in `start'
uri:classloader:/gems/thor-0.15.4/lib/thor/task.rb:27:in `run'
uri:classloader:/gems/thor-0.15.4/lib/thor/invocation.rb:120:in `invoke_task'
uri:classloader:/gems/thor-0.15.4/lib/thor.rb:275:in `dispatch'
uri:classloader:/gems/thor-0.15.4/lib/thor/base.rb:425:in `start'
uri:classloader:/helltool/bin/helltool:42:in `<main>'
org/jruby/RubyKernel.java:1009:in `load'
uri:classloader:/META-INF/main.rb:1:in `<main>'
org/jruby/RubyKernel.java:974:in `require'
uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/rubygems/core_ext/kernel_require.rb:83:in `require'
<script>:3:in `<main>'
uri:classloader:/helltool/lib/runner.class:119:in `start'
uri:classloader:/helltool/lib/helltool/cli.class:85:in `start'
uri:classloader:/gems/thor-0.15.4/lib/thor/task.rb:27:in `run'
uri:classloader:/gems/thor-0.15.4/lib/thor/invocation.rb:120:in `invoke_task'
uri:classloader:/gems/thor-0.15.4/lib/thor.rb:275:in `dispatch'
uri:classloader:/gems/thor-0.15.4/lib/thor/base.rb:425:in `start'
uri:classloader:/helltool/bin/helltool:42:in `<main>'
org/jruby/RubyKernel.java:1009:in `load'
uri:classloader:/META-INF/main.rb:1:in `<main>'
org/jruby/RubyKernel.java:974:in `require'
uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/rubygems/core_ext/kernel_require.rb:83:in `require'
<script>:3:in `<main>'

The encrypt_migration command as suggested in the instructions outlined here says we are already on GCM encryption:

$ java -jar looker.jar migrate_encryption


Error: RuntimeError: This Looker instance is already using GCM encryption
Encryption migration failed

and looking in the MySQL (RDS) back-end DB using the command suggested above:

mysql looke[looker@ip-172-31-9-163 ~]$ mysql looker_dev
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 30103
Server version: 5.7.33-log Source distribution

Copyright (c) 2000, 2021, Oracle and/or its affiliates.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> SELECT `key`,`value` FROM `setting` WHERE `key` LIKE 'crypt_canary%';
+--------------+------------------------------------------------------------------------+
| key | value |
+--------------+------------------------------------------------------------------------+
| crypt_canary | "<REMOVED BY ME FOR POSTING HERE>" |
+--------------+------------------------------------------------------------------------+
1 row in set (0.00 sec)

We are not using KMS, we just used the approach from the instructions like this:

openssl rand -base64 32

export LKR_MASTER_KEY_ENV=<CMK_value>

So,  we’re in a situation where the migrate_encryption says were using GCM, but the database doesn’t show a crypt_canary_gcm key, and looker won’t start (we have rolled back to 22.6.59 in the meantime, and that at least starts).

We are having self hosted looker running on ECS and using external mysql rds database. 

When we try to run the migration , facing the below error’

[Encrypting internal database]++++++++++++++++++++++

Startup failed. Unable to load java dependencies.

You are attempting to start Looker with an internal Looker database that does NOT match this

Looker version (22.6.59). Starting Looker with a mismatched database is not supported

Can someone help me here to find the real issue here

Hi @Deepak_Kumar2 did you manage to resolve this? We’re facing the same issue.

Hello @Deepak_Kumar2 were you able to solve the GCM encryption issue?