Troubleshooting Common Chromium Errors

The Chromium renderer is the process on the Looker server responsible for creating PDFs and PNGs of dashboards and Looks. Errors from Chromium or image render jobs in general can tend to be a bit generic. This page provides an overview of the most common Chromium image rendering errors and how to troubleshoot them. 

These errors most commonly occur with customer hosted Instances, as Chromium is automatically installed and updated on Looker hosted instances.  
 

First, Make sure Chromium is installed
 

The first step to troubleshooting any image rendering error is to confirm that Chromium is installed correctly, and that Looker can access it by running the following commands:

chromium --version

This should return something like:

Chromium 70.0.3538.77 Built on Ubuntu , running on Ubuntu 16.04

Next, make sure Chromium is in the proper place, which is /usr/bin/chromium: 

which chromium

The output should be:

/usr/bin/chromium

Running these two commands confirms that Chromium is installed correctly on a supported version (versions > 65, excluding version 71), and that it is in the proper path for Looker to find. Also make sure Looker has permissions to execute chromium.

RendererNoRenderFinishedEventReceivedError

This is a common catch-all error that often appears on scheduled jobs and is associated with the other common errors mentioned below. It is thrown by the Chromium renderer, which - as mentioned above - is the process on the Looker server responsible for creating PDFs and PNGs of dashboards and Looks.

It is essentially a  timeout error - the renderer was waiting for something to finish, such as a SQL query or visualization, but never received confirmation that the task was completed.

Common causes

  1. Query cache has expired. Normally, queries are run right before the renderer process so that the results are cached and ready for the renderer to use. However, if the cache expires before the renderer is able to use those results, then you may see this error. This commonly happens when a dashboard has generated a lot of queries or long-running queries. We have seen this resolved by increasing the cache duration on the underlying model or models.
     

  2. A custom visualization is missing its completion response. The renderer listens for visualizations to fire a “finished” event when they’re done drawing and ready to be captured. The renderer also polls for a count of visualization elements that are in a “loading” state and will accept the state where that count is 0 as a valid replacement for the finished event. The generic RendererNoRenderFinishedEvenReceivedError error is raised when neither of those conditions happen.

    If a custom visualization's code doesn't properly signal its completion, then the renderer may be left waiting. To check if this is the cause, load the dashboard in the browser and check the Javascript console (command+option+j on Chrome) for errors.
     

  3. Chromium is not properly installed. On Looker-hosted instances, Chromium is installed by default, but this error has appeared for many reasons on customer-hosted instances. Make sure you've followed the Looker-provided Chromium installation instructions closely.

    1. On Ubuntu, we have seen this resolved by running sudo apt-get install fonts-freefont-otf

    2. For CentOS, we have seen this resolved by running sudo yum -y install liberation-*

 

Render error: Script timed out

When Looker starts Chromium to render a PDF or PNG, it runs rendering-specific JavaScript to check for when all of the content (tiles on a dashboard, or a query for a Look) finishes loading before taking a screenshot of the content. 

After the Chromium process starts running, it has 30 minutes to finish rendering before it will timeout with this error. Since all queries associated with a render job are run and cached before the Chromium process is started, the rendering should complete within the time limit. However, the  Render error: Script timed out error in the UI indicates when the rendering process times out.

If the error message includes the word "retried", this means the renderer (Chromium) tried to execute twice.

Common Causes

  1. Query cache has expired. Normally, queries are run before the renderer process so that the results are cached and ready for the renderer to use. However, if the cache expires before the renderer is able to use those results, then you may see this error. This commonly happens when a dashboard has generated a lot of queries or long-running queries. This can potentially be resolved by increasing the cache duration on the underlying model or models.
     

  2. A custom visualization is missing its completion response. The renderer listens for visualizations to fire a “finished” event when they’re done drawing and ready to be captured. The renderer also polls for a count of visualization elements that are in “loading” state and will accept the state where that count is 0 as a valid replacement for the finished event. The generic RendererNoRenderFinishedEvenReceivedError error is raised when neither of those conditions are met.

    If a custom visualization's code doesn't properly signal its completion, then the renderer may be left waiting. To check if this is the cause, load the dashboard in the browser and check the Javascript console (command+option+j on Chrome) for errors.

  1. Unexpected behavior with certain tiles. There are some cases where certain tiles never send the "done" message, even if they have successfully completed. Open a support request with the Department of Customer Love for assistance with confirming whether you are experiencing this unexpected behavior.
     

Could not start Chromium

 

This error most commonly appears on customer hosted Looker instances. 

Common causes 

  1. Chromium fonts are not installed. To confirm that this is the case,  the command chromium --headless --disable-gpu --remote-debugging-port=9222  will return a  core dumped error. Visit the Customer Hosted Installation of Rendering Software documentation page for font installation instructions.

  2. There is not enough memory or swap space. Confirm that enough memory or swap space has been allocated to the Chromium renderer. 
     

Could not connect to dev tools
 

This error most commonly appears on customer hosted Looker instances. 

This usually means that either Chromium was started and immediately failed, or Chromium started but then could not reach the dev tools in order to control the steps necessary to render content. 

Common causes 

  1. Looker is running as the root user. Looker needs to run as a non-root user. When a Chromium process begins, it runs as the same Looker user. Chromium requires being run from a normal account (non-root) to support its security sandbox, and Looker does not have code in place to drop privilege when starting Chromium. To make sure Looker is running as a non-root user:

    Running the the following remote debugging port command below:
    `chromium --headless --remote-debugging-port=9222`

    If the following output is returned, Looker is running as the root user:

    [0308/160359.743951:ERROR:zygote_host_impl_linux.cc(89)] Running as root without --no-sandbox is not supported’

If the message below is returned instead, it is more likely that fonts were not installed:  

[0100/000000.982492:ERROR:zygote_linux.cc(621)] Zygote could not fork: process_type  renderer numfds 5 child_pid -1

Visit the Customer Hosted Installation of Rendering Software documentation page for font installation instructions.

5 18 11.1K
18 REPLIES 18

Hi,

All our scheduled reports/dashboards where a PDF/PNG format is being sent through emails are failing with the same error. We are also unable to download any report in PDF/PNG format individually. This started happening only recently.

Error: [RendererNoRenderFinishedEventReceivedError - retried])

Currently, we are using the latest version of both Looker (23.0.40) and Chromium (110.0.5481.77 ). However, we are still facing this error despite following all steps for troubleshooting this type of error. Can someone please help with this?

We’ve experienced exactly the same error overnight.

The fix for us was to roll chromium back to 109.0.5414.74 (we noticed an automatic upgrade happened yesterday).

Not clear whether this is a bug in chromium or a change in chromium that needs a fix in Looker but hopefully that will become clear soon.

@sihil - THANK YOU so much! This worked for us as well. 

I have the same problem as @sihil and @shivam-1652727242 . Unfortunately there is no any previous version package for ubuntu 18.04 lts bionic - so how can we fix it? Or do you have an idea how to install previous package on that OS?

We’ve had this problem on Ubuntu 20.04, the official apt repo only kept the latest version of the binary so we pulled it using wget and installed. Note this is for google-chrome-stable which works but isn’t explicitly recommended in the docs above:

wget http://dl.google.com/linux/chrome/deb/pool/main/g/google-chrome-stable/google-chrome-stable_109.0.5414.119-1_amd64.deb
dpkg -i google-chrome-stable_109.0.5414.119-1_amd64.deb

Hello - for those on Ubuntu 18, were you able to roll back? if so, how? The older versions like 109>* are no longer available. 

ubuntu$ apt-cache policy chromium-browser
chromium-browser:
Installed: 110.0.5481.100-0ubuntu0.18.04.1
Candidate: 110.0.5481.100-0ubuntu0.18.04.1
Version table:
*** 110.0.5481.100-0ubuntu0.18.04.1 500
500 http://us-east-1.ec2.archive.ubuntu.com/ubuntu bionic-updates/universe amd64 Packages
500 http://security.ubuntu.com/ubuntu bionic-security/universe amd64 Packages
100 /var/lib/dpkg/status
65.0.3325.181-0ubuntu1 500
500 http://us-east-1.ec2.archive.ubuntu.com/ubuntu bionic/universe amd64 Packages

Hello @sihil  - can you please share the steps for rolling back the upgrade? 

For anyone who is still struggling with it, below are the solution that fixed our Looker.

I had to use a private repo to install an older version of chromium-browser and it worked.

sudo apt remove chromium-browser
sudo add-apt-repository ppa:saiarcot895/chromium-beta
sudo apt update
sudo apt install chromium-browser=101.0.4951.15-0ubuntu1~ppa1~18.04.1
sudo apt-get install chromium-browser

chromium --version
Chromium 101.0.4951.15 Ubuntu 18.04

Hopefully this help!

@alin2 thank you so much! it worked! 

For anyone who experienced the same issue, we made these additional changes temporarily as we develop a better method for testing upgrades prior to releasing into production.

  1. We disabled unattended-upgrades using sudo systemctl stop unattended-upgrades
  2. In case the unattended-upgrades was restarted by accident, we set the two upgrade variables to 0 in this file sudo vi /etc/apt/apt.conf.d/20auto-upgrades

Hi all,

Sorry - should have left more instructions (and would advise against using an unknown PPA).

I figured out the version we’d had previously from /var/log/dpkg.log. I then googled the version and landed on the launchpad page (in our case https://bugs.launchpad.net/ubuntu/+source/chromium-browser/109.0.5414.74-0ubuntu0.18.04.14). As people have noted, the published list is no longer there but the builds are - click on the link under the Builds section for your architecture and you’ll see the built files. Then it’s a download and dpkg -i chromium*. Worth doing a dpkg -l | grep chromium to understand which packages you need to grab.

In addition I put a hold on the packages using sudo apt-mark hold chromium-browser chromium-browser-l10n chromium-codecs-ffmpeg-extra to prevent those being automatically updated.

Good luck. If anyone comes across a more permanent solution please post it here 🙂

Im still getting the error: “2023-02-23 10:36:08.555 +0000 [WARN|021ec|com.looker.render.RenderService] :: Chrome process init failure; may retry. Error: Couldn't start Chromium: Could not connect to devtools after 30538 ms.” -  do you guys know why?

I think you’re now getting a different error to us @Saltero. Looks like it’s failing to start Chrome at all - which is addressed by the last section of the main post above. If you rolled back, did you roll back all of the related chromium-browser packages?

Thank you @sihil I went back and successfully tested the following

 

  1. wget https://bugs.launchpad.net/~canonical-chromium-builds/+archive/ubuntu/stage/+build/25482450/+files/chromium-codecs-ffmpeg_109.0.5414.74-0ubuntu0.18.04.14_amd64.deb
  2. wget https://bugs.launchpad.net/~canonical-chromium-builds/+archive/ubuntu/stage/+build/25482450/+files/chromium-codecs-ffmpeg-extra_109.0.5414.74-0ubuntu0.18.04.14_amd64.deb
  3. wget https://bugs.launchpad.net/~canonical-chromium-builds/+archive/ubuntu/stage/+build/25482450/+files/chromium-browser_109.0.5414.74-0ubuntu0.18.04.14_amd64.deb
  4. sudo dpkg -i chromium-codecs-ffmpeg-extra_109.0.5414.74-0ubuntu0.18.04.14_amd64.deb
  5. sudo dpkg -i chromium-codecs-ffmpeg_109.0.5414.74-0ubuntu0.18.04.14_amd64.deb
  6. sudo dpkg -i chromium-browser_109.0.5414.74-0ubuntu0.18.04.14_amd64.deb

We’ve experienced exactly the same error overnight.

The fix for us was to roll chromium back to 109.0.5414.74 (we noticed an automatic upgrade happened yesterday).

Not clear whether this is a bug in chromium or a change in chromium that needs a fix in Looker but hopefully that will become clear soon.

Could you please provide the command to downgrade chromium like you did?

@sihil could you please let me know how did you downgrade the chromium (commands) in ubuntu?

Thank you @sihil and @mk-87  for the insights and the commands. Putting things together from your answers:

1. Downgrade the chromium and its packages

wget https://bugs.launchpad.net/~canonical-chromium-builds/+archive/ubuntu/stage/+build/25482450/+files/chromium-codecs-ffmpeg_109.0.5414.74-0ubuntu0.18.04.14_amd64.deb

wget https://bugs.launchpad.net/~canonical-chromium-builds/+archive/ubuntu/stage/+build/25482450/+files/chromium-codecs-ffmpeg-extra_109.0.5414.74-0ubuntu0.18.04.14_amd64.deb

wget https://bugs.launchpad.net/~canonical-chromium-builds/+archive/ubuntu/stage/+build/25482450/+files/chromium-browser_109.0.5414.74-0ubuntu0.18.04.14_amd64.deb

sudo dpkg -i chromium-codecs-ffmpeg-extra_109.0.5414.74-0ubuntu0.18.04.14_amd64.deb

sudo dpkg -i chromium-codecs-ffmpeg_109.0.5414.74-0ubuntu0.18.04.14_amd64.deb

sudo dpkg -i chromium-browser_109.0.5414.74-0ubuntu0.18.04.14_amd64.deb

2. Put on Hold so the packages wouldn’t update automatically

sudo apt-mark hold chromium-browser chromium-browser-l10n chromium-codecs-ffmpeg-extra

PS: I didn’t restart the looker, I did everything on the fly while looker is running and the changes were propagated smoothly. 

@sihil @mk-87 @jananath  thanks so much for these .. it works!

Does anyone fixed the same issue on linux server? if so could you please let me know the steps?