Cloud Run routes traffic to container before HTTP startup probe succeeds

[ContinueAsGuest]

Hi,

We have a JavaScript application in Cloud Run, with HTTP startup/readiness checks enabled.  We're seeing Cloud Run routes traffic to containers before the startup probe succeeds.

Looking at an individual container, the first request comes in at 18:32:01 and eventually times out.  According to the logs, the startup health check succeeds 9 seconds after, at 18:32:00.

The Cloud Run Health Check documentation says:
"Startup probes determine whether the container has started and is ready to accept traffic."
So we're wondering why we see traffic coming into containers before the startup check succeeds.

Here are the two log entries showing the sequence (I've removed sensitive information from the logs, so they're missing some fields):

[
    {
    "textPayload": "The request has been terminated because it has reached the maximum request timeout. To change this limit, see https://cloud.google.com/run/docs/configuring/request-timeout",

    "httpRequest": {
      "requestMethod": "GET",
      "requestUrl": "https://foo/bar",
      "status": 504,
      "latency": "9.998326453s",
      "protocol": "HTTP/1.1"
    },
    "resource": {
      "type": "cloud_run_revision",
      "labels": {
        "service_name": "webapp",
        "configuration_name": "webapp",
        "revision_name": "webapp-123456"
      }
    },
    "timestamp": "2024-05-07T18:31:51.166461Z",
    "severity": "ERROR",
    "labels": {
      "instanceId": "00f46b92853c27bd9df5f9f8a8a080371a7f6a79326a8420449ef4a2cdda8b9e2b97740a7ce77f4f0ceb9e550476358c91668b4b67d6ee232ea6b109b519d8c3"
    },

    "receiveTimestamp": "2024-05-07T18:32:01.377377472Z",
    "spanId": "13942351057703038538",
    "traceSampled": true
  },

  {
    "textPayload": "STARTUP HTTP probe succeeded after 1 attempt for container \"webapp-1\" on path \"/api/startup\".",

    "resource": {
      "type": "cloud_run_revision",
      "labels": {
        "service_name": "webapp",
        "configuration_name": "webapp",
        "revision_name": "webapp-123456"
      }
    },
    "timestamp": "2024-05-07T18:32:00.096356Z",
    "severity": "INFO",
    "labels": {
      "instanceId": "00f46b92853c27bd9df5f9f8a8a080371a7f6a79326a8420449ef4a2cdda8b9e2b97740a7ce77f4f0ceb9e550476358c91668b4b67d6ee232ea6b109b519d8c3"
    },
    "receiveTimestamp": "2024-05-07T18:32:00.104944032Z"
  }
]