Log Explorer count vs Dashboard Widget count don't match

[vdsouza]

I have a log-based metric based on the query filter and used that metric to create the dashboard.  However, for certain time buckets the log entry number that is shown on the log explorer does not match what is shown on the chart.  How come the same query results in two different record counts for some time buckets?

[lawrencenelson]

Hi @vdsouza,

Welcome to the Google Cloud Community!

Here are some possible reasons for the discrepancy based on the official documentation:

• New log entries might not match your metric's filter. A log-based metric gets data from matching log entries that are received after the metric is created. Logging doesn't backfill the metric from previous log entries.

• New log entries might not contain the correct field, or the data might not be in the correct format for extraction by your distribution metric. Check that your field names and regular expressions are correct.

• Your metric counts might be delayed. Even though countable log entries appear in the Logs Explorer, it may take up to 10 minutes to update the log-based metrics in Cloud Monitoring.

• The log entries that are displayed might be counted late or might not be counted at all, because they are time-stamped too far in the past or future. If a log entry is received by Cloud Logging more than 24 hours in the past or 10 minutes in the future, then the log entry won't be counted in the log-based metric.

  The number of late-arriving entries is recorded for each log in the system log-based metric logging.googleapis.com/logs_based_metrics_error_count.

  Example: A log entry matching a log-based metric arrives late. It has a timestamp of 2:30 PM on February 20, 2020 and a receivedTimestamp of 2:45 PM on February 21, 2020. This entry won't be counted in the log-based metric.

• The log-based metric was created after the arrival of log entries that the metric might count. Log-based metrics evaluate log entries as they're stored in log buckets; these metrics don't evaluate log entries stored in Logging.

I hope this answers your question. Thank you. 😃 

[vdsouza]

Thank you, @lawrencenelson for your prompt response.  Yes, I am aware of these reasons documented.  However, none of them are applicable here because of the following reasons:

1. Since I am literally right clicking on the metric and chose View logs for metric option there is no way to have filter mismatch
2. The metric was create a long time ago, so there is no question of needing to backfill of data
3. if you look at the screen shot, Chart is showing more numbers than the Log Explorer for the same query as mentioned in bullet #1
4. I am looking at the chart and logs after 12 hours they have been logged
5. the metric is not a distribution but the counter type to observe how many times status code 200 has been logged during a certain time window
6. Just to be clear this happens only for certain time windows and does not happen most of the time

I can understand if I used two different queries to validate the numbers but it is literally using the same query that is used by the metric which seems like a bug to me somewhere in the chart logic.

[Humandc]

✌️