Chronicle SIEM alerts detection delay.

[AThebrand]

Hi everyone, 

Yesterday I noticed that there could be a problem with our SIEM. I'll give you an example:

In curated detections, there is a rule to catch the command 

cmd  /c "echo hello_chronicle_world!"

I've activated this rule to make some tests and noticed something weird. 

As you can see in the picture above there are four different timestamps:

metadata.event_timestamp THIS SHOULD BE THE TIMESTAMP AT WHICH THE EVENT IT'S BEEN GENERATED ON THE HOST

metadata.ingested_timestamp THIS SHOULD BE THE TIMESTAMP AT WHICH THE EVENT IT'S BEEN INGESTED IN CHRONICLE

Detected at THIS SHOULD BE THE TIMESTAMP AT WHICH THE DETECTION RULE DETECTED THE EVENT

Created FINALLY THIS SHOULD BE THE TIMESTAMP AT WHICH THE ALERT IT'S BEEN CREATED

 If I'm right, why is there so much delay? The time between event creation and ingestion is minimal but there is 1 hour delay between ingestion and detection and then another hour from detection to alert generation.

I don't know if it could be related, the log type Is WINDOWS_SYSMON and all the logs of this type reach chronicle through chronicle forwarder for Linux.

I want also to specify that to be sure that the problem was not related to curated detections I've created a custom rule that matches the same things. The result is the same.

Thank you in advance.

A

[Chris_B]

In my analysis of instances of delayed ingestion | parsing | detection | alerting  I've also seen some delays.

Some delays are simple like e.g. if your detection logic ran every hour and the logs were ingested right after it ran. In this simple example the detection would occur almost an hour after ingest. But this is not your example - I think you ran your detection logic right after you created the event ... maybe repeatedly until you got your result (alert) .

I've seen some delays in Chronic ingest/parsing compared to another SIEM were running on parallel... in some instances I've observed the Chonic logs are delayed... by 8 minutes I've seen in one example. We're working with our Google team to find out why.

[Chris_B]

Update: we found more info about log delays related to an EDR. The EDR is clooud based and its logs are pushed to a cloud bucket. The bucket is then polled per a cadence and the hop to a bucket then to  SIEM ingest  together with the cadence time add to log delays when compared to another SIEM we used.