Hi All,
Is there any way that we can find the duplicate events ingested into chronicle. If yes, could you please share more information.
With Regards,
Shaik Shaheer
@jstoner @mikewilusz @manthavish - Could you please help me identifying the duplicate logs into chronicle.
This should be possible by using a pivot table, on the basis that the log contains a unique identifier (Global Event ID, Event ID, Log Id etc). In the following case we are using Google Chronicle's demo instance, and utilizing the 'Crowdstrike Falcon' log source, with the UDM field that contains an event's unique identifier being "metadata.product_log_id".
[1] - First we search for the log type we want, in this case 'Crowdstrike Falcon' is the following: metadata.log_type = "CS_EDR"
[2] - Navigate to 'Pivot'
[3] - Apply Pivot settings like the screenshot below (grouping by the unique identifier}
[4] - Click on the :, export the data into a .csv, and remove all the ones that are equal to "1" (which if you order by Descending will be at the bottom) :).
This should show you the Event count based on the UDM field that is grouped (in this basis we are implying that metadata.product_log_id for the 'CS_EDR' logs is a unique identifier for each log). Depending on the need of this, it is likely that the creation of a dashboard may be better suited.
Hope this helps!
Hi Ayman C,
Greetings...!!!
Thank you for your suggestion, and we attempted to implement this method. However, it makes the analyst's job tedious as they have to manually export and individually check the logs. Is there an alternative automation process available?
With Regards,
Shaik Shaheer
Hi Shaik,
Google Chronicle SIEM customers can leverage several automation strategies to check for duplicate ingested data. Here's a breakdown:
1. Hash-Based Deduplication
Mechanism:
Pros:
Cons:
2. Similarity Detection with Chronicle Rules
Mechanism:
Pros:
Cons:
3. External Data Deduplication
Mechanism:
Pros:
Cons: