Converting data from json log into Chronicle UDM events

[NastyaS]

Hello everyone!
I am creating a custom parser for json logs and I need to convert domains_list into principal.user.attribute.labels udm event.

"activity_metadata": {
    "settings_new_value": {
      "block_type": "whitelist",
      "domains_list": [
        "test.com",
        "attack.mitre.org"
      ]
    }, 

I converted block_type already, using the following code

#Get block_type

  mutate {

    replace => {

      "block_type.value" => "%{activity_metadata.settings_new_value.block_type}"

    }

    on_error => "block_type_not_set"

  }

  if ![block_type_not_set] {

    mutate {

      replace => {

        "block_type.key" => "block_type"

      }

    }

    mutate {

      merge => {

        "token_principal.user.attribute.labels" => "block_type"

      }

    }

  }

However, I can't catch [ "test.com", "attack.mitre.org" ] from domains_list , as it the statedump gives an error

"domains_list_not_set": true,

and doesn't parse the data 

Here is the code I was using: 

#Get domains_list

  mutate {

    replace => {

      "domains_list.value" => "%{activity_metadata.settings_new_value.block_type.domains_list}"

    }

    on_error => "domains_list_not_set"

  }

  if ![domains_list_not_set] {

    mutate {

      replace => {

        "domains_list.key" => "domains_list"

      }

    }

    mutate {

      merge => {

        "token_principal.user.attribute.labels" => "domains_list"

      }

    }

  }

Can you please help me to convert this piece of data inside [] into udm?

 Do I need to use regex to do that?

Thank you in advance!

[ion_ · Accepted solution]

I'm not sure if you can write an array directly like that, but I do know you can iterate over arrays. I've solved something similar and modified the code to suit your example. I've left the statedump in so you can make sure each iteration is working correctly

for index, _domain in activity_metadata.settings_new_value.domains_list {

statedump {label=>"asd"}

mutate {

replace => {

"var_domain_label.key" => "domain_list"

"var_domain_label.value" => "%{_domain}"

}

}

mutate {

merge => {

"target.resource.attribute.labels" => "var_domain_label"

}

}

mutate {

replace => {

"var_domain_label" => ""

}

}

}

Notes:

1). You do need to clear the variable on each iteration otherwise you'll get weird referencing issues

2). I haven't encountered issues with multiple labels with the same name, but more testing might prove the opposite

Lemme know if that makes sense, you'll have to just copy paste your variables names over if you're happy

[NastyaS]

This code also worked perfectly for me, thank you!🙂

In my case, I just used different labels 

#Get domains_list

    for index, _domain in activity_metadata.settings_new_value.domains_list {

    statedump {label=>"asd"}

    mutate {

    replace => {

    "domains_list.key" => "domains_list"

    "domains_list.value" => "%{_domain}"

    }

    }

    mutate {

    merge => {

    "token_principal.user.attribute.labels" => "domains_list"

    }

    }

    mutate {

    replace => {

    "domains_list" => ""

    }

    }

    }

  mutate {

    replace => {

      "domains_list.value" => "%{activity_metadata.settings_new_value.block_type.domains_list}"

    }

    on_error => "domains_list_not_set"

  }

  if ![domains_list_not_set] {

    mutate {

      replace => {

        "domains_list.key" => "domains_list"

      }

    }

    mutate {

      merge => {

        "token_principal.user.attribute.labels" => "domains_list"

      }

    }

  }

and having this output:

principal.user.attribute.labels[5].key"domains_list"
principal.user.attribute.labels[5].value"test.com"
principal.user.attribute.labels[6].key"domains_list"
principal.user.attribute.labels[6].value"attack.mitre.org"

[Mustache]

Ah, the infamous nested arrays within logstash/chronicle implementations.

Nicely done @ion_ . I can confirm that works in labels: 

target.resource.attribute.labels[0].key"domain_list"  
target.resource.attribute.labels[0].value"test.com"  
target.resource.attribute.labels[1].key"domain_list"  
target.resource.attribute.labels[1].value"attack.mitre.org"

This is the output from the custom parser preview.