If so, can you share a high level explanation of how you “normalized” the logs? I had v2 audit logs flowing without much issue, but the optional fields are killing me.

We aren’t using the new API as it’s worthless without filtering capabilities b/c of the insane volume in the audit logs (more like debug logs tbh). We use a pipeline to filter out categories that aren’t security related, then export the dataset to Azure blob storage. Azure Data Factory (ADF) takes it from there and does some minor ETL before sending to a DCE via API.

The issue I’m running into is the “optional” fields in the audit logs that make it hard to define a schema. For example, one log will have the “origin” and “origins” fields, and another will just have the “origins” field. I can likely account for this w/ schema drift in ADF, but the transformation in the Sentinel DCR will fail at ingestion.

It’s incredibly frustrating and these are the worst audit logs I’ve ever had to work with. Any insight into how others are handling this is appreciated!