I’m looking for recommendations on CTF-style threat hunting datasets that integrate well with Microsoft Sentinel. I recently finished a massive investigative threat hunt project using the Splunk BOTSv3 dataset and absolutely loved it. Even though I only uncovered about 60% of the adversary's full execution tree, the sheer scope, deep technical challenge, and open-ended nature of the hunt made it an incredibly rewarding project.

I published my investigative logs and Splunk detection playbooks from that project to my GitHub, put it on my resume, got a Splunk cert, and now I want to do the same exact thing, but with Sentinel. My initial plan was to use BOTSv2, but I've recently discovered the amount of work it would require to get the Splunk logs normalized to the KQL schema, so I'm looking for a backup option.

This upcoming project is designed to serve three distinct goals:

1. Portfolio & Resume Evidence: Documenting the end-to-end hunt, ingestion engineering, and playbook creation.
2. SC-200 Prep: Gaining proficiency with KQL syntax to prepare for the SC-200 exam.
3. Methodology Refinement: Sharpening vendor-agnostic threat hunting and detection engineering methodologies that easily transfer across SIEM platforms.

What I am specifically looking for in a dataset:

• Open-Ended/Full Scope: I want to avoid datasets that are hand-holding or strictly oriented around a single, pre-mapped MITRE ATT&CK technique with no deviations. I want a true "needle in a haystack" investigative challenge. Ideally I'd like a full scoped attack starting from the reconnaissance/initial access phases and ending with exfiltration.
• Realistic White Noise: It needs to contain benign baseline background traffic so I encounter realistic false positives, forcing me to actively tune my KQL detections just like in a real world environment.
• Data Cap Friendly: Because this is for a cloud home lab, I would like to respect a 10GB daily data ingestion limit to keep my Azure workspace under the free trial allocation. I am open to drip-feeding a larger dataset across multiple days or spending a small amount of money, but ingesting a full 300gb dataset like BOTSv2 isn't an option.

Every Sentinel dataset I’ve stumbled across so far seems incredibly limited in scope or feels too "on rails" (e.g., executing one isolated script and immediately querying the single resulting alert).

Does anyone have recommendations for datasets that fit this open-ended criteria while respecting the 10GB daily ingestion cap? Are there any viable options outside of Mordor? Because of how modular it is, I'm concerned it'll lack the broader, interconnected scope I'm looking for.

Are Microsoft still limiting when Data Lakes can be created? It seems incredible that this very useful service can only be setup depending on Microsoft.

Hi everyone, I'm trying to set up Sentinel in a homelab for Soc Learning but I've hit a weird issue in the Defender Portal and I can't figure out if its a bug, or misconfiguration.

​

Setup:

Azure subscription : Free Trial

Tenant : Azure personal account

Sentinel enabled on Log Analytics workspace

Workspace shows as Connected+ Primary in Defender

I have Owner permissions and also added Sentinel Contributor Permissions in Resource Group

​

Problem - In the Defender Portal, everything under Microsoft Sentinel dropdown is broken and whenever I click on any of the dropdowns like ContentHub, it loops me back to the workspace selection page. I've already waited more than 24 hours and tried incognito/different browsers.

​

Any solution would be greatly appreciated.

So we have successfully onboarded our Sentinel to Defender.
It states its connected but every click in the portal like "search", "tables", "Data connectors" etc leads us to "Settings > Microsoft Sentinel" and the "SIEM workspaces".

Its been over 24h, from what i can tell all XDR is configured correctly, even tried a admin role that grants it all along with all workspaces etc. Global admin role is being used, owner on the subscription where Sentinel resides but still the defender integration does not seem to work.

From the Azure sentinel perspective XDR seems onboarded, UEBA configured etc.. Any advice?

Hey everyone,

I’ve been deep in the data infrastructure and security space for a while now, and I keep hitting a fundamental wall with how we handle incident response and triage. I wanted to put this architectural concept out here to see how you all are solving this, or if you can poke some holes in a design I’m working on.

The Problem: SIEMs think in "Rows," not "Graphs"

Every major SIEM on the market (Splunk, Sentinel, Elastic, etc.) treats machine data as discrete, point-in-time entries. When a security alert fires saying a user account read a highly sensitive file, that log is stateless.

If that user then moves that data across platforms, or worse—performs a text state-transformation (like copying the text content over an RDP session clipboard to their local laptop and saving it as a text file), the chain breaks.

To map out the actual blast radius during a triage, a Tier 2/3 analyst has to spend anywhere from 45 minutes to 3 hours running manual, exhausting KQL/SPL pivots across EDR schemas, Active Directory lookup tables, and cloud provider logs. We are trying to track a dynamic relationship using a flat spreadsheet model, leading to massive MTTR delays.

The Proposed Concept: An In-Memory Taint-Graph Middleware

Instead of trying to force a SIEM database to run heavy, expensive, quadratic ($O(N^2)$) pairwise comparisons on a live log stream, what if we treat data drift as a stateful graph using an ephemeral Taint-Inheritance pipeline?

The rough blueprint looks like this:

1. The Ingestion Layer: A lightweight, containerized microservice sits next to the SIEM log forwarders, consuming standard OCSF (Open Cybersecurity Schema Framework) streams (specifically Category 2: File Activity, Class 2001).
2. Canonical Identity Swapping: Using the SIEM’s native asset/identity lookup tables, it instantly resolves fragmented usernames (e.g., DOMAIN\jdoe, [jdoe@company.com](mailto:jdoe@company.com), and local endpoint SIDs) into a single unique Actor node in memory.
3. Temporal Sliding Windows & Taint Propagation:
   • When an event shows a sensitive file is read, that specific Actor node is marked as "Tainted" in an in-memory cache with a 5-minute sliding TTL.
   • If that same Actor node triggers a local file write or an outbound network connection within that 5-minute window, the destination node automatically inherits the data lineage token.
   • This bridges the air-gap. Even if text is copied via an RDP clipboard handshake, the engine uses the network session metadata to pass the taint token from the remote VM to the host endpoint file write.

The output isn't another dashboard screaming new alerts. It’s a pure Forensic Storyteller UI—an interactive, chronologically stitched visual movie of the data's journey that pops up via a deep link inside your existing SIEM alerts.

My questions for the Blue Team / Architecture folks here:

1. The Telemetry Gap: When you are triaging a data leak incident today, where do your lineage chains usually fracture? How are you currently proving that a file created on an endpoint contains the data read from a cloud bucket 5 minutes prior?
2. The RDP/Clipboard Problem: Has anyone successfully mapped remote desktop clipboard or memory-drop telemetry inside Sentinel or Splunk without triggering an absolute avalanche of false-positive noise?
3. UX Preference: If you were using an investigation tool like this, would you prefer to see this path represented as a left-to-right topological node graph, or an interactive vertical swimlane timeline split by environment (On-Prem, Endpoint, Cloud)?

Keen to hear your thoughts, constraints, or if you think this is a solved problem via some tool configuration I’m missing.

Hi all,

I’m looking for guidance from anyone who has dealt with monitoring Copilot Studio / Power Automate connector usage, specifically around the account used to configure or authenticate connectors.

Scenario:

We are evaluating a Copilot Studio DEV environment where makers can use built-in Power Platform connectors. Some connectors, such as SharePoint, Dataverse, SQL, HTTP/custom connectors, etc., could potentially access production data depending on the account used during connector setup.

What we would like to determine:

Is there any audit log that shows when a Power Platform connection/connector is created or modified?

Can the logs show the account used to authenticate the connector, not just the user who created the flow?

Can this be monitored in Microsoft Sentinel, for example through OfficeActivity, CloudAppEvents, PowerAutomate or another table?

Hi
We setup a new sentinel instance and connected entra id, everything looks good, diagnostic settings are created and such but no logs stream to the workspace.

tried recreating the diag. setting but its still not streaming.

There is no limit right on how many workspaces you can stream to right? We have another test sentinel streaming logs but its another workspace and both should be able to get it right?

Hi all,

I've setup an on prem Linux server, with rsyslog, that will just be used to forward syslog events from our firewall. I have it onboarded to Azure Arc and have Sentinel can receive the logs.

I'm just not clear on disk space usage. The events will be sent to Sentinel, but I'm not clear if I still have to manage the on prem disk space using something like log rotate.

Though I am looking at something like Cribl after we do our network refresh

Hi All,

I rarely work with linux so please forgive me if this is a stupid question.

I have a server that is successfully onboarded to arc and sentinel.

The server logs are currently being ingested to Sentinel without any issues. However, the server has apache running and I want to ingest the access logs as well.

I have configured the custom logs connector and the appropriate DCR but I am not able to ingest the apache access logs to Sentinel.

I get the following errors in the mdsd logs:

amacoreagent[xxxxxx]: The required instruction sets are not supported by the current CPU.

Failed to connect port 13005 socketId: Data: 130 to AMACoreAgent: Connection refused.

The AMA agent supported OS page does not specifically mention CentOS 7 but it does mention Red Hat Server 7.9 - 10.

SELinux is disabled, the 13005 port is not being used by another service and is allowed to be used, and I've done the basic troubleshooting.

Thank you in advance.

I got tired of writing KQL from scratch and memorizing column names, so I built KustoForge, a desktop app that lets you build KQL queries through a form-based GUI.

Pick a table, add filters (operators auto-adjust per column type), check the output columns you want, and copy the result. It generates valid KQL in real-time with syntax highlighting.

Covers: MDE, Entra ID/SigninLogs, Sentinel, Azure Monitor, Application Insights, Resource Graph, Defender for Cloud Apps, 52 tables total.

Features:

- Smart operators per data type (string/int/datetime/bool)

- in / !in for filtering value lists

- Save/load query library

- Dark theme, keyboard shortcuts

- Free, open source (MIT), Python + PySide6

GitHub: https://github.com/ChrisHuber1/KustoForge

Feedback welcome! Especially if there are tables or operators you'd want added.

Hello everyone, I'm running into a bit of a pickle with trying to set up a specific Microsoft Sentinel/Defender automation rule and could use some insight.

I would like to create an automation rule that triggers a playbook whenever a specific alert is generated from a Microsoft Security workload—specifically, the Defender for Office 365 alerts "A potentially malicious URL click was detected" or "Suspicious URL clicked".

When this alert is created, the playbook should ping the affected user via Teams to warn them and alert the IT/security team for a priority response.

I have already made some incident-triggered automation rules before, but for this workflow, I specifically need it to be alert-triggered, as an incident coming from these alerts will have a generic name, usually like " Initial access incident on one endpoint reported by multiple sources".

However, when I want to create the automation rule and set the trigger to be alert-based, the conditions section does not let me type in a custom alert name. It only gives me a dropdown set list of native-only Sentinel alerts, not letting me find or select the Defender workload alerts.

So here's my question: Is there a way to force an alert-triggered automation rule to filter by a specific Defender workload alert name?

If this is a UI limitation, what is the best workaround? Should I change the playbook itself?

Appreciate any guidance or workarounds!

How could I ingest webhook´s audit data from a custom saas application? Is it some pre-made logic apps for it?

I know this is well known to seasoned detection engineers, and you'll likely have detection rules that actively monitor these events, but I was just auditing some older detection logic in a client environment and realised their primary credential-dumping alert was still looking for FileName == "lsass.exe" inside DeviceProcessEvents.

If you're doing this, an adversary just has to rename their tool to svchost.exe or update.exe, and you are completely blind. DeviceProcessEvents is for process creation, not for process access.

To reliably detect this without generating massive false-positive fatigue from legitimate system noise, you need to query DeviceEvents, filter for OpenProcessApiCall, and explicitly parse the target image from the JSON fields to check the specific access masks.

Here is the clean KQL block that works well in production and looks for 0x1010 (query/read) and 0x1438 (common tool default):

DeviceEvents | where TimeGenerated > ago(1d) | where ActionType == "OpenProcessApiCall" | extend TargetProcess = tostring(AdditionalFields.TargetImageFile) | extend GrantedAccess = tostring(AdditionalFields.GrantedAccess) | where TargetProcess =~ "lsass.exe" | where GrantedAccess in ("0x1010", "0x1410", "0x1438", "0x143a", "0x1f0fff") | where not (InitiatingProcessFolderPath startswith @"c:\windows\system32\" or InitiatingProcessFolderPath startswith @"c:\program files\") | project TimeGenerated, DeviceName, InitiatingProcessFileName, InitiatingProcessCommandLine, TargetProcess, GrantedAccess 

Found a couple of weird administrative edge cases with legitimate monitoring agents tripping this in a tight loop, so you'll definitely want to tune the folder path exclusions based on whatever endpoint agents your org uses.

Run in your environment to test variants of specific techniques and see what the telemetry looks like.

Curious if anyone else has run into specific bypasses of 0x1010 filtering when attackers manipulate handle rights directly?

Hi

Curious how you utilize the SentinelHealth table? One scenario at the top of my mind - our connector monitoring is queries dependend on the tables to the actual connectors, but we are especially curious to monitor connector health with these diagnostic settings.

As far as diagnostic settings go, there are three categories, 'Analytics', 'Automation' and 'Data Collection - Connectors'. As per my understandings, these won't generate cost and are not billable. Is that correct? Ref. https://learn.microsoft.com/en-us/azure/sentinel/health-audit

We are evaluating switching to Sentinel from AlienVault, but are having a hard time justifying the drop in NIDS traffic from the hardware sensor. We are going to be ingesting logs from zscaler, Meraki (advanced threat protection licensed), and crowdstrike EDR, but the ETPro signatures seem to still be a gap in visibility and alerting.

Has anyone made a similar jump and what did you do in Sentinel to cover the gaps?

I'm trying to export a csv of all the incidents ever registered in Sentinel in my org. This covers 2+ years of Sentinel usage. I did some digging and it turns out you cannot export data from the Threat Mgmt > Incidents tab. You can however export a table of incidents from Log Analytics by SecurityIncident query (set the time frame to the earliest data possible).

So I did exactly that and set the display count to "max limit". Each time, the query only outputs a list going back 90 days in time. Is there a data retention limit in Log Analytics that doesn't allow you to view or export incidents longer than 90 days?

Is there any other way I can go about exporting ALL incidents registered in Sentinel?

Thanks!