SIEM Architecture: Internals

Most SIEM documentation explains how to write a search or build a dashboard. This chapter explains how the platform underneath that search actually works: how a log gets generated in the first place, how it survives, or doesn't, the trip to the indexer, what happens to it structurally once it lands, how a detection rule finds it, and what happens to a human analyst once the alert fires. Every stage is covered mechanically rather than at the marketing-slide level, and grounded throughout in how Splunk and Elastic specifically implement each one, because the two platforms' abstractions diverge in ways that matter operationally, not just syntactically. None of this is theoretical: every number, event ID, and default setting below is the kind of detail that determines whether a real detection rule fires on time, fires at all, or fires against data that already silently rotted somewhere upstream.

The Pipeline

This is the spine the rest of the chapter hangs off. Every stage below is a place an event can be delayed, dropped, malformed, or simply never generated at all, and a detection rule is only ever as good as everything that happened to the data before the rule ran. Treat each subsection as one hop in that chain.

⤢

Log Generation: The Source Layer

On Windows, almost everything downstream of the kernel and the major system services sits on top of Event Tracing for Windows (ETW). Providers, kernel-mode drivers, user-mode services, register with ETW and emit structured events into logging sessions. The Windows Event Log service is one consumer that subscribes to a subset of those sessions and writes what it receives into .evtx files; it is not itself the source of the data, and it does not see everything ETW carries. A custom ETW session, one you configure yourself with a tool that speaks the ETW consumer API directly, can capture telemetry the Event Log never writes to disk at all, because no provider ever routed it there by default. SilkETW and its service counterpart SilkService are built specifically around this: they attach to ETW providers directly and give you visibility the built-in Windows logging pipeline was never configured to surface, which is exactly why they show up in both offensive tooling and defensive telemetry collection with equal frequency.

Most of what a detection engineer actually cares about on Windows is disabled by default and has to be turned on deliberately, and each one has its own separate switch. Command line logging on process creation, Event ID 4688 carrying the ProcessCommandLine field, requires a GPO setting beyond simply enabling "Audit Process Creation" in the basic audit policy; the basic policy only tells Windows to log that a process was created, a completely separate "Include command line in process creation events" setting has to be enabled for the actual command line to appear in the event. PowerShell script block logging, Event ID 4104, is off until someone enables it through Group Policy or the registry, and without it an attacker running heavily obfuscated PowerShell leaves no record of what the script actually did. Sysmon requires a separate install and, more importantly, a real configuration file, the default Sysmon install with no config logs almost nothing useful, and most production Sysmon deployments start from a public baseline like SwiftOnSecurity's config and then get tuned against local noise. More broadly, enabling "Audit Object Access" at the basic policy level does essentially nothing on its own; it has to be paired with a System Access Control List (SACL) configured on the specific file, registry key, or object you actually want access attempts logged against, otherwise Windows has no idea which objects you care about.

Syslog carries the equivalent weight on almost everything that isn't Windows, and most legacy infrastructure still speaks RFC 3164, the original BSD syslog format: no structured data fields, a loosely specified message format that varies by vendor, and no reliable way to parse severity or facility out of the body text without vendor-specific regex. RFC 5424 replaced it with a structured format carrying explicit severity, facility, timestamp, and structured data elements, but adoption lags for the obvious reason that a lot of network and security appliances still in production were built against 3164 and never updated. Transport is a second, independent risk layered on top of format. UDP syslog drops silently under load, there is no acknowledgment, no retry, and no indication to either side that a message never arrived; it is fire and forget in the most literal sense. TCP with TLS is the only combination that gives you both a delivery guarantee and confidentiality in transit, and yet most firewalls and network appliances still default to UDP on port 514 out of the box, because it was the original default and changing it means touching every downstream syslog receiver's configuration too.

Underneath format and transport sits a question almost nobody has an honest answer to: how do you know a given log wasn't altered between the moment it was generated and the moment it landed in the SIEM? Windows Event Log has no native cryptographic signing of event records, an administrator or an attacker with sufficient privilege can modify the underlying .evtx file directly. Auditd's immutable mode, set with -e 2, locks the audit configuration itself until the next reboot, which stops someone from simply disabling auditing, but root still has full control over the transport layer underneath the audit subsystem and can intercept or drop records before they ever leave the host. RFC 5848 defines a signed syslog extension specifically to solve this, cryptographic signatures over syslog messages that let a downstream verifier detect tampering, and it is genuinely rare to find a production environment that has implemented it; the operational overhead of key management for every logging source usually loses out to just trusting the pipe.

Layered on top of all of this is a persistent, underappreciated gap between what a vendor's marketing claims about logging and what the device actually logs in its default configuration. Firewall logging granularity is the clearest example: some vendors log on session creation, some only on session close, some log both, and some only generate a log entry when a rule explicitly matches, silently dropping everything that falls through to an implicit allow or deny with no rule attached. A firewall that only logs on session close gives you zero visibility into a long-lived command and control connection for the entire duration it's open, you don't find out the session existed until it terminates, which for a low-and-slow beaconing implant can mean days or weeks of complete blindness on a connection that was active the whole time. The only real defense against this gap is testing it directly rather than trusting a datasheet: generate traffic that should hit each logging condition and confirm a log record actually appears before depending on that source for detection.

Collection and Ingestion

⤢

Collection is where volume stops being an abstract number and becomes a physical constraint on disk, CPU, and network. The Splunk Universal Forwarder maintains a persistent queue on disk, sized by maxQueueSize, and once that queue fills, the forwarder's behavior depends entirely on configuration: it can block, refusing to accept new data from its inputs until the queue drains, or it can drop, silently discarding events to keep moving. Elastic Agent runs an equivalent internal queue with its own backpressure behavior, and in both cases the failure mode under sustained overload is the same shape even if the specific mechanics differ. During a genuine log storm, a worm propagating across a subnet, a mass authentication failure hitting every domain controller at once, a volumetric DDoS generating firewall log entries by the millions, ingestion lag doesn't creep, it spikes hard and fast. Detection rules scheduled to run against "recent" data start evaluating events that are minutes or hours stale, and "near real time" monitoring quietly becomes "eventually consistent" monitoring at precisely the moment an organization most needs the real-time promise to hold.

This is why sizing a collection tier for sustained average throughput and calling it done is a common and expensive mistake. A tier provisioned for a comfortable average load with no burst headroom will queue, then block or drop, the first time it actually matters, an active incident that is itself generating abnormal log volume. Designing for burst capacity means provisioning queue depth and forwarder throughput against a peak scenario, not the Tuesday-afternoon average, and stress-testing that assumption with synthetic load before an actual incident does it for you.

Log traffic should never share a network path with production traffic, for a reason that goes beyond simple contention: an attacker who compromises the network layer disrupts the attack surface and the monitoring channel in the same single move, which is exactly the scenario a defender least wants to be blind during. Dedicated management VLANs carrying only log transport, TLS mutual authentication between forwarders and indexers so neither side accepts an unauthenticated peer, and certificate pinning to prevent a man-in-the-middle from injecting a trusted-looking certificate on that channel, are the baseline expectation for a log transport architecture that takes its own integrity seriously, not optional hardening bolted on afterward.

Cloud collection is architecturally a different discipline from agent-based collection on a host: it means subscribing to a managed event stream the cloud provider controls, not installing and tuning an agent you own. AWS CloudTrail delivers management-plane events to S3 or CloudWatch Logs on a delay of up to 15 minutes for the S3 path, with near real time delivery only available through CloudTrail Lake or CloudWatch Logs streaming, and data-plane events (object-level S3 access, for instance) have to be explicitly enabled as a separate, billed configuration, they are not on by default alongside management events. Azure routes Activity Log and Entra ID sign-in logs through Diagnostic Settings, a configuration layer that can fan the same event stream out to an Event Hub for near real time consumption or to a Log Analytics Workspace for query-oriented storage, and the choice of destination changes both latency and cost. GCP routes Cloud Audit Logs through Pub/Sub, a push or pull messaging layer that decouples the log source from whatever is consuming it, at the cost of an extra hop and an extra point of configuration that has to be gotten right. Each of these three has a genuinely different latency profile, a different delivery guarantee, and a different failure mode when something upstream breaks, and none of them behave anything like a Windows forwarder writing to a local disk queue.

Large environments do not, as a rule, ship directly from every individual source straight to the SIEM; doing so at scale creates a fan-in problem where ten thousand or more individual sources are all trying to open connections to the same small set of indexers simultaneously. Syslog concentrators, rsyslog or syslog-ng deployed as dedicated relay tiers, absorb that fan-in, aggregating, filtering, and forwarding on to the SIEM as a much smaller number of upstream connections. Reliable delivery through that relay layer is its own separate problem from the transport protocol underneath it: a plain TCP acknowledgment only confirms that the kernel's network stack on the receiving end accepted the bytes into a buffer, it says nothing about whether the receiving application actually parsed and processed the message before, say, crashing or restarting. RELP, the Reliable Event Logging Protocol supported by rsyslog, closes that specific gap by requiring an application-level acknowledgment, the relay only considers a message delivered once the receiving application itself confirms it processed it, not just that the kernel received the bytes.

Parsing and Normalization

A log line sitting as raw text is not useful for detection until it becomes structured fields a query can filter, aggregate, and correlate on, and that transformation step is where silent corruption is easiest to introduce without anyone noticing for months. Grok patterns in Logstash are Oniguruma regular expressions, the same regex engine underneath languages like Ruby, with named captures layered on top, which makes them expressive but also gives them all the performance failure modes regular expressions are known for. A poorly written Grok pattern, one with nested optional groups or greedy quantifiers that can match the same input multiple ways, can trigger catastrophic backtracking on malformed input: the regex engine tries exponentially many combinations trying to find a match, and a single bad log line can stall the entire pipeline behind it while the engine churns. Dissect, Logstash's alternative filter for structured logs, sidesteps this entirely by splitting on fixed delimiters instead of running a regex engine at all, which makes it both faster and immune to backtracking, at the cost of failing outright the moment a source's format deviates even slightly from what was expected.

In Elasticsearch, index mappings are decided once and then largely fixed: the first document sent to a new index defines the type for every field it contains, and Elasticsearch will not silently reinterpret that decision later. A malformed or unusual first document, one where a field that is normally numeric happens to arrive as a string, or vice versa, means every subsequent document inherits that same, likely wrong, type. A field mapped as keyword that later receives a value that should really have been typed ip or long does not throw a loud error, it simply fails to index correctly for that field, or fails to be queryable the way you'd expect, and the failure is often only noticed weeks later when a query that should return results returns nothing. Dynamic mapping templates exist specifically to prevent this class of failure by defining, ahead of time, what type a field matching a given name pattern should always be, regardless of what the first document happened to look like.

Timestamps carry an equivalent but distinct risk, because there is no single universal format in the wild: sources send local time with no time zone marker, UTC, epoch seconds, epoch milliseconds, or genuinely ambiguous strings that could be parsed more than one valid way depending on locale assumptions. A timestamp parsing error of even a few hours is enough to break any correlation rule that depends on temporal proximity, a rule looking for a login followed by a suspicious process launch "within 5 minutes" simply never fires if one of the two events was time-shifted by more than that window during parsing. NTP synchronization across every log source and the SIEM's own infrastructure is not optional hygiene, it is a hard dependency of the entire correlation layer, and a source whose clock has drifted, even without any parsing bug at all, produces exactly the same failure: events that actually happened close together in real time land far apart once they're normalized against a clock that disagrees with everything else.

⤢

Splunk's Common Information Model (CIM) and Elastic's Common Schema (ECS) both exist for the same underlying reason: to normalize fields across wildly different source products into one common vocabulary, so a single search for "failed authentication" works identically whether the underlying source is a Windows domain controller, a Linux SSH daemon, or a cloud identity provider. They diverge in how they handle the edges: fields that don't map cleanly onto the common schema either get dropped, get mapped approximately with some loss of precision, or get preserved under a vendor-specific field name alongside the normalized one. Both models are careful to preserve the original raw event alongside its normalized fields, because normalization is inherently lossy in the edge cases and there always comes a point where an analyst needs to go back to the literal source text to resolve an ambiguity the schema smoothed over. CIM data model acceleration in Splunk takes this further by pre-computing summary indexes over CIM-compliant data ahead of query time, which makes searches against accelerated data models dramatically faster at the cost of the storage and compute spent building those summaries continuously. ECS leans more on runtime fields, computed on the fly at query time from underlying stored fields, trading some query-time cost for not having to pre-decide every derived field's definition up front.

Enrichment, attaching GeoIP location data, ASN ownership, threat intel indicator matches, or resolved user identity to an event, can happen either at parse time or at query time, and the two approaches trade off in opposite directions. Parse-time enrichment embeds the enriched fields directly into the stored document, which makes them instantly queryable with no additional lookup cost on every search, but the enrichment is frozen at whatever the reference data said at ingest time; if a threat intel indicator gets added to a blocklist an hour after an event was ingested, that event's stored document never learns about it retroactively. Query-time enrichment stays perpetually current, because the lookup happens fresh against whatever the reference data says right now, but it costs real CPU on every single search that touches an enriched field, and that cost scales with search volume rather than ingest volume. Which approach is correct for a given enrichment depends almost entirely on how fast the underlying reference data actually changes: MaxMind's GeoIP database updates on a weekly cadence, which makes parse-time enrichment perfectly adequate for location data, while threat intel indicator lists can update multiple times an hour, which makes query-time lookups the only approach that doesn't systematically miss indicators added after ingestion.

Storage and Indexing

⤢

Elasticsearch writes data as new, immutable Lucene segments, Lucene being the underlying search library Elasticsearch is built on top of rather than something Elastic invented itself, and merges smaller segments into larger ones in the background according to a tiered merge policy, rather than updating existing segments in place. Deleting a document does not immediately reclaim its disk space, it simply marks that document as deleted within its segment; the space is only actually reclaimed the next time that segment is merged with others, which means a delete-heavy workload can leave disk usage substantially higher than the live document count would suggest until merges catch up. The refresh interval, index.refresh_interval, defaulting to one second, controls how often newly indexed documents are made visible to search at all; raising that interval to 5 or even 30 seconds reduces the overhead of constantly opening new searchable segments and meaningfully improves indexing throughput, at the direct cost of how quickly a just-ingested event becomes findable by a running query.

Splunk's storage model is bucket-based rather than segment-based: hot buckets, the ones actively being written to, roll to warm status once they hit a size threshold set by maxDataSize or age out past a configured time limit, and warm buckets eventually roll further to cold and frozen storage tiers. Time-series index files, tsidx, are maintained alongside the raw data specifically to make time-range queries fast without a full scan, and bloom filters layer on top of that: a probabilistic structure that lets a search skip an entire bucket outright when it can determine with certainty the bucket doesn't contain a given search term, avoiding the cost of opening and scanning it at all. Both of these mechanisms exist to solve the same underlying problem from different angles, minimizing how much data actually has to be touched to answer a query that only cares about a narrow time range or a specific term.

Durability in both platforms rests on a write-ahead log, but the guarantees differ in ways worth knowing before you rely on them during an incident. Elasticsearch maintains a translog per shard, and a write is only considered acknowledged once it has been written to that translog, meaning an acknowledged write survives a node crash because the translog can be replayed on recovery. Splunk relies on fsync behavior at the hot bucket level to achieve a broadly similar guarantee. In both cases, "acknowledged" has a specific technical meaning that does not automatically cover every failure scenario, a translog itself can be lost if the underlying disk fails before it's replicated, and understanding exactly what failure modes each platform's durability guarantee does and doesn't cover is the difference between trusting a number and verifying it.

Shard strategy in Elasticsearch has consequences that compound at scale in ways that aren't obvious from a single-node test environment. Every shard is its own Lucene index, carrying its own fixed overhead, and every shard's existence is tracked in cluster state, which the master node has to maintain and propagate. Oversharding, spreading data across far more small shards than the data volume justifies, degrades cluster performance because the master node is tracking and coordinating far more state than the actual workload requires. Undersharding, the opposite mistake, too few massive shards, degrades search parallelism because there simply aren't enough independent units of work to spread across available nodes, and it makes rebalancing operations painfully slow because moving a single shard means moving an enormous amount of data in one unit. Twenty to fifty gigabytes per shard is a commonly cited starting point for sizing, and index lifecycle management rollover policies, which roll over to a new backing index once a size or age threshold is hit, directly determine how many shards a given data stream accumulates over its retention period, which means shard strategy and retention policy have to be designed together, not separately.

Retention planning ultimately comes down to one piece of arithmetic that decides whether a SIEM project survives its own budget review: daily ingest volume in gigabytes, multiplied by the retention period in days, multiplied by the replication factor, plus roughly 10 to 15 percent overhead for indexes and metadata that isn't raw event data but still consumes disk. At 500 GB a day, 90 days of retention, and a replication factor of 2 for resilience against node loss, that works out to roughly 100 terabytes before overhead is even added. Compression is what brings that raw number down to something a budget can actually absorb: Splunk applies zstd compression to buckets, Elastic defaults to LZ4, which is fast but leaves some ratio on the table, with an optional best_compression mode using DEFLATE that trades slower indexing for a meaningfully denser result. Typical compression ratios for security log data land somewhere between 5:1 and 10:1 depending on how verbose and repetitive the underlying source format is, and that ratio has a direct, load-bearing line to the number that ends up in the storage budget line item.

Correlation and Detection

⤢

A detection rule's life does not begin at deployment. It is drafted, then tested against historical data to see whether it would have fired on known-good or known-bad periods, then deployed in alert-only mode with no automated response attached so a human can watch its behavior without risking an unwanted action, then tuned against whatever false positives that observation period surfaces, and only after all of that does it graduate to production, with a recurring schedule for periodic review built in from the start rather than added later as an afterthought. Rules rot over time in a way that has nothing to do with the rule itself being wrong: a rule that was high-fidelity six months ago can start generating 90 percent false positives today simply because someone in the organization deployed a new internal tool that happens to trigger the exact same behavioral pattern the rule was built to catch.

The cost of that rot is worth putting an actual number on rather than treating it as an abstract nuisance. A rule firing 50 times a day with 48 of those being false positives, at a conservative 5 minutes of analyst time to investigate and dismiss each one, burns 4 hours of analyst time a day on a single bad rule, and that number multiplies directly across however many hundreds of rules a mature detection program is running simultaneously. Risk Based Alerting in Splunk and entity risk scoring in Elastic both exist as a direct structural answer to this problem: rather than every individual weak signal generating its own standalone alert that demands investigation on its own, low-confidence signals accumulate against a given entity's running risk score, and only once that aggregate score crosses a defined threshold does an actual alert fire, which turns ten independently unremarkable signals on the same user into one well-justified alert instead of ten noisy ones.

Correlation logic that spans multiple sources has to contend with clocks that agree in theory and rarely agree exactly in practice. Authentication logs commonly arrive timestamped in UTC, firewall logs from an appliance configured by a different team might be timestamped in local time, and a cloud log source might carry an inherent 3 minute delivery delay baked into the platform itself before the event even reaches the pipeline. A correlation window written into a rule as "within 5 minutes" might need real-world padding to "within 15 minutes" to reliably catch the exact sequence of events it was designed to catch, once every source's individual timing quirks are accounted for. NTP stratum level, how many hops removed a given time source is from a reference clock, and ordinary network latency both feed directly into how much timestamp accuracy is actually available to build a tight correlation window against, and assuming perfect synchronization when stratum levels vary across sources is a quiet, common cause of correlation rules that mysteriously never fire.

Detection logic also has to account for its own evasion, because a sufficiently aware adversary is not just trying to avoid triggering a rule, they are actively working against the pipeline that feeds it. Log tampering, clearing Windows event logs outright or stopping the auditd daemon on Linux, removes the evidence at the source before it ever reaches collection. Timestomping, altering a file's creation or modification timestamps, is aimed specifically at blending malicious activity into historical noise so it doesn't stand out chronologically during a later investigation. Living off the land, using PowerShell, WMI, or PsExec instead of a custom binary, is aimed at evading detection rules built around known malicious binary signatures, since the "malicious" activity is executed entirely through tools that are also legitimately used by administrators every day. Log flooding, generating a massive volume of benign events deliberately, serves two purposes at once: it can bury a small number of genuinely malicious events in a haystack too large for a human to manually review, and it can simultaneously trigger the exact ingestion backpressure problems described earlier in the pipeline, degrading the SIEM's real-time visibility right when the attacker needs that degradation most. The corresponding counter-detections are specific and well known: watch for Event ID 1102, watch for the auditd service stopping unexpectedly, and watch for PowerShell invocations carrying high entropy in their arguments or specific patterns like -enc, IEX, or [Convert]:: that commonly indicate obfuscated or encoded payloads.

Sigma exists to make detection logic portable across platforms, and the abstraction genuinely works for straightforward logic, but it leaks at the edges in ways worth understanding before depending on a translated rule without review. A Sigma rule compiles from its YAML source into whichever query language the target platform actually speaks, and different backends handle logically identical constructs differently: Sigma's count aggregation compiles cleanly to | stats count in SPL, Splunk's own pipe-based Search Processing Language, but the same logical intent requires a dedicated threshold rule type in Elastic rather than a simple query-time aggregation, because Elastic's detection engine treats thresholded conditions as a distinct rule category. The logsource field inside a Sigma rule is itself only an abstraction, it names a category of log source in the abstract, but mapping that category to an actual index name or Splunk sourcetype happens entirely through separately maintained pipeline configuration that has to be kept in sync by hand. Enough real-world edge cases fail to translate cleanly across this abstraction that manual per-platform tuning after translation is the expected norm for a serious detection program, not a sign that something went wrong.

Alerting and Triage

Analysts working in genuinely high-volume environments routinely see 500 to 1000 or more alerts in a single day, and industry research from Ponemon and SANS consistently puts the share of alerts that never receive any investigation at all somewhere between 25 and 50 percent, not because analysts are careless but because the volume structurally exceeds what a finite team can review. Triage priority in a well-run program is a function of three inputs multiplied together, severity, confidence, and asset criticality, and the resulting number decides where an alert lands in the investigation queue. The split between what can be safely automated, auto-closing alerts that match a known false positive pattern, auto-enriching an alert with context before a human ever sees it, versus what genuinely still requires a human's judgment, a novel pattern nobody has seen before, or any alert touching a high-value target, is the single biggest structural determinant of whether a SOC is sustainable for the people staffing it or is quietly burning them out.

⤢

Splunk Enterprise Security's Notable Event lifecycle moves an alert through a defined sequence of states: New, In Progress, Pending, Resolved, and Closed, and urgency for any given notable is calculated by multiplying the severity assigned by the triggering rule against the priority assigned to the affected asset in the asset framework, so the same rule firing against a domain controller and against a test workstation produces meaningfully different urgency scores automatically. The Incident Review dashboard tracks a small set of core KPIs against this workflow, mean time to detect, mean time to respond, and alert volume trends over time, and adaptive response actions can handle a meaningful share of auto-triage automatically once confidence in a given pattern is high enough to trust without human review.

Deduplication and grouping matter just as much as the quality of the underlying detection, because a single real-world event can otherwise masquerade as hundreds of separate incidents in the queue. Without deduplication, one brute force attack generating 500 individual failed login attempts against the same account becomes 500 separate alerts rather than the single incident it actually represents, overwhelming the queue with volume that carries no additional information beyond the first alert. Splunk's throttling mechanism suppresses duplicate notables matching the same underlying condition for a configured window of time; Elastic's equivalent suppresses alerts sharing the same value in a specified field. Either mechanism, implemented correctly, collapses what would otherwise be hundreds of noisy entries into the single investigation unit the underlying event actually warrants, and alert grouping takes this a step further by rolling multiple related but not identical alerts into one combined incident for a single analyst to work as a unit.

Investigation and Enrichment

⤢

The standard pivot an analyst runs during an investigation follows a consistent shape regardless of platform. It starts from the triggering alert and identifies the specific entity involved, a user account, a host, an IP address, then enumerates the full range of activity attributable to that entity across the time window surrounding the alert, not just the single event that triggered it. From there the analyst looks specifically for lateral movement indicators, checks the source IP against whatever threat intelligence feeds are integrated, checks whether the account shows any signs consistent with credential stuffing or password spraying rather than a single isolated failed login, and maps whatever behavior was observed against known MITRE ATT&CK techniques to give the finding a shared vocabulary other analysts and other tools can reason about. The question the entire pivot is ultimately trying to answer is scope: is this a contained, single-host event, or is it the visible edge of something broader that hasn't fully surfaced yet.

When an investigation escalates past routine triage into a formal incident response engagement, the underlying SIEM data itself needs to be exported and preserved with a chain of custody that will hold up if the finding ever needs to be defended later, whether internally or in a legal context. Splunk supports exporting search results to CSV or JSON with an accompanying hash so the exported data's integrity can be independently verified later. Elastic's snapshot and restore functionality can preserve an entire index's state at a specific point in time, which is a heavier but more complete preservation mechanism than a single search export. Legal requirements for how evidence has to be handled vary meaningfully by jurisdiction, and that variation is something worth understanding and documenting well before an actual incident forces the question, not something to research for the first time while the clock is already running.

Threat hunting is the proactive counterpart to reactive alerting: a hypothesis-driven search conducted with no triggering alert at all, undertaken because an analyst or team has a specific reason to suspect something the existing detection coverage might be missing. The hunting loop follows a repeatable shape: form a hypothesis grounded in current threat intelligence or a gap identified through ATT&CK coverage analysis, craft a query intended to surface evidence for or against that hypothesis, run it against historical data rather than only live data, refine the query based on what comes back, and whatever genuinely gets found along the way gets converted into a permanent, reusable detection rule rather than being a one-time finding that has to be manually rediscovered next time. In a mature detection program, hunting is in practice how a large share of new detection rules actually get born, the loop runs in the opposite direction from what the org chart often implies, hunting feeds detection engineering more often than detection engineering feeds hunting.

Response and Orchestration: SOAR

⤢

A SOAR (Security Orchestration, Automation, and Response) playbook has a consistent internal architecture regardless of which platform implements it. A trigger defines what alert or condition fires the playbook in the first place. A decision tree branches the subsequent logic based on specific attributes of that alert, the affected asset, the user involved, the detection's confidence level. Actions are the concrete API calls the playbook makes outward, to a firewall, an EDR platform, an identity provider, a ticketing system. Approval gates insert a human-in-the-loop checkpoint specifically before any destructive or hard-to-reverse action executes. Post-action validation checks whether the action the playbook just took actually achieved its intended effect rather than just assuming success because an API call returned without an error. And a closure step updates the case record and notifies whichever stakeholders need to know the incident has been handled.

Every one of those action steps is also an integration point, and every integration point is a distinct potential failure mode with its own specific quirks. A REST call to a Palo Alto firewall for an IP block goes through the PAN-OS API; a Fortinet firewall has its own separate FortiOS API with different semantics. Host isolation goes through CrowdStrike Falcon's dedicated containment endpoint, Microsoft Defender's isolate machine endpoint, or Elastic's own host isolation response action, three different APIs accomplishing conceptually the same outcome. Disabling an account routes through Microsoft Graph API for an Entra ID identity or AWS IAM's UpdateLoginProfile call for an AWS-native identity. Ticketing integrations go through the ServiceNow REST API or the Jira API depending on which system the organization actually uses for case tracking. None of these integrations are interchangeable, and a playbook built against one vendor's API semantics does not port cleanly to another vendor without real engineering work.

Mean time to respond is measured from the moment an alert fires to the moment containment is actually achieved, and it moves in clear, discrete steps as the automation level behind a given response increases. Fully manual response, where a human reads the alert and manually executes every containment step by hand, is the slowest tier by a wide margin. Notification-only automation, where the system tells a human what it recommends but takes no action itself, is faster because the human isn't spending time diagnosing from scratch, but is still fundamentally gated on that human's availability and response time. Semi-automated response with a required approval step executes the actual containment action the instant a human clicks approve, cutting out the manual execution time entirely while still keeping a human decision in the loop. Fully automated containment of high-confidence detections, no human step at all for a narrowly scoped, well-validated category of alert, is the only tier that reliably gets mean time to respond down into minutes rather than hours, and it is also the tier that makes the pre-action validation described above absolutely non-negotiable, because there is no human left in the loop to catch a mistake before it executes.

Splunk vs Elastic: Architectural Differences

Splunk and Elastic solve the same underlying problem, ingest a huge volume of heterogeneous log data and make it searchable fast enough to matter for security work, but they make different enough architectural bets in doing so that the choice between them shapes everything downstream of it: deployment topology, cost structure, and even which detection patterns are natural to express versus which ones fight the platform.

Licensing and Its Architectural Impact

Splunk licenses primarily by daily ingest volume measured in gigabytes, and that single fact creates constant, structural financial pressure to filter, trim, and reduce data before it ever reaches the indexer, a pressure that sits in direct tension with the security instinct to simply collect everything and sort out its value later. Elastic self-managed carries no ingest-based licensing at all, its practical ceiling is whatever hardware you're willing to provision, which removes that specific pressure but replaces it with a different one: hardware and operational cost scale with volume regardless of what the license technically allows. Elastic Cloud instead charges by resource consumption, compute and storage actually used, which reframes the same underlying tradeoff in infrastructure terms rather than a per-gigabyte ingest fee. Elastic's tiering model, a genuinely free open source basic tier, then paid Gold, Platinum, and Enterprise tiers above it, gates specific capabilities like ML-based anomaly detection, document and field-level RBAC (Role-Based Access Control, restricting what a given role can see or do), and cross-cluster search (querying multiple separate Elasticsearch clusters as though they were one) behind a subscription regardless of ingest volume, which means the meaningful cost lever for Elastic is often which features you need rather than how much data you're sending. This single licensing difference, ingest-based versus resource-based versus feature-tiered, is arguably the single largest driver of how a given organization's actual deployment architecture ends up looking in practice, more than almost any other design decision either platform offers.

Schema on Read vs Schema on Write

Splunk indexes data in something close to its raw form and applies structure only at search time, extracting fields when a query asks for them rather than deciding field types up front. This preserves full retroactive analysis capability in a way schema-on-write cannot: you can write an entirely new field extraction today and immediately apply it against data that was ingested years ago, because the raw text is still sitting there waiting to be reinterpreted. Elasticsearch defines field structure at index time instead, before the data is ever queried, which pays off directly in query performance against fields that are already typed, indexed, and ready to be filtered on efficiently, but it comes at the cost of needing the schema decided correctly before ingestion, or accepting the operational overhead of reindexing historical data later if that early schema decision turns out to have been wrong. The storage cost implications run in the same direction as the query performance ones: pre-built indexes cost more to store than raw text but pay that cost back in search speed, and CIM versus ECS, discussed earlier, is really the practical, field-level expression of this exact same architectural tradeoff playing out at the schema layer specifically.

Forwarders and Agents

Central management of these components follows the same architectural split as everything else between the two platforms: Splunk's Deployment Server pushes configuration bundles out to fleets of Universal and Heavy Forwarders from a central point, while Elastic Fleet, managed through Kibana's interface, performs the equivalent centralized configuration push and policy management for fleets of Elastic Agent. Choosing a heavier component, a Heavy Forwarder or Logstash, over a lighter one trades endpoint resource consumption for earlier, more capable processing, and that tradeoff decision has to be made per data source rather than platform-wide, since a low-volume critical source and a high-volume noisy source often warrant opposite choices.

Query Languages

SPL reads left to right, each stage of a search taking the output of the previous stage as its input, and that structure makes it genuinely powerful for statistical operations, multi-stage transformations, and subsearches that feed the result of one search into the filter criteria of another. KQL here is Kibana Query Language, Elastic's own simple filter syntax, not to be confused with Microsoft's identically-abbreviated Kusto Query Language used in Sentinel, and it is deliberately simpler and more approachable for someone new to the platform, but that simplicity comes at a real cost in expressiveness for anything beyond straightforward filtering, aggregations of real complexity routinely push a KQL user back to the underlying Elasticsearch Query DSL directly, or to Lens's visual query builder, because KQL alone doesn't have the vocabulary to express what's needed. EQL, Event Query Language, is purpose-built specifically for sequence detection and temporal correlation, expressing "this event followed by that event within this window" as a first-class query construct, and it is explicitly not intended as a general-purpose query language the way SPL and KQL both are, it exists to solve one narrow problem extremely well rather than to be a universal tool.

Storage Tiers

⤢

Elastic's frozen tier stores data as searchable snapshots directly on cheap object storage, and critically, that data remains queryable in place without any explicit restore step first, a search against frozen-tier data simply takes longer than one against hot-tier data, but it works without manual intervention. Splunk's cold and frozen storage tiers generally require an explicit thaw operation, restoring archived buckets back into a searchable state, before a search can run against them at all, which introduces both a manual step and a delay that Elastic's architecture avoids by design. That single architectural difference has outsized consequences for any organization with long-term retention or compliance obligations where old data occasionally needs to be searchable on short notice: the frozen-but-instantly-queryable model and the frozen-but-requires-thawing model represent genuinely different operational commitments, not just a difference in default settings.

Cluster Architecture

⤢

Splunk's cluster architecture is built from a small set of distinct role types working together: search peers forming an indexer cluster that actually stores and searches the data, a search head cluster layered on top that coordinates and dispatches searches across those peers, a cluster master, called cluster manager in newer versions, that coordinates data replication and peer health across the indexer cluster, and a forwarder management layer feeding data in from the edge. Elasticsearch instead assigns granular node roles rather than building around a small number of cluster-level components: master-eligible nodes that can be elected to manage cluster state, data nodes that are further split by storage tier (hot, warm, cold, frozen), coordinating nodes that receive and fan out queries without holding data themselves, dedicated ingest nodes that run ingest pipelines, and ML nodes reserved specifically for machine learning workloads. Node failure handling, how shard or bucket replicas get promoted and rebalanced after a node drops out of the cluster, differs enough in the specific mechanics between the two platforms that the operational discipline required to run either one well at real scale, hundreds of nodes, petabytes of retained data, ends up looking like a genuinely distinct skill set depending on which platform is underneath, not a thin layer of syntax difference over the same operational reality.

Search Performance at Scale

Splunk's search execution model is close to classic map-reduce: the search head dispatches the search to every indexer peer holding potentially relevant data, each peer independently executes its portion and returns partial results, and the search head merges those partial results into the final answer. Elasticsearch's coordinating node instead fans a query out only to the specific shards that are actually relevant to the query's filters, each targeted shard executes its portion of the search locally, and the coordinating node merges just those results. Both models degrade under load, but they degrade in their own characteristic ways: Splunk's model can be bottlenecked by the slowest responding peer among all of them since every peer that might hold relevant data gets queried, while Elasticsearch's model is more sensitive to how well the data is actually distributed across shards, since an unevenly distributed dataset means some shards do disproportionately more work than others even when the query itself is well written. Knowing which failure shape to expect from which platform matters more for real capacity planning than either platform's theoretical performance numbers do.

Multi-tenancy and RBAC

Splunk's access control model is fundamentally index-level: roles are granted access to specific indexes, and a user's effective access is whatever their assigned role or roles grant them at that index granularity, with no finer-grained control over which specific fields or documents within an index a given role can see. Elasticsearch supports genuinely finer-grained document-level and field-level security, letting an administrator restrict a role to specific documents matching a query, or hide specific fields entirely from a role even when it has access to the rest of the document, but that capability sits behind Elastic's paid tiers rather than being available in the open source base. The security implication of this difference is concrete rather than theoretical: an index-level-only RBAC model can hand a junior analyst broad access to an entire index of raw authentication logs, including any plaintext credentials that ended up logged from failed login attempts, simply because there was no finer control available to exclude just that field, a misconfiguration that a field-level security model would have made structurally impossible rather than merely discouraged.

Near Real Time Monitoring

Quantifying End to End Latency

⤢

It is worth walking through real numbers for every stage of the pipeline rather than trusting the vague phrase "near real time" at face value. The event itself occurs on the endpoint at T+0, that's the reference point everything else is measured against. The local agent then has to actually notice and read it, roughly T+2 seconds if the agent is watching a log file for changes and has to detect the write, effectively T+0 if the agent instead holds a direct subscription to something like the Windows Event Log that pushes new events immediately rather than requiring a poll. The agent then batches whatever it has collected and ships it onward, typically somewhere in the T+10 to T+30 second range depending on the batching interval configured. The indexer on the receiving end has to receive, parse, and index the event before it's searchable at all, T+1 to T+5 seconds under normal load, but that same step can stretch anywhere from T+30 seconds to a full T+5 minutes under heavy load, which is exactly the backpressure scenario covered earlier in the collection stage. Only once all of that has happened does the detection rule itself get a chance to evaluate the event, and that evaluation happens on whatever schedule the rule runs on, T+60 seconds for an aggressively scheduled Splunk search, T+300 seconds as Elastic's more conservative default rule interval. Stack every one of those stages together and the realistic floor, everything going well, is 15 to 45 seconds from the moment the event happened to the moment a rule could theoretically catch it. The realistic ceiling once any part of the pipeline is under real load stretches to 5 to 10 minutes, and during an active log storm specifically, the exact scenario most likely to coincide with something worth detecting in the first place, that ceiling can stretch past 30 minutes. None of this is a defect in either platform, it is the actual physics of the architecture as built, and it is the number that should anchor any conversation with stakeholders about what "real time" monitoring can and cannot promise.

Comparison with True Real Time Systems

IDS and IPS platforms, Suricata and Snort being the two most common open examples, perform inline packet processing with sub-millisecond detection latency, because they are evaluating traffic as it physically passes through them rather than waiting for it to be logged, shipped, and indexed anywhere first. EDR agents evaluate behavior directly on the endpoint where it's happening, giving them second-level latency for local detection decisions, again because there is no shipping or indexing step in the loop before the detection logic runs. A SIEM, by contrast, is fundamentally an aggregation and correlation layer sitting behind both of those systems in the overall detection stack, not a first line of detection in its own right, and its entire value proposition is correlating signals across many sources over time, not catching any single event the instant it happens. Understanding this layered latency model explicitly, and being able to explain it in those terms, is what keeps a legitimate architectural tradeoff from being unfairly judged against a standard, true real time, single-event detection, that the SIEM was never designed to meet in the first place.

Streaming vs Batch

A SIEM, underneath any amount of near real time tuning, remains fundamentally search-based in its core architecture: index the data first, then query it, in that order, every time. Genuine streaming architectures work the opposite way, Kafka feeding into Flink or Spark Streaming processes events while they are still in flight, evaluating and acting on them before they are ever durably stored anywhere at all. Splunk's Data Stream Processor, since folded into the broader Splunk Cloud offering, and Elastic's EQL evaluating sequences as data arrives both represent genuine pushes toward that streaming model at the edges of their respective platforms, narrowing the gap for specific use cases. But underneath both of those additions, the core architecture of both platforms remains index-first, and neither one has actually replaced its foundational search-based model with a true streaming one, they have layered streaming-adjacent capabilities on top of it instead.

The Convergence: SIEM, EDR, and XDR

Vendor Landscape

Every one of these vendors is converging toward the same rough destination, a single platform holding both endpoint telemetry and broader log data, from a genuinely different starting point, and that starting point still shapes where each vendor's product is strongest today, a SIEM-origin platform tends to have deeper correlation and search capability, while an EDR-origin platform tends to have deeper endpoint-level detection and response capability, even after both have nominally added the other side.

Native vs Open/Hybrid XDR

Native XDR means a single vendor supplies every sensor in the environment and the platform correlating across them, which buys genuinely deep integration between components that were designed together from the start, at the direct cost of vendor lock-in across the entire stack at once rather than just one component. Open XDR instead ingests third-party telemetry from whatever tools are already in place and correlates across them from outside, trading away some of that integration depth in exchange for the flexibility to keep using best-of-breed tools per category rather than being forced onto one vendor's version of everything. "XDR" as a marketing label, used on its own, is close to meaningless until it's pinned down with three specific questions: what telemetry does this product actually ingest, what response actions can it actually take once it detects something, and does adopting it require ripping out tools the organization already trusts and has invested in operationally.

The Data Gravity Problem

⤢

Detection Parity Between SIEM and EDR

EDR agents see an enormous amount of detail at the kernel level: full process trees showing parent and child relationships, memory operations including injection and allocation patterns, registry writes as they happen, and file system activity in real time as it occurs on disk. The SIEM, by contrast, only ever sees whatever subset of that raw telemetry the EDR agent has been configured to forward onward as a discrete log event, which is almost always a small, curated fraction of everything the agent actually observed locally. If the EDR agent detects and even blocks a threat entirely on its own, locally, but never generates a corresponding log event destined for the SIEM, the SIEM ends up with a blind spot it has no way of even knowing exists, because from the SIEM's perspective nothing happened at all. That distinction, detection happening inside the agent versus detection happening at the platform layer where the SIEM can see and correlate it, matters considerably more in practice than most vendor architecture diagrams, which tend to draw a single unbroken arrow from endpoint to platform, make it look.

Telemetry Pipeline Tax

A single unified agent handling both EDR-style local detection and log shipping duties necessarily shares the same finite CPU, memory, and network bandwidth budget on the endpoint it's running on, because both functions are ultimately competing for the same underlying hardware resources. Under genuinely heavy load, active incident response work generating unusual local activity, a large-scale vulnerability scan sweeping the network, the EDR detection function and the log shipping function end up directly competing with each other for that shared, limited resource pool, and one can degrade the other's effectiveness without either function individually failing outright. Correctly sizing the agent's resource limits ahead of time, and understanding specifically what happens, which function degrades first and how gracefully, when the agent is genuinely starved for resources, is not a minor footnote in an architecture review, it is a real and often underestimated source of operational risk precisely during the moments an organization can least afford either function to quietly fail.

Detection Engineering

Detection as Code

⤢

Treating detection rules as software rather than as one-off configuration entries means putting them under real version control in Git, requiring code review from another engineer before a new or modified rule merges, and building an actual CI/CD pipeline around the whole process: validating Sigma syntax automatically, compiling the validated rule to whatever the target platform's native query language is, running it against a known test dataset to confirm it behaves as expected, and only then deploying it to a staging environment ahead of production rather than pushing straight to production on faith. The DeTT&CT framework, short for Detect Tactics, Techniques and Combat Threats, layers directly on top of this discipline, providing a structured way to map existing detection coverage against the full MITRE ATT&CK matrix and systematically surface exactly which techniques have no corresponding detection at all, a gap that's very easy to miss when reviewing rules one at a time by hand rather than against the full matrix at once.

Purple Teaming Feedback Loop

A purple team exercise runs a tight, deliberate loop: a red team operator executes one specific, well-defined technique, the blue team checks in near real time whether the SIEM actually detected it, and if it didn't, that gap immediately becomes the input for a new detection rule, which then gets written, tested against the same technique, and deployed, closing the loop. Atomic Red Team tests serve as the practical unit-testing equivalent for this entire process: small, narrowly scoped tests of individual ATT&CK techniques in isolation that can be re-run on demand to confirm a given detection rule still actually fires, the same way a unit test confirms a function still behaves correctly after a code change elsewhere in the system.

Tuning Methodology

Reducing a rule's false positive rate without simultaneously introducing false negatives that let real attacks slip through requires a genuinely structured process rather than ad hoc adjustment based on whatever complaint came in most recently. Whitelisting approaches explicitly exclude specific known-good processes, IP addresses, or user accounts that are confirmed to be the source of a recurring false positive. Threshold adjustment raises the count or frequency required before the rule fires at all, filtering out low-volume noise while still catching genuinely elevated activity. Logic refinement goes a level deeper than either of those, adding actual conditions to the rule that structurally distinguish malicious behavior from benign behavior that merely resembles it, rather than just suppressing whichever specific instances have already been observed and complained about. Every exclusion added through any of these methods is also, unavoidably, a potential blind spot: an attacker who has learned, through reconnaissance or insider knowledge, exactly what's on a given whitelist has effectively been handed a documented, tested path around that specific rule, which means over-tuning a rule into silence carries a real security cost that's easy to underweight in the moment when the immediate goal is simply making an annoying alert stop firing.

Detection Coverage Metrics

A small set of tracked numbers, looked at together rather than individually, tells you far more about whether a detection engineering program is actually working than rule count alone ever could. The percentage of MITRE ATT&CK techniques with at least one corresponding detection rule shows breadth of coverage across the full range of known adversary behavior. The percentage of existing rules that have actually fired at least once within the last 90 days is a useful, if imperfect, health check, since a rule that has fired zero times in three months might genuinely be catching nothing because there's nothing to catch, or might simply be silently broken and nobody has noticed yet. The mean time from a new piece of threat intelligence becoming available to a corresponding detection rule actually being deployed measures how responsive the whole program is to a changing threat landscape. And the false positive rate tracked per individual rule, rather than averaged across the whole rule set, is what actually identifies which specific rules are quietly consuming a disproportionate share of analyst time relative to the value they provide.

Sigma Rules: Deep Dive

Sigma compiles from a vendor-neutral YAML source format into whichever native query language the target platform actually understands, and the compilation step is where logically identical rule constructs can end up meaning subtly different things depending on the backend compiling them. The same count aggregation construct that compiles cleanly and directly to a simple | stats count pipeline stage in SPL requires an entirely dedicated threshold rule type to express the equivalent logic in Elastic, because Elastic's detection engine architecturally treats threshold-based conditions as their own distinct rule category rather than as a general-purpose aggregation any rule can express inline. The logsource field inside a Sigma rule is itself only ever an abstract category label, mapping that abstract category to a concrete index name or Splunk sourcetype happens entirely through separately maintained pipeline configuration files that a rule author has no direct control over and has to trust are kept accurate and current. Enough genuinely tricky edge cases fail to translate cleanly across this whole abstraction layer that manual tuning after automated translation remains the expected, normal final step for any serious Sigma-based detection program, not a sign that the translation tooling failed. The SigmaHQ public repository, holding well over 3000 community-curated rules covering a huge range of known tactics, techniques, and procedures, is genuinely useful as a starting point and a real time-saver, but it functions as a head start toward a working detection program, not as a finished one dropped in wholesale.

MITRE ATT&CK Mapping

Mapping every detection rule to the specific MITRE ATT&CK technique ID it's intended to catch converts what would otherwise be a flat, unstructured rule list into an actual coverage visualization, a heatmap showing exactly which parts of the ATT&CK matrix have detection coverage and which parts have none, making genuine gaps identifiable at a glance rather than only discoverable by someone manually cross-referencing a spreadsheet against the matrix by hand. Splunk Enterprise Security ships a dedicated Content Update app specifically built around maintaining this ATT&CK mapping over time as the matrix itself evolves, and Elastic's Detection Rules interface includes its own built-in coverage heatmap serving the identical underlying purpose within that platform.

Log Source Reference

For each of the major source categories below: what has to be explicitly enabled before it produces anything useful, the blind spots that persist even once it's properly configured, and the essential fields worth building real detections against rather than just collecting for volume.

⤢

Windows

Sysmon functions as an enhancement layer sitting on top of native Windows event logging, providing several specific categories of telemetry that native logging simply does not produce on its own no matter how the built-in audit policy is configured: process creation events that include the full command line together with the parent process relationship, network connection events tied to the specific originating process rather than just the connection itself, file creation events carrying a computed hash of the created file, and WMI activity monitoring that native logging has essentially no visibility into at all. None of that telemetry exists without Sysmon deployed and, just as importantly, deployed with a real configuration file tuned against local noise rather than the default install, which logs comparatively little of value on its own.

Essential audit policy subcategories go well beyond the small set of event IDs in that table, and Windows' advanced audit policy, configured and inspected through auditpol, exposes dozens of individually toggleable subcategories the basic policy editor doesn't surface at all. Logon events, account management, process tracking, object access, and policy change each have their own advanced subcategory that has to be deliberately enabled, and a subcategory left at its default, frequently disabled or set to log successes only, produces exactly the kind of quiet coverage gap that's invisible until an investigation goes looking for an event that was never actually being generated in the first place.

Linux

Auditd rules built specifically for security monitoring, rather than the distribution's default ruleset, should center on a small number of syscalls that carry a disproportionate share of the useful signal: execve for process execution, capturing what actually ran; connect for outbound network connections, capturing where a process reached out to; and open or openat for file access, capturing what a process actually touched on disk. Capturing full command arguments alongside the bare syscall event is what turns a log entry from "a process executed" into "this exact command with these exact arguments executed," and it requires explicit rule configuration rather than coming for free with a default auditd setup. Audit logging and syslog serve genuinely different purposes on a Linux host and shouldn't be treated as interchangeable or redundant with each other, auditd is purpose-built for security-relevant kernel-level events specifically, while syslog carries general application and system messages of every kind. Journald versus traditional syslog is its own separate decision layered on top, with different retention defaults, different query tooling, and different forwarding behavior depending on which one, or which combination, a given distribution and configuration actually uses. Containerized environments compound every one of these challenges at once: dozens or hundreds of ephemeral containers, each carrying its own identity and its own short lifecycle, come and go far faster than most traditional log pipelines were ever tuned to track, and a container that spins up, does something malicious, and terminates within seconds can be gone entirely before a slower collection cycle ever captures it.

Cloud

Management plane logs record who called which API and when, AWS CloudTrail, Azure Activity Log, GCP Admin Activity Logs all serve this same fundamental role for their respective platforms. Data plane logs instead record who actually accessed which specific piece of data, S3 access logs recording individual object access, Azure Storage analytics logs doing the equivalent for blob storage. Most organizations, when standing up cloud logging for the first time, enable management plane logging because it's the more obvious, more heavily documented default, and stop there without separately and deliberately enabling data plane logging, which frequently costs extra and has to be turned on explicitly per resource. The practical consequence is a large, common, and often unrecognized blind spot specifically around data exfiltration, since exfiltration by definition happens on the data plane, an attacker who has already compromised a set of credentials can read or copy an enormous amount of sensitive data while generating precisely zero management plane events, because they never touched a configuration API, they only ever touched the data itself.

Network

Flow data, NetFlow or its more modern IPFIX equivalent, captures metadata about a network connection, who talked to whom, when, and roughly how much data moved, without capturing any of the actual content of that conversation. Full packet capture captures the opposite end of that same tradeoff, the literal content of what was said, at a storage cost that most environments simply cannot sustain continuously at any real scale, which is exactly why a SIEM typically ingests flow data and IDS alerts as its primary network telemetry rather than raw PCAP, reserving full packet capture for narrowly targeted, temporary collection during an active investigation rather than as standing infrastructure.

Operational Considerations

SIEM as a Target

Monitoring the Monitor

The SIEM needs its own dedicated health alerting, entirely separate from the security detections it runs against everyone else's data, covering at minimum: forwarder connectivity, specifically which sources have stopped sending data and exactly when they stopped, since a source going silent is functionally identical to a blind spot even if nothing malicious caused it; ingestion rate anomalies, sudden unexplained drops or spikes in overall volume that suggest either a source went dark or a pipeline stage broke somewhere upstream; detection rule execution failures, rules that are supposed to be running on schedule but are silently failing to execute at all rather than simply finding nothing; storage capacity thresholds, since running out of disk on an indexer is a self-inflicted outage that's entirely preventable with enough lead time; and license usage approaching its configured limit, particularly relevant under Splunk's ingest-based model where crossing the limit has direct operational consequences for indexing. Splunk's Monitoring Console and Elastic's cluster health APIs paired with Stack Monitoring both provide the tooling to track every one of these, but the tooling only has value if someone on the team is actually configured to watch it continuously and empowered to act on what it shows, monitoring infrastructure that nobody is actually watching provides exactly the same protection as no monitoring at all.

Cost Management

A SIEM is routinely one of the single largest line items in an organization's entire security budget, and under Splunk's ingest-based licensing model specifically, cost scales directly and predictably with log volume, which means cost control and data strategy end up being the same conversation whether anyone frames it that way or not. The realistic levers for reducing ingest volume without actually losing meaningful visibility fall into a few well-established categories: filtering out noisy, genuinely low-value logs at the forwarder before they ever reach the indexer, allowed traffic on well-known, already-trusted firewall ports is a common and relatively safe first cut; aggregating data before ingestion wherever the individual event itself isn't the point, ingesting a rolling count of connections per minute instead of every individual connection record for an extremely high-volume, low-signal source; and tiered retention policies that keep genuinely high-value security logs available for a full year or longer while letting purely informational, low-value logs age out and get deleted after a much shorter window, commonly 30 days. None of these levers are free, each one trades away some amount of raw visibility for cost control, and the actual skill in cost management is knowing precisely which specific data can absorb that tradeoff safely and which data absolutely cannot.