← all systems · see it in The System
Security information and event management (SIEM)
A field within computer security that combines security information management (SIM) and security event management (SEM) to enable real-time analysis of security alerts generated by applications and network hardware; NIST defines a SIEM tool as an application that gathers security data from information system components and presents it as actionable information via a single interface (src:wp:security-information-and-event-management). Researchers describe SIEMs as systems with 'the capacity to collect, aggregate, store, and correlate events generated by a managed infrastructure', the central platform of modern security operations centres (src:gonzalez-granadillo-2021-siem). Walked as the technology and the content its operators write: collectors and agents, parsing and normalisation, the event store, correlation rules and use cases, alarms, dashboards, searches, reports and automated reactions, with the people who run it at its top floor.
Properties published scienceour proposal
Sourced fields, and our own readings.
| aliases | SIEM |
| born at | 2005 (term); SIM/SEM products before it; forerunners 1977 (NBS SP 500-19 audit criteria) and 1980 (Anderson's audit-trail surveillance) |
| born precision | years |
| boundary | Inside: the collectors and agents, parsers and normalisers, the event store and index, the correlation and rule engine with its rules and use cases, the prioritisation module, the knowledge base of assets and threats, the consoles, dashboards, search node and reporting, and the automated reactions it triggers; with the content its operators write (rules, parsers, use cases). Outside: the estate it watches (hosts, applications, networks, users, OT plant), the tools whose alerts it reads (IDS, firewalls, antivirus, which other walks own), threat intelligence platforms, the SOC analysts who judge its alarms, SOAR platforms, the organisation, auditors and regulators. Whether its operators belong inside is contested (C-1623). |
| nested level | A technological system (rung 12): software and appliances on log and database servers, a part of the security operations centre (its collection and correlation floors) inside an organisation; built on logs written by operating systems, applications and security tools; its top floor is run by people. |
| scale | technological |
Measures (18) published science
| live retention | no more than 90 days |
| raw event retention (typical archive) | about 6 months, about 6 TB months; terabytes |
| correlation window (failed-login rule) | 3 or more failed logins in 1 minute; 15 or more firewall drops in 1 minute events per minute |
| characteristic clock: log rotation | hourly, daily or weekly, or at a size limit schedule |
| events per day (large enterprise) | millions or billions events per day |
| alerts per day at one SOC | 24,000-134,000; 0.01% tied to true attacks; 49% benign triggers alerts per day; share |
| network alerts in the dataset (2018-2022) | 115,617,526 alerts |
| post-attack investigation time | 53 days on average |
| manual correlation per ticket (one SOC, slow SIEM) | 10 minutes on average |
| evadable Sigma rules | 110 of 292 fully (38%), 19 partially (7%) rules |
| benign to malicious events in validation data | about 145,000 : 1 ratio |
| redundant rules in public rule sets | 139 of 2,347 Sigma; 58 of 1,640 Splunk rules |
| SIEM market size (Gartner, as cited) | 3.55 billion (2019) to 3.58 billion (2020) US dollars (currency as cited) |
| vendors in the 2020 SIEM vendor map | 22 vendors |
| share of surveyed SOC practitioners using a SIEM | 80% (IDS 90%) of 20 survey participants |
| anomaly threshold in the 1980 surveillance plan | 2.58 standard deviations |
| log volume of one commercial cloud application | about gigabytes per hour data per hour |
| signal yield per hand-off (alerts to true attacks) | 0.0001 (0.01%) usable out / sent in |
Open contentions (5) open contention
Where sources or readings disagree. Both stay on the map.
- C-736 When a system first throws away its excess energy and later learns to store it for reuse, is that one process that grew or two separate ones?
booked: The map keeps both readings side by side and rules neither. A few examples fit the 'one process that grew' reading, but it hasn't been tested.
the other reading: the toll and its dark twin as one Function before and after a fold - C-1118 In cybersecurity, do attackers have the advantage?
booked: The map keeps both: attack is easier because defenders must cover everything, and attackers have no inherent advantage because one block can break their chain.
the other reading: attacker advantage; no inherent advantage - C-1622 When did security event management (SIEM) begin?
booked: The map dates SIEM to its naming in 2005 and keeps the 1970s-80s audit-trail work as forerunners.
the other reading: SIEM is the latest form of security audit-trail surveillance, which begins with the 1970s audit working groups (NBS SP 500-19, 1977) and Anderson 1980; SIEM begins with the separate SIM and SEM products before 2005, the name only joining them - C-1623 Is a SIEM just the software, or the people and rules around it too?
booked: The map books the software plus the rules its operators write, with the operators themselves at its top floor.
the other reading: the SIEM is only the software; use cases and rules belong to the SOC; the SIEM is the whole operational practice including its engineers and analysts (its 'operational role') - C-1624 Does SIEM correlation work?
booked: The map keeps both: correlation sits at the core of most security centres, and many deployments could not do it well.
the other reading: SIEM correlation works when its rules and data sources are designed and tuned for the environment; SIEMs are bought as 'best practice' and add little beyond central storage
Cross-scale patterns seen here (16) our proposal
Shapes that recur when the same kind of problem is solved at different scales, and how each shows in Security information and event management (SIEM). Each links to every other system that shows it.
- partner-becomes-part (structural)
Two product kinds that worked side by side, security information management (auditing, long-term storage) and security event management (real-time monitoring, incident response), became one system: 'At one time, many products were either SEM-specific ... or SIM-specific ... Most current products per - same role, different builds (structural)
One job, turning a flood of alarms from many sources into a few that carry meaning, done by unrelated builds: the SIEM's correlation rules; telecom network management's alarm correlation ('a conceptual interpretation of multiple alarms such that new meanings are assigned'); public-health syndromic s - templating (structural)
Sigma rules: one rule written once in a generic YAML format and converted unchanged into the queries of Splunk, Elasticsearch or QRadar's AQL (RuleGenie converted all 2,347 Sigma rules into AQL); syslog and Windows Event Log as shared formats every product reads. - the whole sets the local reference (structural)
Each log source reads its own clock and writes its own format; the SIEM fixes one time zone, one event format and one field meaning that all must be read against. Locally each timestamp is true, yet unsynchronised clocks mean the readings may not close into one order until times are added or subtrac - trade one property for another (structural)
Three trades with kinds at different points: catching false positives versus missing true ones ('I can eliminate 100% false positives, but I am going to miss some of the true positives'); agentless collection (nothing to install) versus agents (filtering at the host, less data on the network); rules - Umwelt: each system carves its own slice of one shared medium (structural)
A SIEM sees only the sources onboarded and parsed: 'none of them have all the data needed'; most correlate logons, malware and web logs, few DNS, endpoint and e-mail logs. Two SIEMs on the same estate carve different slices, and an attack in the unread slice is invisible to it. - who opens the door (structural)
Every input arrives through a door set up ahead of time by the receiver: ports opened on the firewall before log forwarding, an agent installed on the host, credentials for pulling, a parser for that source type, a rule written for the use case. Traffic then runs into doors that already exist. - borrowed-then-held (partial)
Log analysis first borrowed, then held: syslog servers 'did not perform any analysis'; administrators used separate add-on programs; later implementations built correlation in, and the SIEM holds it as its own floor. The correlation job is still partly borrowed from the analysts' heads ('analysts ar - holding / entrainment (partial)
The log management infrastructure holds every host's clock to a common time source and sets when logs rotate (hourly, daily, weekly). Partial: the reference is an outside NTP server, so it borders on the dark twin (parts sharing a driver), though administrators impose and check it. - namespace completeness (the map predicts its gaps) (partial)
Use cases are derived from MITRE ATT&CK, whose tactics and techniques (phishing is T1566 under Initial Access) lay out where detection should exist; a technique with no rule shows as an empty cell. Partial: the walk read the use of the matrix, not a measured coverage map. - red-queen (partial)
Public rules are improved by a large community; adversaries test attacks against them and modify them to evade; defenders add rules or adaptive detection; Uetz et al. find evasions 'an intrinsic problem' and fixing every rule 'practically impossible'. Both sides keep moving; whether either gains gro - same shape, opposite sign (partial)
One matching step run against reference sets of opposite sign: misuse detection matches events against known-malicious rules; the adaptive detector of Uetz et al. compares each event to the rules on one hand and to known-benign events on the other and decides which it is closer to. Partial: the lear - same toolkit, independent builds (partial)
Many vendors (22 in the 2020 vendor map) and open-source projects build the same blocks (collection, parsing normalisation, rule engine, storage, monitoring) into separate products. Partial: shared lineage through syslog, Gartner's category and acquisitions may make some builds less independent than - The carry (partial)
Overflow passed on: the live store holds about 90 days (raw events about 6 months, about 6 TB); the excess is archived out by scripts, and archived data is not used by the SIEM again, though some threats are found months later. The wait: a threshold rule cannot fire until its window fills (3 failed - The Payment to Entropy (partial)
Signal form: from events to alerts to alarms to true attacks the yield falls (0.01% of alerts tied to true attacks at one SOC), and agentless collection pays extra network transfer for the same data. Most of the drop is filtering by design (benign triggers, failed attempts, unneeded fields), the dar - nested ladder (surface)
Event, alert, alarm, incident: each rung built from those below (tools turn events into alerts, the SIEM turns correlated alerts into alarms, analysts escalate alarms into incidents). Surface: this is a ladder of aggregation, not one where each rung implies the ones below.
Valence: what it trades, and with whom (19) published science
Each exchange with a neighbour, read from both sides: who gains, who pays, and what carries it.
- Intrusion detection system via sys:written-record · Security information and event management (SIEM) gain
- Security operations center (SOC) via sys:written-record · Security information and event management (SIEM) gain
- Insider threat via sys:computer-network · Security information and event management (SIEM) gain
- Human (Homo sapiens) via sys:electromagnetic-field · Security information and event management (SIEM) gain
- Law (rules and law) via sys:human-society · Security information and event management (SIEM) mixed
- Large language model via sys:written-record · Security information and event management (SIEM) gain
- Operating system via sys:written-record · Security information and event management (SIEM) gain
- Malware via sys:written-record · Security information and event management (SIEM) loss
- Incident response via sys:written-record · Security information and event management (SIEM) even
- Threat actor via sys:written-record · Security information and event management (SIEM) mixed
- Organization via sys:written-record · Security information and event management (SIEM) gain
- Firewall (computing) via sys:written-record · Security information and event management (SIEM) gain
- Database via sys:written-record · Security information and event management (SIEM) gain
- Computer network via sys:electromagnetic-field · Security information and event management (SIEM) gain
- Cyber threat intelligence via sys:written-record · Security information and event management (SIEM) gain
- Outsourced incident response (MSSP, contractors) via sys:human-society · Security information and event management (SIEM) gain
- SOAR and automation via sys:written-record · Security information and event management (SIEM) gain
- Network Time Protocol servers via sys:time · Security information and event management (SIEM) gain
- Vulnerability management via sys:organization · Security information and event management (SIEM) gain
Partners and parts published science
Built from / builds (11)
- Security operations center (SOC)
- Database
- Collectors, aggregators and agents
- Parsers and normalisation
- Correlation and rule engine, prioritisation module
- Rules and use cases
- Consoles, dashboards, search node and reporting
- Security information management (SIM) functions
- Security event management (SEM) functions
- Detection rules against evasion
- Concealment against the log record
Sits inside / holds (3)
expressed by (27)
- Generation/Synthesis
- Recombination/Composition
- Scaffolding/Enablement (Meta-Tooling)
- Templating / Standardization
- Quality Improvement/Refinement
- Interoperability/Transduction (Bridging)
- Encapsulation & Modularity (Abstraction)
- Distributed Autonomy/Decentralized Execution (Subsidiarity)
- Internal Coordination/Regulation
- Sensing/Scanning
- Hypothesis/Goal Selection
- Mapping/Modeling
- Uncertainty Reduction/Learning
- Multi-Scale Data Fusion / Triangulation
- Trail-Marking / Externalized Memory (Stigmergy)
- Surprise / Change-Point Detection
- Social Learning / Public Information Use
- Reactive Response
- Proactive Defense
- Physical/Virtual Barrier
- Technique-Based Identification (a.k.a. Technique-Based Detection)
- Self-Monitoring
- Validation of Effectiveness
- Resilience & Repair
- Redundancy & Diversity
- Triage & Prioritized Allocation
- Quality Control & Error Correction
floor filled by (21)
- Intrusion detection system
- Security operations center (SOC)
- Incident response
- Insider threat
- Human (Homo sapiens)
- Law (rules and law)
- Operating system
- Threat actor
- Organization
- Firewall (computing)
- Database
- Computer network
- UEBA and network traffic analysis
- Antivirus / anti-malware software
- Collectors, aggregators and agents
- Parsers and normalisation
- Correlation and rule engine, prioritisation module
- Rules and use cases
- Consoles, dashboards, search node and reporting
- Network Time Protocol servers
- Cyber threat intelligence
Kinds (4)
- Open-source SIEM (Wazuh, OSSIM)
- SIEM as a managed service or as a service (MSSP, cloud)
- Lighter domain-specific SIEMs in a hierarchy (IoT gateways, 5G slices)
- SIEM for critical infrastructures (enhanced SIEM with multi-layer collectors, decision support and resilient storage)
wired (6)
- Intrusion detection system
- Management servers, database servers and consoles
- Human (Homo sapiens)
- Black hole router (automatic blocking)
- Tier 1 analyst (triage specialist)
- Threat intelligence platform (TIP)
Notes published science
- A SIEM's rulebook is public on purpose, and that cuts both ways: of 292 widely used open Sigma rules, 110 (38%) could be fully dodged with tricks as small as an extra quote mark or a hyphen in place of a slash, because attackers can test their moves against the same rules defenders download (src:uetz-2023-siem-evasion).
- A SIEM has to fix time before it can find anything: every computer keeps its own clock, often in a different time zone and out of step, so the SIEM must add or subtract time from each source's entries and hold them to one clock before it can say which event came first (src:nist-sp800-92).
- The same job turns up in a hospital and a phone network: telecom engineers call it 'alarm correlation', 'a conceptual interpretation of multiple alarms such that new meanings are assigned', and intensive-care monitors face the SIEM's own problem, with 72% to 99% of alarms false or not actionable (src:zheng-2002-alarm-correlation, src:chromik-2022-icu-alarm-fatigue).
Layers
The ways it takes in its world, where they meet, and how it acts back, each from L1 (where it meets the world) up.
the SIEM as security engineers and analysts describe it: the monitored estate's activity up to the use cases it serves (reader's view)
- L1: The monitored estate and its activity: hosts, applications, network hardware, users, physical plant and OT, and the attackers acting on them
- L2: Source devices and sensors that write logs and alerts (IDS, antivirus, firewalls, OSs, applications, badge readers)
- L3: Log collection: collectors and aggregators, agents or agentless push and pull
- L4: Parsing and normalisation into one event format, with filtering, aggregation and reduction
- L5: Storage and index: database servers hold the events for search and correlation over time
- L6: Correlation, alerting and prioritisation: rules link events into alarms
- L7: Use cases served: detection, compliance reporting, forensics and hunting, through dashboards, reports and responses, set by the people who run it
the boardroom (shared floors)
- L5: Storage and index: one or more database servers hold the normalised events (live for about 90 days, raw events about 6 months), searchable across sources and times
- L6: Correlation, alerting and prioritisation: directive alarms (correlation rules) link related events from many sources, enriched with asset and threat context, into prioritised alarms
- L7: Use cases and the monitoring programme: what the SIEM is for (detection, compliance reporting, insider threat, threat hunting, forensics), which sources and rules serve it, how long data is kept; set and tuned by the people who run it (SIEM engineers, content authors, log management administrators) with the analysts' verdicts
way in: Alerts from security tools (IDS/IPS, antivirus, firewalls)
- L1: Alert records the security tools write when they detect a threat
- L2: The sensors: intrusion detection systems, anti-virus, firewalls
- L3: Collector plug-ins: SDEE-enabled devices keep logs that the SIEM retrieves; others send logs directly
- L4: Parsed into a security event when the raw log matches a criterion
way in: Analysts' searches and hunts
- L1: An analyst's question over many nodes and periods
- L2: The search node or console where queries are typed
- L3: Written in a search processing language
- L4: Fast queries driven by the analyst's own reasoning
way in: Analysts' verdicts and tuning requests on alarms
- L1: Each alarm judged false positive or escalated, from knowledge of the environment and tacit experience
- L2: Tracked as incidents and tickets in the SIEM's workflow
- L3: Turned into tuning: false positives corrected one by one
way in: Asset, vulnerability and identity context (inventories, scan results, directories)
- L1: Records of the organisation's assets, users and vulnerabilities
- L2: Vulnerability scans and asset discovery
- L4: Asset information stored for correlation
- L5: Held in the knowledge base
way in: Detection rules and use cases (expert-written, community Sigma rules)
- L1: Rules that define known malicious behaviour: signatures, title, description, references
- L2: Authors: security experts in the SOC and a large open community (the Sigma repository)
- L3: Converted from the generic format into the SIEM's own query language
- L4: Use cases configured into the SIEM to alarm on chosen scenarios
- L5: Stored in the SIEM's rule database
way in: Endpoint behaviour events (process creations, network connections, file changes)
- L1: Low-level events on endpoints: process creations, network connections, file manipulations
- L2: Behaviour collected from endpoints because encryption and obfuscation evade traditional detection
- L3: Endpoint agents (e.g. Wazuh agents) send events to the SIEM
- L4: Events captured by rule matching and logged in JSON
way in: Laws, regulations and control frameworks (HIPAA, PCI DSS, FISMA, NIST SP 800-53, EO 14028)
- L1: Regulations that require logging and reporting
- L2: Control frameworks that say what to audit: NIST SP 800-53 AU-2 event monitoring, SI-4 system monitoring
- L3: The organisation's logging requirements and goals, prioritised by risk
way in: Network traffic seen by network monitors (flows, Zeek logs)
- L1: Traffic on the networks
- L2: Network monitoring tool (Zeek) and a customised rule engine over network and system logs
- L3: Network-level alerts carried to the SIEM
- L4: Each alert characterised by timestamp, IPs, category and metadata
way in: Operating system, application and network device logs
- L1: Log entries: records of events on systems and networks (syslog, Windows Event Log)
- L2: Log generators: OSs, security software, application servers, web and e-mail servers
- L3: Agents on the hosts, or agentless push and pull with credentials
- L4: Parsing and normalisation: the SIEM knows the important fields of each source type; free text is reduced to templates
way in: Physical-access and operational-technology events (badge readers, industrial plant probes)
- L1: Physical plant: doors, badge readers; production and manufacturing OT environments; a hydroelectric dam
- L2: Physical security devices such as badge readers; probes on industrial processes
- L3: A security collector (GET) monitors the plant's events
- L4: Translated into a common format for the correlator
way in: The SIEM's own state (sources logging, transfers failing, storage, malfunctions)
- L1: The infrastructure's own condition: failures in logging software or archival media, sources that stop sending
- L2: Logging status of every source checked
way in: Threat intelligence and open-source intelligence
- L1: Threat information published and exchanged; open-source intelligence (OSINT)
- L2: Cyber threat intelligence exchange platforms
- L3: Custom connectors and APIs bring the data in
- L4: Contextual data from threat intelligence platforms fed into the SIEM
- L5: Held in the security knowledge base the analysts customise
way in: Time: the clocks of every log source
- L1: Each log source's clock; systems in many time zones; clocks out of sync
- L2: NTP servers keep log sources' clocks consistent
- L3: Administrators make sure each logging host's clock is synched to a common source
- L4: Logged times normalised to a single time zone
way out: Alarm to the analysts' console and dashboards
- L1: Light from the screens to the analyst's eyes
- L2: Shown through dashboards and visualisations
- L6: Rule triggered: alarm fired and prioritised
way out: Archive, reduce and dispose of old data
- L1: Old entries cleared
- L5: Archived out of the live system, by custom scripts to attached storage, HDFS or cloud
- L7: Retention set by policy and law
way out: Automated reaction (scripts, tickets, e-mails, firewall changes)
- L1: The unauthorised data path closed on the network
- L3: Reaction system acts on the security controls
- L6: Directive triggers an action
way out: Compliance and audit reports
- L1: Reports handed to compliance audits
- L2: Reports generated automatically
- L7: Reports chosen to meet regulatory frameworks
way out: Reconfigure agents and log sources
- L2: The log sources' settings changed
- L3: Server reconfigures the agent; forwarding filtered at the source
- L7: Changes to logging decided (what entries and fields are sent, which format)
way out: Search results and forensic answers
- L1: Read by the analyst
- L2: Shown on the search node
- L5: Matching events retrieved across nodes and periods