← all systems · see it in The System
Security audit and compliance
The checking of an organisation's or product's security against standards, laws and contracts by a party that did not build it: an independent review and examination of system records, activities and related documents (src:wp:information-security-audit), done by examining, interviewing and testing (src:nist-sp800-53a-r5), reported as findings, and turned into a decision others rely on: a certificate (ISO/IEC 27001, Common Criteria), an attestation report (SOC 1/2/3), an authorization to operate (src:nist-sp800-37-r2), a compliance validation (PCI DSS) or a regulator's fine (GDPR). Compliance is the goal it checks for: 'conforming to a rule, such as a specification, policy, standard or law' (src:wp:regulatory-compliance). Walked as auditors, assessors, certification schemes and regulators describe it, with the research on whether it works kept beside it.
Properties published scienceour proposal
Sourced fields, and our own readings.
| born at | contested (see C-1775) |
| born precision | contested |
| boundary | Inside: the auditors and assessors (internal auditors, external audit firms, certification bodies and their audit teams, evaluation laboratories, approved scanning vendors, inspectors of regulators), their methods (examine, interview, test; sampling; depth and coverage), the criteria they check against once selected (scope, objectives, determination statements), their findings and reports, and the decision bodies that turn findings into certificates, attestations, authorizations or penalties. Outside: the audited organisation and its people (each a person with their own channels), the systems and records examined, the standards bodies and legislatures that write the criteria, accreditation bodies that check the checkers, and the relying parties (customers, acquirers, card brands, regulators, the public) who act on the result. |
| nested level | A practice inside the oversight floor of governance: part of the monitor-and-evaluate loop of security policy and governance (its audit channel), and of the audit-review step of information security's monitoring. Law, standards and contracts shape it from outside; certification schemes and regulators run it at the scale of whole markets. |
| scale | social |
Measures (23) published science
| characteristic clock: ISO/IEC 27001 surveillance audit | at least 1 per calendar year (first within 12 months) |
| characteristic clock: certification cycle (ISO/IEC 27001; UN R155 CSMS) | 3 years |
| clock: PCI DSS self-assessment questionnaire and external scan | 4 per year (quarterly, all merchant levels) |
| Common Criteria certificates crawled | 1631 active, 3725 archived certificates |
| time and cost to obtain a product certificate (CC, FIPS 140) | months or even years; hundreds of thousands of dollars or more time; USD |
| CC certificates with at least one known vulnerability | 88 of 1631 active; 274 of 3725 archived certificates |
| correlation of EAL with average CVE base score | -0.28 Spearman rho (no positive association found) |
| delay of one fix after certification | 3 years (maintenance update fixing 3 CVEs disclosed before certification) |
| PCI scanners fully compliant with ASV guidelines | 0 of 6 scanners |
| PCI scanner prices | $250 to $2,995 USD per year |
| e-commerce sites with at least one PCI DSS violation | 94 % of 1,203 |
| e-commerce sites with a violation that should disqualify them | 86 % of 1,203 |
| per-hand-off efficiency, merchant compliance claim re-checked independently (Payment to Entropy) | about 0.14 share of scanned sites whose compliance survives (derived: 100% - 86%) |
| organisations not compliant at interim validation (Verizon 2018) | 47.5 % |
| Heartland breach (validated compliant) | more than one hundred million card numbers |
| sum of GDPR fines analysed | about 1,839 million (max 746 million; mean 2,149,000) EUR |
| GDPR fines analysed | 856 fines |
| GDPR fines rooted in organisational vs technical problems | 601 vs 314 fines |
| ISO/IEC 27001 studies citing outcomes of certification | 26 (half with empirical evidence) % of reviewed studies |
| ISO/IEC 27001:2022 Annex A controls | 93 controls in four themes |
| reduction in incident probability from adopting six basic precautions (riskiest Israeli firms) | 80 to 42 % probability |
| CISOs and executives interviewed on investment drivers | 40 people |
| one ISO certification practice in its first year | over 1 million USD revenue; more than 20 certifications USD; certifications |
Open contentions (5) open contention
Where sources or readings disagree. Both stay on the map.
- C-1697 Does security certification make organisations more secure?
booked: The map counts certification as a route to assurance and leaves its effect on security open, since few studies have measured it.
the other reading: certification provides independent assurance of conformity (schemes, accreditation, labs); few studies (26%) cite outcomes of ISO/IEC 27001 certification, half with empirical evidence (Culot et al. 2021); EAL correlates only with average CVE severity (rho -0.28), never positively; 88 of 1631 active CC certificates carry a known vulnerability (Janovsky et al. 2024); no tested PCI scanner fully compliant; 86% of 1,203 sites disqualifiable; Heartland, Hannaford and Target breached while validated (Rahaman et al. 2019; encyclopedia) - C-1712 Do penalties or shared values make staff follow security rules?
booked: The map books both readings.
the other reading: detection and punishment deter violators (audit games, after Becker 1968); rewards and fines can crowd out intrinsic motivation and undermine compliance (motivation research via the encyclopedia) - C-1736 Can an auditor paid by the company it audits stay independent?
booked: The map books independence as a condition the auditor's decision is judged by, and leaves open whether paid arrangements keep it.
the other reading: independence required and judged (AICPA for SOC, IIA standards, the authorizing official under NIST); internal auditors are employees of the audited; rigorous scanners might lose customers to laxer ones (Rahaman et al. 2019); vendors in a conflict of interest about why certificates end (Janovsky et al. 2024) - C-1746 Does following security rules actually make organisations safer?
booked: The map keeps both: mandated controls cut incidents, and compliance can drift into box-ticking.
the other reading: controls adopted under a government directive reduce incident probability (80% to 42% for the riskiest firms; Gandal et al. 2022, instrumented); 'Good compliance does not equal good security' (CISO in Moore et al. 2015); invoking compliance is the most reliable way to get funding; audit attends to easily auditable risks and breeds box-ticking (Power, via LSE) - C-1775 When did security auditing begin?
booked: The map leaves the start contested, from ancient account checks to 1960s computer audits and 1990s certification.
the other reading: auditing as a safeguard since ancient times; medieval hearing of accounts; IT (EDP) audit formed in the mid-1960s; information security management certification from BS 7799 (1995) and product certification from the Common Criteria arrangement (1998)
Partners and parts published science
Built from / builds (8)
- Auditors and assessors (internal, external, consultants, regulators' staff)
- Assessment methods: examine, interview, test
- Audit criteria: standards, laws, contracts, protection profiles
- Certification bodies, evaluation labs and attestation firms
- Decision officials (authorizing official, certification decision maker)
- Regulators and enforcement authorities (data protection authorities, card brands' programmes, PCAOB)
- Auditors against violators over detection
- Compliance-driven against risk-driven security spending
Sits inside / holds (3)
Trades with (valence) (15)
- Formal security policy model via sys:security-audit · Security audit and compliance gain
- Human (Homo sapiens) via sys:human · Security audit and compliance gain
- Government via sys:law · Security audit and compliance gain
- Law (rules and law) via sys:law · Security audit and compliance gain
- Insurance via sys:organization · Security audit and compliance gain
- Security policy and governance via sys:organization · Security audit and compliance gain
- Risk management via sys:organization · Security audit and compliance gain
- Organization via sys:organization · Security audit and compliance gain
- Transaction log via sys:audit-log · Security audit and compliance gain
- Security standard via sys:security-standard · Security audit and compliance gain
- Penetration testing via sys:information-system · Security audit and compliance gain
- Computer law via sys:organization · Security audit and compliance gain
- Certification bodies, evaluation labs and attestation firms via sys:organization · Security audit and compliance gain
- Regulators and enforcement authorities (data protection authorities, card brands' programmes, PCAOB) via sys:law · Security audit and compliance gain
- Accreditation bodies and auditor overseers (PCAOB, PCI SSC) via sys:organization · Security audit and compliance mixed
expressed by (14)
- Templating / Standardization
- Sensing/Scanning
- Hypothesis/Goal Selection
- Uncertainty Reduction/Learning
- Active Experimentation / Probing (Optimal Experimental Design)
- Multi-Scale Data Fusion / Triangulation
- Trail-Marking / Externalized Memory (Stigmergy)
- Social Learning / Public Information Use
- Technique-Based Identification (a.k.a. Technique-Based Detection)
- Self-Monitoring
- Validation of Effectiveness
- Triage & Prioritized Allocation
- Deterrence
- Quality Control & Error Correction
floor filled by (12)
- Formal security policy model
- Human (Homo sapiens)
- Law (rules and law)
- Security policy and governance
- Organization
- Transaction log
- Security standard
- Penetration testing
- Auditors and assessors (internal, external, consultants, regulators' staff)
- Certification bodies, evaluation labs and attestation firms
- Decision officials (authorizing official, certification decision maker)
- Regulators and enforcement authorities (data protection authorities, card brands' programmes, PCAOB)
Kinds (14)
- Penetration testing
- Internal audit
- External (third-party) audit
- Regulatory inspection and enforcement
- First-party (self) assessment
- Second-party (user or customer) assessment
- Third-party (independent) assessment
- ISMS certification (ISO/IEC 27001)
- SOC attestation reports (SOC 1, SOC 2, SOC 3; type 1 and type 2)
- Product evaluation and certification (Common Criteria, FIPS 140, EMVCo)
- Authorization to operate (RMF, FedRAMP)
- Payment card compliance validation (PCI DSS: QSA, ISA, ASV, SAQ)
- Cybersecurity type approval (UN R155 CSMS)
- Accreditation (checking the checkers)
wired (2)
Notes published science
- The word audit comes from Latin audire, 'to hear': in medieval Britain auditors literally listened to the accounts being read out to them, and the interview is still one of the three ways auditors take in evidence today (src:wp:audit, src:nist-sp800-53a-r5).
- Researchers built a fake online shop with 35 planted security holes and ran six PCI scanning services on it: none followed the scanning rules fully, all six issued certificates anyway, and the approved vendors caught no SQL injection at all. Scanning 1,203 real shops, they found 86% had a problem that should have failed them (src:rahaman-2019-pci).
- Across 25 years of Common Criteria certificates, a higher assurance level was not linked to fewer known vulnerabilities, only to somewhat less severe ones (rho -0.28), and the authors found no positive link at all. One product fixed three vulnerabilities, already public before it was certified, in an update three years after certification (src:janovsky-2024-sec-certs).
Layers
The ways it takes in its world, where they meet, and how it acts back, each from L1 (where it meets the world) up.
security audit and compliance as auditors, assessors and regulators describe it: the audited organisation's systems, people and records, up to the assurance others rely on (reader's view)
- L1: The audited organisation's systems, people, records, facilities and products: technical, physical and administrative controls
- L2: Evidence taken in by three methods: examine (documents, records, configurations), interview (people), test (mechanisms exercised, scans, penetration tests); plus evidence reused from earlier assessments and operations
- L3: Criteria, scope and plan: preliminary assessment of time, cost and scope; audit objectives; criteria from standards, laws and contracts; depth and coverage (basic, focused, comprehensive) chosen to the assurance required
- L4: Evidence weighed against determination statements: each yields 'satisfied' or 'other than satisfied'; sampling and point-in-time readings
- L5: Findings reported: exit meeting with the audited, the report (procedures, findings, recommendations, rankings high/medium/low), corrective actions and plans of action and milestones
- L6: The decision: a certification body's certification decision, an authorizing official's authorization to operate, an attestation opinion, a validation of compliance, or a regulator's fine
- L7: Assurance relied on by others: customers, acquirers, card brands, procurers, regulators and the public act on the certificate or report instead of checking for themselves; accountability for claims
the boardroom (shared floors)
- L4: Evidence weighed against criteria: each determination statement yields 'satisfied' or 'other than satisfied' (which can also mean the assessor could not obtain enough information); sampling at a point in time
- L5: Findings reported: exit meeting, report dated at completion, recommendations ranked, corrective action plans; the audited answers
- L6: Decision: certify or not, authorise to operate (accepting residual risk), issue the attestation opinion, validate compliance, or fine; the decision maker also judges whether assessor independence suffices
- L7: Assurance and accountability: relying parties (customers, acquirers, card brands, procurers, the public) act on the certificate or report; certification confers market access or liability shields; failed claims bring penalties
way in: Audit trails and operational evidence (logs, incident records, continuous monitoring)
- L1: Audit trails: chronological records written by a privileged process that users cannot stop or change
- L2: Operational evidence collected: remediation records, incident reports, continuous monitoring results
- L3: Read at organisation-defined frequencies for ongoing assessment
way in: Complaints and breach reports (the regulator's way in)
- L1: Data subjects, employees and other companies complain; breaches become public
- L2: The data protection authority receives and opens an investigation
- L3: The issue categorised (organisational or technical; which GDPR article)
way in: Earlier assessments and certificates reused (product certificates, third-party evaluations, previous reports)
- L1: Prior assessment results, validation reports and certificates for components
- L2: Accepted when owners and authorizing officials agree; reciprocal acceptance across agencies
- L3: References between certificates (composite and updated products) carry earlier findings forward, including vulnerabilities
way in: Examine: documents, policies, records, configurations
- L1: System records, activities and related documents; policies and procedures documented at the site
- L2: The examine method: reviewing, inspecting, observing, studying or analysing assessment objects
- L3: Depth and coverage of examination set to the assurance required
way in: Interview: hearing the people (the audit as hearing, audire)
- L1: The organisation's people and what they say about how the work is done
- L2: The interview method: discussions with individuals or groups to understand, clarify or obtain evidence
- L3: What is heard is checked against records and observation (meet IT management, review job descriptions)
way in: Laws and regulations that mandate audits or set the rules checked (HIPAA, SOX, GDPR, UN R155, state laws)
- L1: Statutes and regulations: in certain regulated industries audits are mandated by law
- L2: The organisation and its auditors become aware of the requirement (regulatory compliance as the goal of being aware and taking steps)
- L3: The requirement turned into audit criteria and scope (e.g. SOX section 404 internal control; UN R155 CSMS requirements)
way in: Observe: the auditor's site visit (sight)
- L1: Premises and equipment: locked cages, man traps, single entrances, environmental controls
- L2: Travelling to the site and observing processes
- L3: Equipment inspected and checked against utilization reports and downtime records
way in: Standards, control catalogues and contracts (ISO/IEC 27001 Annex A, PCI DSS, Trust Services Criteria, protection profiles)
- L1: Published standards and contracts: PCI DSS v1.0 (2004) by the card brands; ISO/IEC 27001 (2005, 2013, 2022); protection profiles for classes of product
- L2: Selected for the engagement: Security Target with chosen SFRs and SARs; Statement of Applicability against Annex A
- L3: Turned into determination statements and assessment procedures tailored to the system
way in: Test: mechanisms exercised (scans, penetration tests, configuration checks)
- L1: The live systems and products: networks, servers, web applications, certified devices
- L2: Tests run: approved scanning vendors' external scans (quarterly for PCI), lab evaluation of products, penetration testing
- L3: The non-intrusive, low-interaction constraint limits what can be tested on live production sites
way in: The auditor's own judgement and independence (inner channel)
- L1: The auditor as a person with interests: paid by the audited, competing for clients
- L2: Independence held by professional standards (IIA, AICPA) and judged by the authorizing official
way in: The engagement request (client, acquirer, procurer or authorizing official asks for the audit)
- L1: A relying party requires a report: a merchant must submit its compliance report to its acquirer; CC certification specified in procurement
- L2: Preliminary assessment of maturity, time, cost and scope
- L3: Planning and preparation: objectives, manpower, timeline; Stage 1 readiness review agrees the audit plan
way out: Certificate, attestation report or authorization issued to relying parties
- L1: Relying parties keep accounts active, buy, procure or admit to market (a merchant keeps its account with the acquirer)
- L3: Report written for its audience: SOC 1/2 for a limited audience, SOC 3 for the public; review reports state 'limited assurance'
- L6: Certification decision or authorization to operate
way out: Certification refused, withdrawn or left to expire
- L1: The product or organisation loses the standing the certificate gave (procurement, market access)
- L6: The scheme ends a certificate's validity (archived certificates; recertification needed before expiry)
way out: Findings and corrective actions back to the audited
- L1: Controls changed in the audited systems
- L3: Owners decide: remediate, or accept a finding that presents no significant risk; plans of action and milestones
- L5: Findings and suggested corrective actions communicated in a formal meeting
way out: Fines and penalties
- L1: Organisations adjust how they handle data, organisationally and technically
- L3: Fines published (enforcement trackers) and passed along contracts
- L6: A fine imposed: by a data protection authority (GDPR), or by card brands to the acquirer who passes it to the merchant