← all systems · see it in The System
Penetration testing
An authorised simulated cyberattack on a computer system, performed live to evaluate its security (src:wp:penetration-test); in NIST's words, security testing in which assessors mimic real-world attacks to identify methods for circumventing the security features of an application, system, or network (src:nist-sp800-115). Walked as testers, their standards and researchers describe it: the engagement (scope, rules of engagement, timebox), reconnaissance and scanning, vulnerability analysis, exploitation and pivoting, reporting and re-testing; its kinds (vulnerability assessment, application and network tests, internal 'assumed breach' tests, OT tests, red teaming, bug bounties, automated and LLM-driven testing); its ethics and law; the people who do it.
Properties published scienceour proposal
Sourced fields, and our own readings.
| born at | 1965 (first formal request at the SDC conference); tiger teams in the 1970s |
| born precision | year |
| boundary | Inside: the authorised engagement (scope and rules of engagement, the testers and their tools, reconnaissance, scanning, exploitation, pivoting, logs, the report and the re-test), and the standing practice (methodologies, checklists, training, the profession and its ethics). Outside: the client organisation and its systems and people (the target), the defenders it tests (blue team, SOC), real attackers it emulates, the vulnerability management that acts on its findings, vendors, law and regulators. |
| nested level | A practice inside computer and information security, a component of a full security audit (src:wp:penetration-test); one technique among NIST's target vulnerability validation techniques (src:nist-sp800-115); run by people (white-hat hackers) on an organisation's request. |
| scale | social |
Measures (21) published science
| characteristic clock: penetration test cycle (NIST: annual may suffice) | 1 year |
| mandated minimum test frequency (FISMA) | no less than annually per year |
| engagement effort: penetration test | 5-10 (1-2 testers) days |
| engagement effort: vulnerability assessment | 2-4 (1 tester) days |
| engagement effort: internal network and OT tests | 7-10 (1-2 testers) days |
| engagement effort: red teaming | 30+ (3-4 testers) days |
| tester attention span on one web test | 3-4 days before motivation degrades (one tester) |
| interviewed testers' work experience | mean 9.0, sd 6.5, median 8 (n = 12) years |
| time for injection attacks to become rare (testers' report) | about 10 years |
| cybersecurity workforce growth vs gap growth | 11.1% vs 26.2% per year (ISC2 2022, as cited) |
| PentestGPT sub-task completion gain | 228.6% over GPT-3.5; 58.6% over GPT-4 percent |
| PentestGPT benchmark | 13 targets, 182 sub-tasks count |
| PentestGPT real-world machines solved | 4 of 10 HackTheBox machines for 131.5 USD of API use count |
| PentestGPT CTF placing | 24th of 248 teams (1500 of 4200 points in one passage, 1400 points in another) rank |
| bug bounty discovery decay exponent | -0.40 (p < 0.001, R2 = 0.79; 35 HackerOne programs) power-law exponent of awards over time |
| bug bounty programs on the two largest platforms | about 3,300 programs |
| Forbes 500 without a disclosure program | 80 percent |
| bug bounty contributors studied | 2,504 across 82 programs count |
| AI red-team attacks released | 38,961 attacks |
| Hobbs's 1851 lock-pick payment | 20,000 (adjusted for inflation) USD |
| interviewed testers who named intuition when choosing where to look | 8 of 12 testers |
Open contentions (6) open contention
Where sources or readings disagree. Both stay on the map.
- C-1667 When did penetration testing begin?
booked: The map dates the first formal request to 1965 and keeps earlier lines (Cold War red teams, Victorian lock challenges) alongside.
the other reading: 1965: first formal requests to use computer penetration to study security (encyclopedia, after Hunt 2012); early 1960s: red teaming in RAND's Cold War simulations; 1851: Hobbs paid to pick a lock (bounty-like) - C-1668 How much assurance does a penetration test give?
booked: The map keeps both: a required method for assurance, and the warning that a test can show holes but never prove there are none.
the other reading: a method for gaining assurance (NCSC); demanded by FISMA and PCI DSS; cannot prove absence; used as an excuse to declare victory (Arkin, via Pierce); red-teaming as a panacea verges on security theater (Feffer, for AI) - C-1669 Should testers get inside knowledge before testing?
booked: The map keeps both: practitioners favour full knowledge, while ethicists recommend testing blind from outside first.
the other reading: practitioners overwhelmingly recommend moving toward white-box (time, cost, coverage); ethics literature: black-box offsite first, against insiders' tunnel vision - C-1670 Should staff be told a security test is happening?
booked: The map keeps both: covert tests read everyday security better, overt tests cost less and teach staff.
the other reading: covert tests read everyday security better (NIST); only key personnel told (Pierce); overt tests cost less, carry less risk, teach staff and are used more (NIST) - C-1671 Where does a vulnerability scan end and a penetration test begin?
booked: The map books breadth versus depth as the usual line and notes that practitioners draw it differently.
the other reading: vulnerability assessments for breadth, pen-tests for depth; the demarcation varied between interviewees; scans are a first step of pen-tests - C-1672 Is a penetration tester's intuition learned or innate?
booked: The map lists both views, with agreement that practice sharpens it.
the other reading: learned: 'nobody is born a super hacker'; built from tests, CTFs and former development work; innate: the best testers showed hacking-style behaviour in kindergarten (one interviewee)
Partners and parts published science
Built from / builds (7)
- White hat
- Rules of engagement (scope, permission, contacts)
- Testing tools (Nmap, Burp Suite, Metasploit, BloodHound, sqlmap; Kali Linux)
- Methodologies and checklists (PTES, OSSTMM, NIST SP 800-115, ISSAF, OWASP Testing Guide)
- The report
- Competition for undetected progress (red team vs blue team)
- Race to find the client's flaws (tester vs real attacker)
Sits inside / holds (3)
Trades with (valence) (14)
- Social engineering via sys:organization · Penetration testing gain
- Human (Homo sapiens) via sys:human · Penetration testing gain
- Law (rules and law) via sys:law · Penetration testing gain
- Large language model via sys:software · Penetration testing gain
- Security operations center (SOC) via sys:computer-network · Penetration testing gain
- Threat actor via sys:internet · Penetration testing gain
- Organization via sys:organization · Penetration testing gain
- Software vulnerability via sys:software · Penetration testing gain
- Physical security via sys:physical-security · Penetration testing gain
- Bug bounty (vulnerability rewards) programs via sys:internet · Penetration testing gain
- White hat via sys:human · Penetration testing gain
- Vulnerability management via sys:organization · Penetration testing gain
- Computer ethics via sys:law · Penetration testing gain
- Security audit and compliance via sys:information-system · Penetration testing gain
expressed by (20)
- Templating / Standardization
- Symbiosis/Co-Creation (Collaborative Specialization)
- Sensing/Scanning
- Hypothesis/Goal Selection
- Search/Foraging
- Mapping/Modeling
- Uncertainty Reduction/Learning
- Risk-managed Exploration
- Active Experimentation / Probing (Optimal Experimental Design)
- Trail-Marking / Externalized Memory (Stigmergy)
- Optimal Stopping & Patch-Switching
- Offline Simulation & Replay-Guided Planning
- Social Learning / Public Information Use
- Evade Detection
- Overcome Barrier
- Subversion & Co-optation
- Prepositioning & Latency
- Validation of Effectiveness
- Triage & Prioritized Allocation
- Quality Control & Error Correction
floor filled by (12)
- Human (Homo sapiens)
- Large language model
- Computer security
- Organization
- Computer network
- Software vulnerability
- White hat
- Vulnerability scanner
- Rules of engagement (scope, permission, contacts)
- Testing tools (Nmap, Burp Suite, Metasploit, BloodHound, sqlmap; Kali Linux)
- The report
- Security audit and compliance
Kinds (21)
- Bug bounty (vulnerability rewards) programs
- Social engineering
- Security audit and compliance
- Vulnerability assessment
- Internal network test (assumed breach)
- OT / ICS test
- Red teaming
- Purple teaming
- Physical red teaming
- Automated and LLM-driven penetration testing (PentestGPT, AI sparring partners)
- AI red teaming
- Capture the flag
- Black-box test
- Gray-box test
- White-box test
- Covert (black hat) testing
- Overt (white hat) testing
- Web application test
- Wireless test
- Remediation verification (re-test)
- Continuous DDoS testing
Notes published science
- Penetration testers have a 'spidey sense' and say so: 8 of 12 professionals interviewed named intuition when choosing where to attack, one putting it as 'some organ that tingles if an operation looks fishy'. Whether that sense is learned or innate split the group: 'nobody is born a super hacker', but one said the best testers they knew showed hacking-style behaviour already in kindergarten (src:happe-cito-2023-hackers).
- A penetration test can only prove that holes exist, never that there are none, which makes it 'a science of insecurity'. And it can be spoiled by telling people: if word of a test spreads, staff behave differently and the result goes wrong, which is why covert tests read everyday security better but cost more (src:pierce-2006-ethics, src:nist-sp800-115).
- Hatcheries run penetration tests on fish: young salmon smell pike paired with alarm cues from injured salmon before release, and in one of two experiments the trained fish survived real pike better. Vaccines do the same job for the immune system with a weakened or killed attacker, after centuries of variolation with real smallpox material (src:klemme-2025-salmon, src:iwasaki-omer-2020-vaccines).
Layers
The ways it takes in its world, where they meet, and how it acts back, each from L1 (where it meets the world) up.
penetration testing as testers and their standards describe it: the client's systems, people and premises in scope, up to the assurance the test gives (reader's view)
- L1: The target in scope: the client's systems, applications, people and premises, with their weaknesses
- L2: Reconnaissance and scanning: information gathering, port and service identification
- L3: Vulnerability analysis: scanned hosts compared with vulnerability databases and the testers' own knowledge; hypothesised flaws ranked
- L4: The attack: exploitation to confirm, chaining and pivoting, looping back to discovery
- L5: The engagement: scope, rules of engagement, approval, timebox, stop points, notification and escalation
- L6: The report and risk rating; re-test; the testers' quality control
- L7: Assurance: what the client learns of its security posture, bounded by presence-not-absence; the profession's integrity
the boardroom (shared floors)
- L4: The attack: exploitation to confirm vulnerabilities, chaining flaws and pivoting, attack choice (most cost-effective for an attacker first), mental models and the red team's map of the company
- L5: The engagement: scope and rules of engagement, management approval, timebox, stop points, coordination with the client, escalation of detected test activity (the white team)
- L6: Reporting and risk rating: logs and periodic reports during the test, the final report (vulnerabilities, risk rating, mitigation guidance), the re-test report; the testers' own quality control
- L7: Assurance and purpose: assurance about the system's security for the client, within the limit that a test shows presence not absence; the profession's integrity
way in: Known vulnerabilities and exploits
- L1: Public vulnerability databases (NVD) and exploit collections (Bugtraq, Metasploit's database of known exploits)
- L2: Versions of scanned services and software looked up, automatically by scanners or by hand
- L3: A list of hypothesised flaws ranked by the chance the flaw exists and the ease of exploiting it (flaw hypothesis methodology)
way in: Law and professional ethics
- L1: The law on computer access (the CFAA's 'exceeds authorized access'); consent is what separates white hats from others
- L2: Legal safe harbours valued by hunters; non-disclosure and limited liability in contracts
- L3: Professional ethics built on integrity: serve and protect the client, uphold the profession, avoid conflicts of interest and false results
way in: Mandates and standards
- L1: FISMA periodic testing no less than annually; PCI DSS penetration tests on a schedule and after changes; GSA's pre-vetted HACS services
- L2: The organisation's assessment policy: requirements, roles, methodology, frequency, documentation
- L3: Tests scheduled; methodologies (PTES, OSSTMM, NIST SP 800-115, ISSAF, OWASP) chosen
way in: People: the human element
- L1: Employees and their awareness of security
- L2: Pretexts, phone calls, phishing; staff names found in reconnaissance
- L3: Information revealed (for example a password), used to attack systems
way in: Reconnaissance: public information about the target
- L1: Open information about the target: host names and IP addresses, employee names and contact details on web and directory servers
- L2: Gathering: DNS interrogation, WHOIS queries, search engines (open source intelligence), and network sniffing on internal tests
- L3: A target profile: hosts and people, data usable later, for example in a social engineering attack
way in: Scanning: hosts, ports and services
- L1: The target's hosts, open ports and services
- L2: Port and service identification (Nmap) and vulnerability scanners; slowed to stay under the radar in covert tests
- L3: Scanned hosts compared with vulnerability databases; no tool covers an area, so several are run and merged by hand
way in: The application's behaviour: requests and responses
- L1: HTTP requests and responses, API operations and their documentation
- L2: An intercepting proxy (Burp, used by every web tester interviewed), crawlers, one test user per access group
- L3: A model of expected behaviour and access rules; deviations classified (vulnerability, input filter or harmless error?)
way in: The client's fixes, seen by re-testing
- L1: Items remediated by the client according to its risk appetite
- L2: The reported vulnerabilities attacked again
- L3: Remediation confirmed or not
way in: The client's request and what it hands over
- L1: The client's goals and written permission; what it provides (nothing but scope, credentials and architecture, or full access and source code)
- L2: The contract: scope, emergency contacts, source IP addresses, failsafe procedures; scope is what bounty hunters weigh most
- L3: The box chosen (black, gray, white); rate limits or firewalls disabled; dedicated test environments
- L4: Shortcuts built into the plan: breach assumed, local admin accounts provided, anti-virus disabled
way in: The community: new techniques and tools
- L1: Twitter/X, YouTube channels, blog posts, Reddit, GitHub, courses, conferences and colleagues
- L2: Reading, CTFs and write-ups; skills flow both ways between CTFs and work
- L3: New attack classes from security researchers trickle down to practitioners; community-extended tools evolve faster
way in: The defenders' response: detection by the blue team
- L1: Defenders' ability to detect attacks and respond
- L2: Red teams watch whether they are detected; honey-tokens on shares; some automated attacks 'too loud'
- L3: Tools and moves re-judged by their chance of detection; test traffic told apart from attacks by the named source IPs
way in: The premises: physical security
- L1: Fences, cameras, alarms, locks and employee behaviour at the site
- L2: Physical reconnaissance (people, places, security devices, weather); hardware tools for RFID, NFC, Sub-GHz radio and side channels
- L3: Weaknesses identified and an operational plan written, sent to the client for approval
way in: The target's condition during the test
- L1: Systems may be damaged or rendered inoperable; OT side effects can be catastrophic
- L2: Written logs kept; periodic reports to system administrators and management
- L3: Stability of the target judged before each exploit; 'unstable' attacks held back
- L4: The point where an additional action would cause damage
way in: The tester's experience and intuition (inner)
- L1: Experience from previous tests, CTFs, the same client or industry, and former work as a developer; where intuition comes from is contested
- L2: Intuition felt in the body: 'some organ that tingles if an operation looks fishy'; triggered by recently read exploits
- L3: Preconceptions about features (uploads, XML) and technologies (PHP) as likely weak
way out: Disclose new flaws to vendors
- L1: The flaw reported to the vendor
- L5: Responsible disclosure requests, met with resistance by some ICS suppliers; testers averse after time-wasting responses
- L6: A new vulnerability found in a vendor's product
way out: Exploit and pivot
- L1: The target's state changed: access held, or the network's security state altered
- L2: Access gained, privileges escalated, other machines reached (pivoting)
- L3: Payload delivered (Metasploit automates attacks on known vulnerabilities)
- L4: An attack chosen to verify a hypothesised flaw
way out: Leave things as found
- L1: Lights, locks, keyboards and chairs left as they were found
- L3: All equipment brought in is listed and checked out
- L5: Clean-up part of the plan
way out: Notify at once
- L1: The client acts before the test ends
- L3: Notification with countermeasures sent to the client's emergency contact
- L5: A highly vulnerable flaw found, for example one endangering human life
way out: Physical entry
- L1: Entry to a restricted area (a server room, an executive's office)
- L2: Security devices identified and defeated with tools and techniques
- L4: Operational plan approved by the client
way out: Raise the blue team's skills
- L1: Blue team skills raised; beachheads found and attackers evicted
- L2: Detection rehearsed against repeated attacks
- L6: Findings on detection shared; purple teams repeat attacks so the blue team can tune
way out: Report and recommend
- L1: The client remediates by its risk appetite
- L5: Results presented to the organisation's managers
- L6: Report: executive summary, vulnerabilities classified by risk matrix, remediation recommendations; social engineering results only in summary
way out: Social engineering
- L1: An employee tricked into revealing information
- L3: Delivered by hand: fully automated tools avoided for fear of detection and ethical qualms
- L4: A pretext chosen from reconnaissance