← all systems · see it in The System
Standardized test
A standardized test is a test that is administered and scored in a consistent or standard manner.
Properties published scienceour proposal
Sourced fields, and our own readings.
| born at | earliest evidence in Han China (encyclopedia); the psychometric form from Galton's work (late 1880s) and Binet and Simon's scale (commissioned 1904), mass group testing from 1917 |
| born precision | centuries (first evidence); years (the psychometric form) |
| boundary | Inside: the standard form and conditions, the items and item bank, the taker's responses, the scoring and scaling rules (CTT, IRT, equating), the norms or criterion and the score report. Outside: the takers as persons (Human), their schooling and families, the institutions that set stakes and use scores (states, schools, universities, employers, the army), and the coaching market. |
| nested level | A kind of examination (rung 11), built from items, takers, scoring rules and norms; part of school systems, admissions and licensing; carried by psychometrics as its science. |
| scale | social |
Measures (26) published science
| PISA cycle (characteristic clock) | triennial (reading 2000, mathematics 2003, science 2006 as main domains, repeating) years |
| re-norming interval (WISC editions) | 1949, 1974, 1991, 2003, 2014 (11-25 years apart) |
| Flynn effect, full-scale IQ | 0.28 (fluid 0.41, spatial 0.30, crystallized 0.21); 271 samples, almost 4 million people, 31 countries, 1909-2013 IQ points per year |
| time-of-day effect on test scores | -0.9% of an SD per hour later (95% CI 0.7-1.0); +1.7% after a 20-30 minute break |
| coaching effect on SAT I (median estimates) | verbal 8, math 18 (Becker's meta-analysis of 48 studies: 9 and 19) SAT points |
| coaching yield | about one extra math question per eight hours of effort; one verbal question per 25-30 hours questions per hour |
| scorer agreement (human scoring) | 60-85 percent |
| direct cost of US state tests (2012) | $27 per student; $669 million across 45 states US dollars |
| SAT-intelligence correlation | 0.5-0.9 (up to 0.8 with fluid reasoning and g in Frey and Detterman) |
| SAT's added predictive power over HSGPA | about 15 percent |
| first-year GPA predicted by HSGPA and SAT (34 colleges) | average R-squared .226 |
| college average SAT/ACT vs US News ranking | very nearly 0.9 |
| overprediction of Black students' first-year GPA | .11-.24 GPA points; reduced 54-72% by omitted variables (N = 499,622, 142 institutions) |
| score inflation (math, state vs NAEP) | state +0.088 vs NAEP +0.048 SD per year |
| NCLB effect on 4th-grade math (NAEP) | 0.22 by 2007 (no reading effect) SD (effect size) |
| income achievement gap growth | about 75 percent larger for the 2001 cohort than for the early-1940s cohort |
| Black-White gap narrowing (cohorts mid-1950s to mid-1970s, NAEP) | about 0.5 SD |
| WWI army testing | nearly two million men examined; about 5% scored very poorly and were retested individually people |
| test anxiety prevalence | 10-40 percent of students |
| colleges dropping SAT/ACT requirements after COVID (2020) | more than 1,600 (60%) US colleges |
| coaching uptake (1995-96 SAT takers) | nearly 12 percent |
| standard error of measurement (formula) | SD of observed scores times the square root of (1 - reliability) score points |
| median elementary school size per grade (accountability sample) | 69 children per grade |
| Flynn effect turning point (Norway) | rise, turning point and decline within cohorts born 1962-1991 birth cohorts |
| PISA mode effect (computer minus paper, randomized field trial) | maths about -0.10 to -0.20; reading -0.15 to -0.20; science Germany -0.25, Sweden -0.07 SD |
| pollution effect on an exam score (Israel, Bagrut) | 1 SD more PM2.5 (AQI): -0.93 points, 3.9% of an SD; +10 units across exams: -109 shekels ($30) monthly salary in 2010 points; shekels |
Open contentions (11) open contention
Where sources or readings disagree. Both stay on the map.
- C-3569 What do college admission tests predict?
booked: The map keeps the competing readings; none is ruled.
the other reading: the SAT measures cognitive skills learned in and outside school, not immutable aptitude (Jackson 1980; Messick and Jungeblut 1981, as reviewed by Amrein-Beardsley et al. 2025); coachability challenges the claim that it measures intrinsic aptitude (Slack and Porter 1980, as reviewed by Amrein-Beardsley et al. 2025); the makers moved the name away from 'aptitude': Scholastic Aptitude Test, then Scholastic Assessment Test, now SAT stands for nothing more than the name of the test (Amrein-Beardsley et al. 2025) - C-3572 Did No Child Left Behind raise what pupils learned, or only their scores on the tests that counted?
booked: The map keeps the competing readings; none is ruled.
the other reading: high-stakes gains outpace low-stakes gains in several states (score inflation; Jacob 2007); test-based accountability often produces score inflation (Koretz et al. 2016); related: C-3507 (Chicago high-stakes gains driven by test-specific skills); related: C-3566 (consequences and validity) - C-3573 How much do admission tests add to school grades in predicting college grades?
booked: The map keeps the competing readings; none is ruled.
the other reading: (a) range restriction and the criterion: predictive analyses on admitted students only are of limited interpretability (restriction of range; Freedle as reported by Santelices and Wilson 2015); how far observed validities understate operational ones is argued (specialist; no source read for the size of the correction); (b) SES: critics hold that tests measure nothing more than SES; Sackett et al. 2009 find SES related to scores (r = .42) but controlling it moves the test-grade correlation only from .47 to .44 (abstract); (c) decisions: the tests are psychometrically strong but the validity of admission decisions is at issue for minority and low-SES students (Amrein-Beardsley et al. 2025); related: C-3569 (Chetty et al.: SAT/ACT predictive of post-college success); test-optional evidence after 2020 not opened here - C-3576 Does test anxiety lower scores by crowding working memory, or do anxious students simply know less?
booked: The map keeps the competing readings; none is ruled.
the other reading: deficit or skills account: anxious students know less and know it (specialist's point; no source read) - C-3577 Did the US Army's 1917 picture test measure intelligence?
booked: The map keeps the competing readings; none is ruled.
the other reading: Gould (1981): the Army Beta was culturally biased, its administration a shambles and its summary statistics meaningless (as quoted by Warne et al. 2019) - C-3598 How much does coaching raise SAT scores?
booked: The map keeps the competing readings; none is ruled.
the other reading: positive relationships between special preparation and scores raise fairness concerns (Alderman and Powers 1980; Slack and Porter 1980, as reviewed by Amrein-Beardsley et al. 2025); commercial claims of 100+ point gains (customer surveys, not controlled studies) - C-3603 Do admission tests predict college grades equally well for every group of students?
booked: The map keeps the competing readings; none is ruled.
the other reading: over- and underprediction attributed to statistical artefacts (unreliability), to different college experiences, or to differences not captured by scores or grades; 'a matter of debate' (Santelices and Wilson 2015) - C-3627 What would make an admission test fair, and are today's tests fair?
booked: The map keeps the competing readings; none is ruled.
the other reading: the validity of decisions is at issue for minority and low-SES students though the tests are psychometrically strong (Amrein-Beardsley et al. 2025); competing definitions of culture-fair selection (Petersen and Novick 1976: record only, not read; the specialist's point that they cannot all be met is unsourced here) - C-3656 How should international school tests such as PISA be scaled, and how much do rankings depend on the choice?
booked: The map keeps the competing readings; none is ruled.
the other reading: PISA reading data did not fit the Rasch model (Kreiner and Christensen 2014); a two-factor model fits better (Goldstein 2004; Goldstein et al. 2007) (both as reviewed by Hopfenbeck et al.); no consensus on which IRT model to use; three-parameter variants fit PISA 2009 best; the model choice moves country SDs and percentiles substantially, so model error should be reported beside sampling error (Robitzsch 2022); plausible values for subjects a student did not sit at all are contested (Kreiner and Christensen 2014, as reviewed by Hopfenbeck et al.); delivery mode: the 2015 move to computer lowered scores 0.10-0.25 SD in a randomized trial; the adjustment helps but is unlikely to solve all comparability problems (Jerrim et al. 2018) - C-3664 Do some SAT questions work differently for Black and White test takers of equal ability?
booked: The map keeps the competing readings; none is ruled.
the other reading: technical criticisms from ETS researchers (Camara and Sathy 2004; Dorans 2004, 2010; Dorans and Zeller 2004; Wainer 2009, as listed by Santelices and Wilson 2015; not opened); a score from only the hardest items (R-SAT) did not predict better or reduce overprediction (Santelices and Wilson 2015) - C-3688 What is test validity: a property of how scores are used, or of what causes them?
booked: The map keeps the competing readings; none is ruled.
the other reading: causal-realist: a test is valid if the attribute exists and causally produces variation in the outcomes; validity should be about ontology, reference and causality, not epistemology, meaning and correlation (Borsboom et al. 2004); consequences matter but are the wrong concept for validity: 'right concern, wrong concept' (Popham 1997, title and abstract only)
Cross-scale patterns seen here (15) our proposal
Shapes that recur when the same kind of problem is solved at different scales, and how each shows in Standardized test. Each links to every other system that shows it.
- nested ladder (structural)
Two ladders of implication: validity needs reliability (adequate reliability is a prerequisite, not a guarantee), and equating is the strongest form of linking (equated forms are linked, not every link equates). - same role, different builds (structural)
One job, comparing people from many schools on one scale for admission or certification, done by different builds: the Polish Matura, the SAT and ACT, Colombia's Saber 11, the gaokao, a practical driving test, an audition with agreed weights. - templating (structural)
The defining move: the same test, given the same way, scored the same way for every taker, and kept interchangeable across forms by equating; PISA carries one test across languages from two source versions; the SAT was built on the Army tests. Over the century what is held the same moved up a layer: - The Payment to Entropy (structural)
Signal form: each hand-off from ability to response to score adds error (observed = true + error; reliability is the true-score share of observed variance, below one), human scoring loses agreement, and under stakes the indicator itself loses value (Campbell). No numeric per-hand-off efficiency read - the whole sets the local reference (structural)
A norm-referenced score means nothing alone: IQ 100 is defined as the standardization sample's mean, so the whole sets each person's reference; as the whole drifts (Flynn effect, ~0.28 points a year) readings against different norms no longer close over long periods. - trade one property for another (structural)
Multiple choice buys cheap, quick, reliable scoring and pays in what it can show (it cannot see whether a taker can write a paragraph); essays buy that view and pay in cost and scorer agreement of 60-85 percent; tests spread along the trade (all-machine, mixed, human-scored). - Umwelt: each system carves its own slice of one shared medium (structural)
The test's eight channels are its Umwelt: takers, what they know, the body on the day, the item bank, the norms, coaching, stakes and the audit. Each kind of standardized test (driving, creativity, fitness, personality, academic) carves its own slice of the same person. - who opens the door (structural)
The door at L2 is opened by the test ahead of the taker: standard instructions and forms; the Army's literacy screen routed men to Alpha or Beta, with pantomime instructions built for those who could not read English; accommodations change conditions without undermining the test. - causal ceiling (partial)
The true score is by definition never observed: every observed score is true score plus error, so no single sitting reaches it; only aggregation (group means, repeated forms) narrows the gap. Partial: the limit is set by the measurement model, but more items or sittings move it. - conservation law plus closure (partial)
Classical test theory's balance, Var(observed) = Var(true) + Var(error), leaves the split open; it needs added assumptions (uncorrelated errors, parallel forms) before reliability can be estimated. - lightning (partial)
One sitting, one discharge: a transitory disturbance on exam day (PM2.5) changes scores and, through them, attainment and earnings years later. - same shape, opposite sign (partial)
Cognitive fatigue, one process run both ways: each hour later in the day lowers scores 0.9% SD, a 20-30 minute break raises them 1.7% SD. - symmetry breaking (name the cut) (partial)
The cut score: a continuous scale cut at a named point (Proficient); students just below the cut are where score inflation concentrates. - holding / entrainment (surface)
Testing calendars time schools: NAPLAN in years 3, 5, 7 and 9; annual state tests; PISA every three years. Surface: a shared schedule more than a whole setting its parts' clocks. - seed (surface)
IRT's common scale for items and people let banks assemble different forms for different takers (adaptive testing, equating); 'IRT has made its way into every aspect of the testing industry'. Surface: a method's spread, not shown to unpack new combinatorics.
Valence: what it trades, and with whom (13) published science
Each exchange with a neighbour, read from both sides: who gains, who pays, and what carries it.
- Human (Homo sapiens) via sys:human-society · Standardized test gain
- State (polity); the nation state its predominant modern form via sys:human-society · Standardized test gain
- University via sys:human-society · Standardized test gain
- Family (the human family as a system) via sys:human-society · Standardized test even
- School via sys:human-society · Standardized test gain
- US Army (occupation and military districts) via sys:human-society · Standardized test gain
- Air pollution via sys:human-society · Standardized test loss
- Intelligence via sys:human-society · Standardized test gain
- Psychometrics via sys:human-society · Standardized test gain
- Test-preparation (coaching) companies via sys:human-society · Standardized test even
- Examination via sys:human-society · Standardized test gain
- PISA via sys:human-society · Standardized test gain
- Curriculum via sys:written-record · Standardized test gain
Partners and parts published science
Built from / builds (5)
- Human (Homo sapiens)
- Psychometrics
- Tasks and items (questions, essays, performances)
- Norm-referenced test
- The test maker
Sits inside / holds (4)
expressed by (9)
- Templating / Standardization
- Sensing/Scanning
- Mapping/Modeling
- Multi-Scale Data Fusion / Triangulation
- Surprise / Change-Point Detection
- Self-Monitoring
- Validation of Effectiveness
- Triage & Prioritized Allocation
- Quality Control & Error Correction
floor filled by (12)
- Human (Homo sapiens)
- State (polity); the nation state its predominant modern form
- University
- Family (the human family as a system)
- School
- Computer
- Statistics: the practice of collecting and reasoning from data, the intuitive statistics it grows from, and the structure (samples, distributions, estimators) it works on
- Air pollution
- Psychometrics
- Tasks and items (questions, essays, performances)
- The test maker
- Curriculum
represents (1)
Kinds (12)
- Examination
- Intelligence quotient
- Norm-referenced test
- Criterion-referenced test
- Licensing examination
- Performance test
- College admission test (SAT, ACT, Matura, gaokao, Saber 11)
- State accountability test
- International large-scale assessment (PISA, TIMSS, PIRLS)
- Computer-adaptive test
- Personality test
- National Assessment of Educational Progress (NAEP)
wired (1)
Notes published science
- Campbell's 1976 law of corrupted indicators calls achievement tests 'highly corruptible indicators', and backs it with Soviet factories: judged by weight, a nail factory made only its heaviest nails; judged by count, only the smallest (src:campbell-1976).
- Paid SAT coaching buys about one extra correct math question for every eight hours of effort, far from the 100-point gains companies advertised (src:powers-rock-1998).
- Bad air on exam day follows students for years: in Israel, a one-SD rise in PM2.5 on a matriculation exam day cost 3.9% of an SD on that exam, and ten more units of PM2.5 across a student's exams went with 109 shekels ($30) less monthly salary years later (src:ebenstein-2016).
Layers
The ways it takes in its world, where they meet, and how it acts back, each from L1 (where it meets the world) up.
standardized test: from a population of test takers to scores read the same way for all, and the decisions they open (reader's view)
- L1: The tested world: a population of test takers (students of an age, applicants, draftees) and the domain of knowledge or ability to be measured
- L2: The standard form and conditions: the same questions, instructions, timing and scoring rules for all; registration, routing to a form (Army Alpha or Beta), accommodations
- L3: The items: multiple choice, true-false, essays, performances; written, tried out and calibrated (IRT) into a bank
- L4: The response under standard conditions: where the construct meets construct-irrelevant variance (fatigue, anxiety, air)
- L5: Scoring and scaling: machine and rubric scoring; true score plus error; IRT ability estimates; equating
- L6: Meaning: norms or criterion; reliability and validity of the interpretation for a use
- L7: Use: admission, licensing, accountability, comparison
the boardroom (shared floors)
- L5: Scoring and scaling: machine scoring of multiple choice, rubric scoring with two or more scorers (agreement 60-85 percent); the observed score read as true score plus error (classical test theory) or as an ability estimate on a scale shared with the items (IRT); equating of forms so scores are interchangeable
- L6: Meaning: the score read against norms (a standardization sample) or a criterion (a cut score); validity as evidence and theory supporting an interpretation for a use (five sources of evidence); reliability a prerequisite, not a guarantee; group means more trustworthy than single scores
- L7: Use acting on the world: admission, placement, licensing, diagnosis, school accountability, national and international comparison
way in: Delivery mode (paper or screen)
- L1: The medium of the test itself: paper booklets or computer delivery (PISA moved most countries to computer in 2015)
- L2: The same questions presented on paper or on screen, pupils randomly assigned within schools (field trial)
- L3: Some questions function differently across modes; with those removed the estimates fall and most lose significance
- L4: Scores shifted: maths -0.10 to -0.20 SD and reading -0.15 to -0.20 SD on computer; science -0.25 Germany, -0.07 Sweden
way in: Item tryouts and item statistics (the item bank)
- L1: Draft items written to cover a domain
- L2: Items tried out and adapted; for PISA translated and adapted from two source versions, English and French
- L3: Item statistics under IRT: difficulty, discrimination, guessing; differential item functioning screens (the R-SAT used only the hardest items)
- L4: The calibrated item bank: items and abilities on one scale, from which forms and adaptive tests are assembled
way in: Low-stakes audit tests (NAEP, audit items)
- L1: A second, low-stakes measurement of the same students or states (NAEP)
- L2: Given without the high-stakes test's incentives
- L3: Trends on the audit and the high-stakes test compared; the comparison carries a content confound (different exams sample different parts of the domain)
- L4: Audit items built into the high-stakes test itself (self-monitoring assessments)
- L5: Gaps between the two read as inflation
way in: Test preparation and coaching
- L1: A commercial test-prep market, its claims (a guaranteed 100 points)
- L2: About 12 percent of 1995-96 SAT takers attended outside coaching programs
- L3: Practice on the test's item types; uncoached takers also gain on retesting
- L4: Small score change: about 8 verbal and 18 math points (median estimates)
way in: Test takers arriving (the tested population)
- L1: The population a test is given to: all students of a certain age, all adults who wish to acquire a licence for a particular job
- L2: Registration and routing to the right form: the Army sorted the literate to the Alpha and the rest to the Beta, by asking whether a man could read a newspaper and write a letter home; accommodations for disability that keep the point of the test
- L3: Seated under the same conditions: same test, same time, same circumstances
- L4: Sitting the test (paper, computer, computer-adaptive, practical); in sample surveys (PISA, NAEP) each student sits only part of the items and results are population estimates (plausible values)
way in: The body and conditions on the test day (construct-irrelevant variance)
- L1: The air, the clock and the body on the day: PM2.5 during Israeli matriculation exams; the hour of the test in Danish schools
- L2: Felt in the taker: 10-40 percent of students experience test anxiety
- L3: Associated with lower scores: all three dimensions of test anxiety negatively associated with high-stakes reading scores (correlational; the authors cite cognitive load theory's working-memory account as one explanation)
- L4: Performance shifted: -0.9% of an SD per hour later in the day, +1.7% after a 20-30 minute break
way in: The norming (standardization) sample
- L1: The population whose performance moves over time: IQ gains of about 0.28 points a year across 1909-2013 (Flynn effect); slowed or reversed in several countries (Norwegian conscripts born 1962-1991)
- L2: A standardization sample drawn from it takes the test under standard conditions
- L3: Its score distribution
- L4: The norms: IQ 100 defined as the sample's mean; percentiles and ranks
- L5: Norms age as the population moves; WISC re-normed 1974, 1991, 2003, 2014; stale norms meet the stakes: all 50 states reported above the national average on normed tests (Cannell 1988)
way in: What rides on the score (stakes and incentives)
- L1: Laws and institutions that attach rewards and sanctions: ESEA 1965, NCLB 2001, performance pay, admissions
- L2: Incentives reaching schools, teachers and takers
- L3: Responses: teaching to the test, excluding low scorers, cheating
- L4: Scores rising on the high-stakes test
- L5: Score inflation in the trend: state math +0.088 SD a year against NAEP +0.048
way in: What the taker knows and can do, drawn out by the items (the construct)
- L1: The taker's knowledge, skills and abilities: the construct the scores are meant to indicate
- L2: The item as stimulus: the same questions for everyone, read or heard
- L3: The response process: reasoning and retrieval; SAT scores correlate 0.5-0.9 with measures of intelligence (g)
- L4: The marked response; under the 3PL model a lower asymptote gives low-ability takers a chance of a correct answer
way out: Decisions on individuals (admission, placement, licence)
- L1: People sorted: some commanding officers balanced units by score, others ignored scores
- L6: Score reports and ranks; colleges' average SAT/ACT tracks their US News ranking (r near 0.9)
- L7: The decision: admission, a licence, an army assignment
way out: National and international comparison (NAEP, PISA, TIMSS, PIRLS)
- L1: Education policy formed on PISA data in many countries
- L6: Country reports and secondary analyses (654 peer-reviewed PISA articles, 2000-2015)
- L7: Triennial PISA results; international benchmarks; population estimates from an incomplete design (plausible values), not individual scores like an admission test
way out: School accountability and backwash
- L1: Classrooms changed: what is not tested is not taught
- L3: Instruction narrowed to what is tested
- L6: Ratings and rankings of schools and teachers; school means are imprecise (median elementary school: 69 children per grade), so rewards and sanctions at the extremes fall mostly on small schools
- L7: Funds allocated and poorly performing schools closed on test results
way out: Scores as data for research
- L1: The research literature and the policy debate
- L6: Findings: income gaps 75 percent larger for the 2001 cohort than the early-1940s cohort
- L7: Test data used to study gaps, growth and intelligence (SAT as a proxy for g)
Sources (24) published science
Flynn effect (Wikipedia, via Wikimedia Enterprise snapshot 2026-11-01) (behind its links)