← all systems · see it in The System
Experiment, measurement and uncertainty (the physics lab as a practice: designing, measuring, estimating error)
An experiment is a procedure carried out to support or refute a hypothesis, or determine the efficacy or likelihood of something previously untried. In metrology's words, the result of a measurement is only an estimate of the value of the measurand and is complete only when accompanied by a statement of its uncertainty.
Properties published scienceour proposal
Sourced fields, and our own readings.
| born at | 1021 (Book of Optics) |
| born precision | contested by centuries either way |
| boundary | Inside: the prepared phenomenon and its controls, the apparatus and its sensors (eyes included), readout and triggers, data reduction with its uncertainty, comparison with models and other data, the strategies and checks that decide belief, and the claim. Outside: the systems measured (electrons, the Higgs boson), the units and standards it borrows (sys:measurement), the statistics and probability it uses, and the science that holds its results. |
| nested level | A human practice (rung 11) built on hypothesis testing seen in infants, carried by labs and collaborations, feeding the scientific record; its apparatus reaches down to particles (rung 3) and fields. |
| scale | social |
Measures (23) published science
| Physics measurement uncertainty halving time (characteristic clock of the practice) | about 15 years per factor-of-2 improvement (nuclear and particle physics; constants twice as fast) |
| ATLAS minimum time between first-level accepts (simple dead-time) | 100 ns (4 bunch crossings) |
| ATLAS collision rate in | up to 40 MHz |
| ATLAS first-level trigger maximum output | 100 kHz (75 kHz in Run 1), set by detector readout capability |
| ATLAS recording rate out | about 1 kHz |
| Trigger chain kept fraction (recorded / collisions) | about 1/40000 ratio (1 kHz / 40 MHz), a selection, not a toll |
| Discovery threshold (5 sigma, one tail) | 3e-7 p-value, or smaller |
| Earlier new-particle criteria | 3 sigma (0.27% per bin), later 4 sigma (0.0064%) probability per bin |
| 5 sigma claim with systematics underestimated by 2 | 2.5 sigma; p-value up by a factor of 2e4 significance |
| Fraction of measurement pairs disagreeing by more than 5 sigma | as high as 0.12 (against 6e-7 for a Normal distribution) fraction |
| Student-t degrees of freedom of measurement disagreements | about 2-3 (best consistency about 10) nu |
| Measurements analysed in Bailey 2017 | 41000 measurements of 3200 quantities count |
| Median improvement over best previous uncertainty (Nuclear; Constants; Particle; Stable; Medical) | 2.0; 1.8; 1.7; 1.3; 0.62 u_best/u_new |
| Higgs observation significance (ATLAS 2012) | 5.9 sigma; background fluctuation probability 1.7e-9 significance |
| Higgs mass measured (ATLAS 2012) | 126.0 +/- 0.4(stat) +/- 0.4(sys) GeV |
| OPERA neutrino baseline | about 730 km |
| OPERA corrected (v-c)/c | (2.7 +/- 3.1 (stat) +3.4/-3.3 (sys)) x 1e-6 relative |
| Millikan drops published versus recorded | 58 published; about 175 in the notebooks (Nov 1911 - Apr 1912); of about 100 in the 63 days used, about 25 aborted drops |
| Millikan-Fletcher elementary charge | 1.5924(17)e-19 C, about 0.6% from 1.602176634e-19 C |
| Personal equation between Bessel and his assistant | more than 1 s |
| Coverage factor k | typically 2 to 3 dimensionless |
| Odds of students spontaneously improving their methods after comparison-cycle training | 12 times (and 4 times more likely to identify a model's limitation) relative to control |
| Infants in Stahl & Feigenson 2015 | 110 (11-month-olds) infants |
Open contentions (9) open contention
Where sources or readings disagree. Both stay on the map.
- C-413 Where do thoughts come from?
booked: No theory is proven, so the map lists the main ones side by side and doesn't pick one. It shows where thinking is handled in the brain, not where thoughts start.
the other reading: see research/CONTENTIONS.md C-413 - C-1169 Is 'blind analysis' worth it in physics?
booked: The map keeps it as standard practice for reducing experimenter bias, and keeps the 2023 argument that its costs outweigh the benefits alongside.
the other reading: its costs (inhibits the search for systematics, hides unexpected events) outweigh a bias for which there is 'surprisingly little evidence' in physics (Golub 2023) - C-1178 How often do measurements disagree by more than their stated errors?
booked: The map keeps both: the standard statistical treatment for normal data, and the finding that real disagreements between measurements follow much heavier tails.
the other reading: exponential tails (earlier studies cited by Bailey); Gaussian errors as the working assumption; Bienayme (1853) ridiculing Cauchy uncertainties in any sensible instrument - C-1179 When did experimental science begin?
booked: The map dates it to Ibn al-Haytham's Book of Optics (1021), one of the first methodical approaches to experiment, and lists the other proposed origins without ruling.
the other reading: Daniel 1:12-16 as an early description of a controlled comparative test; Galileo's study of gravity as the origin of experimental research (History of experiments page); hypothesis testing in every human from infancy (Stahl & Feigenson 2015: 11-month-olds test what surprised them) - C-1180 Can experiments ever settle a dispute on their own?
booked: The map follows the view that strategies like calibration and independent confirmation give reasonable confidence, and keeps the 'experimenters' regress' argument alongside.
the other reading: the experimenters' regress: a good apparatus is one that gives correct results, and no formal criteria decide whether it works (Collins); 'a sufficiently determined critic can always find a reason to dispute any alleged result' - C-1181 Is 'five sigma' the right standard for a discovery?
booked: The map records the uniform five-sigma rule (a chance of about 3 in 10 million) as practised, and keeps the case for thresholds that vary by claim alongside.
the other reading: search-dependent thresholds (Lyons 2013's table: more for surprising, high-impact or systematics-dominated claims, fewer for expected ones); statisticians' doubt that distributions are accurate that far into the tails; Bailey 2017's heavy tails - C-1186 Do lab classes help students learn physics?
booked: The map keeps both findings: optional labs added nothing to exam results, while redesigned labs greatly improved students' experimental thinking.
the other reading: the encyclopedia: hands-on experiments can raise test scores (Stohr-Hunt 1996, K-12 frequency of hands-on work; not read) - C-1195 Did Millikan cherry-pick his oil drops?
booked: The map keeps both: the reading that he published only drops with complete observations, with his result barely changed by reanalysis, and the charge that he selected data.
the other reading: Holton (1978) read the omissions as without apparent reason (selection of data fitting his hypothesis); Franklin conceded possible 'cosmetic surgery' on the stated error - C-1209 What kind of probability is a measurement uncertainty?
booked: The map keeps both, as the international measurement guide does: probability from observed frequencies, and probability as degree of belief.
the other reading: frequentist flavour of particle-physics analyses against the Bayesian prior-odds reading of discovery claims (Lyons 2013)
Partners and parts published science
Built from / builds (9)
- Particle detector
- Measuring instrument
- Measurement uncertainty
- Scientific control
- Observational error (systematic and random)
- Hypothesis
- Trigger and data acquisition
- Blind analysis
- Statistical significance (p-value, n sigma)
Sits inside / holds (2)
- Science (as a human social and cultural system)
- Scientific method (conjecture, predict, test, report for retest)
Trades with (valence) (10)
- Higgs boson via sys:ionizing-radiation · Experiment, measurement and uncertainty (the physics lab as a practice: designing, measuring, estimating error) gain
- Tokamak via sys:written-record · Experiment, measurement and uncertainty (the physics lab as a practice: designing, measuring, estimating error) gain
- Electron via sys:electromagnetic-field · Experiment, measurement and uncertainty (the physics lab as a practice: designing, measuring, estimating error) gain
- Thermal energy (heat, temperature and heat flow) via sys:thermal-energy · Experiment, measurement and uncertainty (the physics lab as a practice: designing, measuring, estimating error) loss
- Human (Homo sapiens) via sys:sensory-environment · Experiment, measurement and uncertainty (the physics lab as a practice: designing, measuring, estimating error) gain
- Science (as a human social and cultural system) via sys:written-record · Experiment, measurement and uncertainty (the physics lab as a practice: designing, measuring, estimating error) gain
- Measurement (the practice of comparing a quantity with a unit, and the units and standards it uses) via sys:sensory-environment · Experiment, measurement and uncertainty (the physics lab as a practice: designing, measuring, estimating error) gain
- Statistics: the practice of collecting and reasoning from data, the intuitive statistics it grows from, and the structure (samples, distributions, estimators) it works on via sys:sensory-environment · Experiment, measurement and uncertainty (the physics lab as a practice: designing, measuring, estimating error) gain
- Probability: weighing chances, in brains and in people's mathematics, and the structure it works on via sys:sensory-environment · Experiment, measurement and uncertainty (the physics lab as a practice: designing, measuring, estimating error) gain
- Computational physics (computer simulation of physical systems: the practice and the simulated systems it stands for) via sys:sensory-environment · Experiment, measurement and uncertainty (the physics lab as a practice: designing, measuring, estimating error) gain
expressed by (22)
- Templating / Standardization
- Quality Improvement/Refinement
- Interoperability/Transduction (Bridging)
- Sensing/Scanning
- Hypothesis/Goal Selection
- Mapping/Modeling
- Uncertainty Reduction/Learning
- Intrinsic Motivation / Curiosity-Driven Sampling
- Active Experimentation / Probing (Optimal Experimental Design)
- Multi-Scale Data Fusion / Triangulation
- Trail-Marking / Externalized Memory (Stigmergy)
- Surprise / Change-Point Detection
- Hypothesis Portfolio / Multiple Hypothesis Tracking
- Optimal Stopping & Patch-Switching
- Social Learning / Public Information Use
- Physical/Virtual Barrier
- Technique-Based Identification (a.k.a. Technique-Based Detection)
- Self-Monitoring
- Validation of Effectiveness
- Redundancy & Diversity
- Triage & Prioritized Allocation
- Quality Control & Error Correction
floor filled by (21)
- Higgs boson
- Radioactive decay
- Scientific Revolution (the 16th-17th-century change in how Europeans investigated and understood nature)
- Human (Homo sapiens)
- Science (as a human social and cultural system)
- Particle detector
- The world through the senses (sensory environment)
- Measurement (the practice of comparing a quantity with a unit, and the units and standards it uses)
- Measuring instrument
- Measurement uncertainty
- Statistics: the practice of collecting and reasoning from data, the intuitive statistics it grows from, and the structure (samples, distributions, estimators) it works on
- Scientific control
- Observational error (systematic and random)
- Hypothesis
- Trigger and data acquisition
- Blind analysis
- Statistical significance (p-value, n sigma)
- Scientific model
- Computational physics (computer simulation of physical systems: the practice and the simulated systems it stands for)
- Condensed matter physics
- Wave-particle duality
Kinds (8)
- Controlled experiment (rule: the test)
- Natural experiment
- Field experiment
- Blinded experiment
- Instructional laboratory
- Observational study
- Randomized controlled trial (field experiment)
- Laboratory market experiment
Notes published science
- When thousands of physics measurements of the same quantity are compared, they disagree by more than 5 sigma up to 12% of the time, where a bell curve says 6 in ten million; real error bars have heavy, almost-Cauchy tails, and even the most careful work disagrees about a thousand times more often than textbooks expect (src:bailey-2017-not-normal).
- In 1796 the Astronomer Royal fired his assistant for timing star transits differently from him; twenty-odd years later Bessel tested it and found he himself was more than a second ahead of his own assistant, and astronomers began correcting for each observer's 'personal equation' (src:wp:personal-equation).
- Eleven-month-old babies already run experiments: after seeing a toy seem to pass through a wall they bang it on the table, and after seeing one hover in midair they drop it, testing exactly the rule it seemed to break (src:stahl-feigenson-2015).
- Does nature need imaginary numbers? A 2021 theory paper and two 2022 experiments said yes (one beat the real-number limit by 43 standard deviations); a 2025 preprint argues a real-number quantum theory fits the same postulates and no experiment can tell them apart. The question is open again (src:chen-2022, src:hoffreumon-woods-2025).
- The first 'numerical experiment' (1955) found that energy in a vibrating chain came back instead of spreading out, and it was found only because someone forgot to stop the computer. The programmer, Mary Tsingou, was left off the authors for 50 years (src:dauxois-2008-fput).
- Supercomputers computed the masses of the proton and neutron from just three inputs (two quark masses and the coupling) and matched experiment; about 95% of their mass is the energy of confined quarks and gluons, not the quarks' own mass (src:durr-bmw-hadron-masses).
Layers
The ways it takes in its world, where they meet, and how it acts back, each from L1 (where it meets the world) up.
experiment: a prepared intervention read through instruments and weighed against its uncertainty (reader's view)
- L1: The phenomenon under prepared conditions: the measurand, one factor manipulated, others held by controls, influence quantities that are never held perfectly constant
- L2: The apparatus and its sensors: instrument, detector, observer's eye; calibrated by reproducing known phenomena; finite resolution
- L3: Readout and data taking: indications, repeated observations, trigger selection; every indication carries random and systematic error
- L4: Data reduction and uncertainty: corrections, the measurement model, Type A and Type B evaluation; the result is complete only with its uncertainty
- L5: Comparison: data with data and data with models; significance against a background-only hypothesis
- L6: Scrutiny: strategies for reasonable belief, consistency checks, blind analysis, the decision that the result will stand up
- L7: The claim: supporting or refuting a hypothesis, a discovery at 5 sigma, a new experiment designed
the boardroom (shared floors)
- L4: Data reduction and uncertainty: corrections for systematic effects, Type A and Type B evaluation, the measurement model, combined standard uncertainty
- L5: Comparison: data against data and data against models; significance against a background-only hypothesis
- L6: Scrutiny and belief: calibration, elimination of alternative explanations, independent confirmation, blind analysis, consistency checks; when to stop
- L7: The claim and what it changes: a measured value with uncertainty, a discovery at 5 sigma, a refuted model, the next experiment
way in: Critique: colleagues, collaboration members and critics disputing or weighing the result
- L1: Objections from critics
- L5: Different members of a large group find different evidence convincing
way in: Detector: particles from collisions or decays registered by detector layers and selected by triggers
- L1: Collision products at up to 40 MHz of bunch crossings
- L2: Detector layers around the interaction point (inner tracker, calorimeters, muon spectrometer)
- L3: Hardware first-level trigger, then events buffered and passed to the software high-level trigger
way in: Electrical: voltmeters and electronics reading potentials and currents, over a thermal noise floor
- L1: The potential difference across the device measured
- L2: The instrument loads the circuit (finite impedance) and adds thermal noise of its own
- L3: The indication, limited by resolution
way in: Environment: influence quantities (temperature, pressure, stray currents) that act on the apparatus
- L1: Influence quantities not held completely constant
- L2: Their effect on the apparatus: thermal expansion of a metal ruler; railway leakage currents in LEP's vacuum pipe
- L3: Monitoring that reveals them (or does not: the unknown unknown)
way in: Literature: earlier published values and others' results
- L1: Published values of the same quantity (Review of Particle Physics, Table of Radionuclides)
- L4: Compatibility check against them
way in: Reference: standards, reference materials and known phenomena the apparatus is checked against
- L1: Measurement standards and reference materials, with inexact values
- L2: Calibration: the apparatus made to reproduce known phenomena (the Balmer series in a new spectrometer)
- L3: Type B input: uncertainty from an assumed distribution
way in: Sight: the experimenter reading scales, dials, screens and the drops in the eyepiece
- L1: Light from the object watched and the instrument's scale (an oil drop held in the field of view)
- L3: The reading, carrying personal bias in reading analogue instruments
way in: Surprise: the felt jolt of an unexpected result
- L1: An event that violates expectation (an object passing through a wall; data that disagree with both models)
- L4: Surprise: more looking, a changed face, and learning directed at the surprising object
way in: Thought: the hypothesis or model the experiment is built to test
- L1: The hypothesis (where it comes from is contested: C-413)
- L4: The mathematical model of the measurement and the prediction it gives
way in: Timing: the observer marking when something happens (transits, falls), by eye and ear and pulse
- L1: An event in time: a star crossing the wires of a reticule; a ball passing marks on Galileo's inclined plane
- L2: A timer the body sets: Galileo's pendulum synchronised to the human pulse
- L3: The recorded time, offset by the observer's own reaction (the personal equation)
way out: Calibrating: the apparatus corrected against standards
- L2: Instrument adjusted or calibrated against standards
- L6: The check fails or drifts
way out: Intervening: changing the apparatus and the conditions (the next run, the improved method)
- L1: One factor manipulated in the world, the rest held by controls
- L2: The apparatus adjusted (the oil-drop voltage tuned until the drop hangs still)
- L7: Deciding what to change
way out: Reporting: the result written with its uncertainty, and claimed (or not) as a discovery
- L1: The paper and the lab notebook
- L4: The value with its stated uncertainty
- L7: Whether to call it a discovery
Sources (28) published science
Holmes N.G., Wieman C.E., Bonn D.A. (2015) Teaching critical thinking. PNAS 112:11199 (doi:10.1073/pnas.1505329112) (behind its links)
Roodman A. (2003) Blind Analysis in Particle Physics, PHYSTAT2003 (arXiv:physics/0312102) (behind its links)
ATLAS Collaboration (2017) Performance of the ATLAS Trigger System in 2015. Eur. Phys. J. C 77:317 (arXiv:1611.09661) (behind its links)
Lyons L. (2013) Discovering the significance of 5 sigma (arXiv:1310.1284) (behind its links)
Bailey D.C. (2017) Not Normal: the uncertainties of scientific measurements. R. Soc. Open Sci. 4:160600 (arXiv:1612.00778) (behind its links)
Committee for the Prize in Economic Sciences 2021, Scientific Background: Answering causal questions using observational data (behind its links)
Blinded experiment (Wikipedia, via Wikimedia Enterprise snapshot 2026-11-01) (behind its links)
Committee for the Prize in Economic Sciences 2019, Scientific Background: Understanding development and poverty alleviation (behind its links)
Franklin A. & Perovic S., Experiment in Physics, Stanford Encyclopedia of Philosophy (peer-reviewed reference work) (behind its links)
OPERA Collaboration, Measurement of the neutrino velocity with the OPERA detector in the CNGS beam (arXiv:1109.4897, v4 corrected) (behind its links)
Goodstein D. (2000) In defense of Robert Andrews Millikan. Engineering and Science 63(4):30-38, Caltech (behind its links)
Personal equation (Wikipedia, via Wikimedia Enterprise snapshot 2026-11-01) (behind its links)
Tokamak (Wikipedia) (behind its links)
Hollingsworth, S. A. & Dror, R. O. (2018) Molecular dynamics simulation for all. Neuron 99, 1129, doi:10.1016/j.neuron.2018.08.011 (PMC6209097) (behind its links)
History of experiments (Wikipedia, via Wikimedia Enterprise) (behind its links)
OPERA experiment (Wikipedia, via Wikimedia Enterprise), abstract and tau-neutrino section (behind its links)