← all systems · see it in The System
Operant conditioning
Operant (instrumental) conditioning is a learning process in which voluntary behaviours are modified by their consequences: the frequency or duration of a behaviour increases through reinforcement and decreases through punishment or extinction. In practice it is the study of reversible behaviour maintained by reinforcement schedules; it requires the learner's own action, and like Pavlovian learning it runs on prediction errors.
Properties published scienceour proposal
Sourced fields, and our own readings.
| born at | kind: associative learning probably early Cambrian, about 540 Mya (proposed; contested lower); one act: the first contingency |
| born precision | proposed, not dated directly |
| boundary | Inside: the learning process the field names: an emitted act meets a consequence; reinforcement (positive and negative) makes it more frequent, punishment (positive and negative) and extinction less; schedules of reinforcement set the pattern of responding; discriminative stimuli and context come to set the occasion; shaping, chaining and conditioned reinforcement build sequences; escape and avoidance; choice and matching; goal-directed action and habit; the neural sites where act and outcome meet, the prediction error that sets how much is learned, and the changed behaviour. Outside: the senses and motor systems it borrows, the reward system and memory systems that carry it, classical conditioning (its sibling, often running alongside), and what runs on it: animal training, applied behaviour analysis, token economies, gambling machines and game design, addiction, reflex-conditioning therapy and brain-machine interfaces. |
| nested level | A kind of associative learning (self-learning, about the consequences of one's own acts) run by nervous systems: an identified CPG neuron in Aplysia, FoxP-positive wing-steering motoneurons and mushroom-body synapses in Drosophila, corticostriatal circuits and midbrain dopamine neurons in mammals, the inferior olive, cerebellum, sensorimotor cortex and spinal cord for reflex conditioning. |
| scale | organism |
Measures (20) published science
| food hopper presentation per reinforcement | 5 s |
| immediacy of reinforcement (dog training) | within 5 s teaches faster than after 30 s |
| effective interval for superstitious conditioning | 15 s very effective; 1 min much less |
| superstitious conditioning in pigeons | 6 of 8 birds with clearly defined responses |
| post-reinforcement pause on a fixed interval (pigeon) | 10, 15 or even 20 s |
| species and animals conditioned by the Brelands | 38 species; over 6,000 animals count |
| pig coin task ratio | 4 or 5 coins per reinforcement responses per reinforcer |
| matching in Drosophila | choice ratio approximately equal to reward ratio with some undermatching; 54 blocks from 18 flies |
| FoxP knockout delay | more than a week before operant self-learning is impaired days |
| fly self-learning protocol | nine periods of 2 min |
| H-reflex conditioning course | rapid small change, then gradual; asymptote by 50-60 days |
| H-reflex size after conditioning | rats and monkeys about 175% (up) and 55% (down); humans 140% and 69% with 3-5% as many trials % of initial |
| H-reflex conditioning success rate | 75-80% of rats (change of at least 20%) |
| trials and rewards per day (rat H-reflex) | 2000-6000 trials; 500-1000 rewards per day |
| within-session adaptation onset (human H-reflex) | after 4-6 sessions |
| walking after down-conditioning | reflex smaller in two-thirds of subjects over 30 sessions, months later |
| dopamine response falls with delay | smaller response the longer the delay to reward (temporal discounting) qualitative |
| associative learning origin | early Cambrian, about 540 Mya; 6-9 phyla |
| matching law formulated | 1961 year |
| RLHF model size against base model | 1.3B InstructGPT preferred to 175B GPT-3 (100x fewer parameters) |
Open contentions (4) open contention
Where sources or readings disagree. Both stay on the map.
- C-2570 Is addiction a choice, a habit or a compulsion?
booked: The map keeps the competing readings; none is ruled.
the other reading: excessive goal-directed drug choice under negative affect (Hogarth 2020: better supported); habit: strong stimulus-response associations (Hogarth: weaker support); compulsion: insensitivity to costs (Hogarth: weak support) - C-2574 Why do animals keep avoiding a danger that no longer comes?
booked: The map keeps the competing readings; none is ruled.
the other reading: two-process: classical conditioning of fear then operant escape from the signal; one-factor: avoiding animals often show little fear; a missed shock acts as a reinforcer - C-2593 How far does the law of effect explain behaviour?
booked: The map keeps the competing readings; none is ruled.
the other reading: autoshaping (sign tracking): pigeons peck a key paired with food though food comes regardless; appears to contradict the law of effect (encyclopedia; Gardner and Gardner's feedforward alternative); instinctive drift: conditioned acts drift toward instinct though never reinforced (Breland and Breland 1961); Skinner 1948: non-contingent food still conditions, by accidental contingency (superstition), read within the law - C-2605 Can creatures without nervous systems learn from consequences?
booked: The map keeps the competing readings; none is ruled.
the other reading: Thorndike 1911: learning by exercise and effect found as low as the starfish and perhaps in the protozoa; Bronfman et al. 2016: echinoderm reports lacked the controls separating associative learning from sensitization
Partners and parts published science
Built from / builds (8)
- Motor neuron
- Central pattern generator
- Spinal cord
- Mushroom body
- Striatum
- Midbrain dopamine neurons
- Adenylyl cyclase (convergence site)
- Instinctive behaviour and the conditioned act compete for the response (instinctive drift)
Sits inside / holds (3)
Trades with (valence) (17)
- Motivation via sys:sensory-environment · Operant conditioning gain
- Basal ganglia via sys:electrochemical-gradient · Operant conditioning gain
- Spinal cord via sys:electrochemical-gradient · Operant conditioning even
- Human (Homo sapiens) via sys:sensory-environment · Operant conditioning gain
- Large language model via sys:written-record · Operant conditioning even
- Money via sys:money · Operant conditioning gain
- Honeybee (western honey bee, Apis mellifera) via sys:sensory-environment · Operant conditioning even
- Dopamine via sys:electrochemical-gradient · Operant conditioning gain
- Memory via sys:sensory-environment · Operant conditioning gain
- Striatum via sys:electrochemical-gradient · Operant conditioning gain
- Midbrain dopamine neurons via sys:electrochemical-gradient · Operant conditioning gain
- Fruit fly (Drosophila melanogaster) via sys:sensory-environment · Operant conditioning even
- Classical conditioning via sys:sensory-environment · Operant conditioning mixed
- Learning via sys:sensory-environment · Operant conditioning even
- Aplysia via sys:sensory-environment · Operant conditioning even
- Addiction via sys:sensory-environment · Operant conditioning loss
- Gambling (slot machines, loot boxes) via sys:sensory-environment · Operant conditioning mixed
expressed by (11)
- Recombination/Composition
- Quality Improvement/Refinement
- Hypothesis/Goal Selection
- Search/Foraging
- Uncertainty Reduction/Learning
- Trail-Marking / Externalized Memory (Stigmergy)
- Surprise / Change-Point Detection
- Quick Harm Avoidance
- Reactive Response
- Proactive Defense
- Validation of Effectiveness
floor filled by (15)
- Motor neuron
- Nervous system
- Central pattern generator
- Basal ganglia
- Amygdala
- Prefrontal cortex
- The world through the senses (sensory environment)
- Orbitofrontal cortex
- Mushroom body
- Memory
- Inferior olive
- Striatum
- Midbrain dopamine neurons
- Reinforcement schedule
- Experimenter (laboratory procedure)
Kinds (12)
- Habit
- Associative learning
- Learning
- Positive punishment
- Negative punishment (penalty)
- Avoidance learning (discriminated and free-operant)
- Operant self-learning (Drosophila torque meter)
- Neuronal operant conditioning (biofeedback of firing)
- Operant conditioning of spinal reflexes (H-reflex)
- Goal-directed action (action-outcome)
- Positive reinforcement
- Negative reinforcement
Notes published science
- People can be rewarded into changing a spinal reflex they cannot feel: in rats and monkeys the reflex grew to about 175% or shrank to about 55% of its size over 50-60 days, and in people with incomplete spinal cord injury shrinking it came with faster, more even walking (src:thompson-wolpaw-2014).
- Fruit flies obey the same "matching law" Herrnstein found in pigeons and people in 1961, splitting their choices between two odours in proportion to the rewards each pays; switch off the fly's reward-expectation signal and the matching disappears (src:rajagopalan-2023).
- Feed a hungry pigeon on a clock, no matter what it does, and it still learns: six of eight birds settled into rituals such as turning counter-clockwise or tossing the head, repeating whatever they happened to be doing when food last came. Skinner called it superstition (src:skinner-1948).
Layers
The ways it takes in its world, where they meet, and how it acts back, each from L1 (where it meets the world) up.
operant conditioning: from an act and its consequence to a changed act (reader's view)
- L1: An act in the world meets a consequence: a cat presses a latch and reaches food; a lever press brings a pellet; a hot stove burns; a schedule sets when acts pay
- L2: The act is emitted, not elicited, and varies; the senses take in the consequence and the situation (discriminative stimuli), and a copy of the motor command registers the act from inside
- L3: Pathways carry the outcome to where the act is made: the dopamine-rich esophageal nerve En2 carries seaweed's reward in Aplysia; midbrain dopamine broadcasts to striatum, frontal cortex and amygdala; the inferior olive carries the reward contingency to the cerebellum
- L4: Act and outcome meet (credit assignment): an identified CPG neuron whose excitability rises, adenylyl cyclase as the convergence site, FoxP-positive motoneurons, mushroom-body synapses, the dorsomedial striatum encoding the action-outcome contingency
- L5: A prediction error sets how much is learned: dopamine neurons fire above baseline for more reward than predicted, stay at baseline for predicted reward, dip for less; in flies the plasticity rule must include reward expectation
- L6: The stored act-outcome association and the habit, gated by context: goal-directed action rests on contingency and value circuits; habits on sensorimotor cortex to posterior putamen; extinction is new inhibitory learning expressed in its context, not erasure
- L7: The act becomes more or less likely: responses followed by satisfaction recur, those followed by discomfort less; rates follow the schedule; choices are divided in proportion to the rewards received (matching)
the boardroom (shared floors)
- L5: Prediction error: both Pavlovian and operant learning involve prediction errors; most midbrain dopamine neurons fire above baseline for more reward than predicted, stay at baseline for fully predicted reward and dip for less; the signal matches the teaching term of temporal-difference learning; in flies the plasticity rule must carry reward expectation for matching to appear
- L6: The act-outcome association and the habit, and their gating: prefrontal cortex with dorsomedial striatum (anterior caudate in humans) encodes the contingency, orbitofrontal cortex and basolateral amygdala the outcome value; habits run from premotor and sensorimotor cortex to the posterior putamen; extinction adds context-bound inhibitory learning rather than erasing
- L7: The act made more or less likely, its rate set by the schedule, and choices divided among options in proportion to the rewards each returns (Herrnstein's matching law, seen in pigeons, people and flies)
way in: Appetitive outcome: food, seaweed, juice, a pellet (positive reinforcement)
- L1: Seaweed or seaweed juice presented after ingestion
- L3: The reward travels on the dopamine-rich esophageal nerve En2; in mammals, dopamine neurons respond to reward
- L4: Convergence with the act: adenylyl cyclase, the molecular site where the two signals meet
way in: Aversive outcome: heat, shock, a burn (punishment; escape and avoidance)
- L1: A punishing heat beam (infrared laser) on the fly's head; shock in avoidance experiments
- L4: The heat is linked to one turning direction with no guiding cue; the plasticity sits in the motoneurons
way in: Conditioned reinforcer: a clicker, tokens, coins, money
- L1: A clicker sounded immediately after the response, then food; tokens exchanged for primary reinforcers
- L4: The dopamine response transfers from the reward to the earliest reward-predicting stimulus
way in: Context: the room, the chamber, the situation continuously present
- L1: Walls, tables, the interior of the chamber
- L4: Extinction and renewal are context-bound: distinct corticostriatal, striatopallidal and striatohypothalamic ensembles and thalamic returns
way in: Discriminative stimulus: a light, a tone, an odour that sets the occasion
- L1: A light that comes on; the rattle of a food bag; odours that predict probabilistic reward
- L4: Odour-representing Kenyon cells sense dopamine; plasticity at Kenyon cell to output neuron synapses
way in: Instrumented act: a reflex size or a neuron's firing read by electrodes and rewarded
- L1: Reward given for modulations of neuronal firing not linked to overt behaviour
- L2: The instrument reads the act: reward based on reflex size measured by EMG
- L3: The reward contingency acts through the inferior olive on the cerebellum, then sensorimotor cortex, then the corticospinal tract
- L4: The spinal cord plasticity directly responsible for the reflex change; corticospinal tract and sensorimotor cortex required
way in: Internal state: deprivation and satiation
- L1: Blood sugar and appetite: deprivation makes sugar an effective reinforcer, satiation weakens it
- L4: Orbitofrontal and ventromedial prefrontal activity tracks hunger and pleasantness and falls after satiation
way in: Social consequence: praise, attention from another person
- L1: Praise given contingently on a behaviour
way in: The learner's own act (lever press, peck, turning, biting)
- L1: The act on the world: a cat presses a latch; a fly turns left or right; Aplysia bites
- L2: The act registered from inside: a copy of the motor command (efference copy)
- L3: The act is produced by its own motor circuit: the feeding central pattern generator in the buccal ganglia
- L4: The motor circuit itself is the port: an identified CPG neuron grows more excitable; aPKC in FoxP-positive wing-steering motoneurons
way in: Time since the last reinforcement (interval timing)
- L1: The interval between reinforcements: the pigeon waits 10, 15 or 20 s after eating before responding
way out: A spinal reflex made larger or smaller (H-reflex)
- L1: Walking: in people with incomplete spinal cord injury, a smaller soleus reflex came with faster, more symmetric walking
- L3: Corticospinal tract to the spinal cord, where the plasticity that changes the reflex sits
- L7: The reward criterion steers the reflex up or down
way out: Feeding (Aplysia biting, ingestion)
- L1: The radula protracts open and retracts closed around food
- L7: Conditioning raises the rate of ingestion-like motor patterns over yoked controls
way out: Flight steering (fly yaw torque)
- L1: Turning manoeuvres around the vertical body axis
- L7: The fly avoids the heated torque domain
way out: Neuronal firing changed without movement (neurofeedback, brain-machine interfaces)
- L1: Firing changed with no body movement; synchrony and oscillation proposed as the keys to BMI
- L7: Prefrontal neurons change their firing to meet the reward requirement
way out: Skeletal acts: pressing, pecking, pulling, turning, running
- L1: The act on the world: the lever, the key, the latch, the coin
- L7: Responses retained when they lead to a successful outcome
way out: Verbal behaviour (speech as operant behaviour)
- L1: Speech, learned as a form of motor learning
- L7: Skinner extended operant principles to language (mands and tacts)
Sources (20) published science
Thorndike EL 1911. Animal Intelligence, chapter 5: Laws and Hypotheses for Behavior. Classics in the History of Psychology (behind its links)
Schultz W (2016) Dopamine reward prediction error coding. Dialogues Clin Neurosci 18:23-32. doi:10.31887/DCNS.2016.18.1/wschultz (PMC4826767, open) (behind its links)
Skinner BF 1948. "Superstition" in the pigeon. J Exp Psychol 38:168-172. Classics in the History of Psychology (behind its links)
Bouton ME, Maren S, McNally GP 2021. Behavioral and neurobiological mechanisms of Pavlovian and instrumental extinction learning. Physiol Rev. (PMC8428921) (behind its links)
Rajagopalan et al. 2023. Reward expectations direct learning and drive operant matching in Drosophila. PNAS. doi:10.1073/pnas.2221415120 (PMC10523640) (behind its links)
Breland K, Breland M 1961. The misbehavior of organisms. Am Psychol 16:681-684. Classics in the History of Psychology (behind its links)
Hawkins RD, Byrne JH 2015. Associative Learning in Invertebrates. Cold Spring Harb Perspect Biol. doi:10.1101/cshperspect.a021709 (PMC4448622) (behind its links)
Ouyang et al. 2022, Training language models to follow instructions with human feedback (arXiv:2203.02155) (behind its links)
Motivation (Wikipedia, rev 1372798998, 2026-09-02, via Wikimedia Enterprise snapshot) (behind its links)
Classical conditioning (Wikipedia, rev 1380298957, 2026-10-14, via Wikimedia Enterprise snapshot 2026-11-01) (behind its links)
Learning (Wikipedia, via Wikimedia Enterprise) (behind its links)