EVOPEDIA

← all systems · see it in The System

Operant conditioning

Operant (instrumental) conditioning is a learning process in which voluntary behaviours are modified by their consequences: the frequency or duration of a behaviour increases through reinforcement and decreases through punishment or extinction. In practice it is the study of reversible behaviour maintained by reinforcement schedules; it requires the learner's own action, and like Pavlovian learning it runs on prediction errors.

Properties published scienceour proposal

Sourced fields, and our own readings.

born atkind: associative learning probably early Cambrian, about 540 Mya (proposed; contested lower); one act: the first contingency
born precisionproposed, not dated directly
boundaryInside: the learning process the field names: an emitted act meets a consequence; reinforcement (positive and negative) makes it more frequent, punishment (positive and negative) and extinction less; schedules of reinforcement set the pattern of responding; discriminative stimuli and context come to set the occasion; shaping, chaining and conditioned reinforcement build sequences; escape and avoidance; choice and matching; goal-directed action and habit; the neural sites where act and outcome meet, the prediction error that sets how much is learned, and the changed behaviour. Outside: the senses and motor systems it borrows, the reward system and memory systems that carry it, classical conditioning (its sibling, often running alongside), and what runs on it: animal training, applied behaviour analysis, token economies, gambling machines and game design, addiction, reflex-conditioning therapy and brain-machine interfaces.
nested levelA kind of associative learning (self-learning, about the consequences of one's own acts) run by nervous systems: an identified CPG neuron in Aplysia, FoxP-positive wing-steering motoneurons and mushroom-body synapses in Drosophila, corticostriatal circuits and midbrain dopamine neurons in mammals, the inferior olive, cerebellum, sensorimotor cortex and spinal cord for reflex conditioning.
scaleorganism

Measures (20) published science

food hopper presentation per reinforcement5 s
immediacy of reinforcement (dog training)within 5 s teaches faster than after 30 s
effective interval for superstitious conditioning15 s very effective; 1 min much less
superstitious conditioning in pigeons6 of 8 birds with clearly defined responses
post-reinforcement pause on a fixed interval (pigeon)10, 15 or even 20 s
species and animals conditioned by the Brelands38 species; over 6,000 animals count
pig coin task ratio4 or 5 coins per reinforcement responses per reinforcer
matching in Drosophilachoice ratio approximately equal to reward ratio with some undermatching; 54 blocks from 18 flies
FoxP knockout delaymore than a week before operant self-learning is impaired days
fly self-learning protocolnine periods of 2 min
H-reflex conditioning courserapid small change, then gradual; asymptote by 50-60 days
H-reflex size after conditioningrats and monkeys about 175% (up) and 55% (down); humans 140% and 69% with 3-5% as many trials % of initial
H-reflex conditioning success rate75-80% of rats (change of at least 20%)
trials and rewards per day (rat H-reflex)2000-6000 trials; 500-1000 rewards per day
within-session adaptation onset (human H-reflex)after 4-6 sessions
walking after down-conditioningreflex smaller in two-thirds of subjects over 30 sessions, months later
dopamine response falls with delaysmaller response the longer the delay to reward (temporal discounting) qualitative
associative learning originearly Cambrian, about 540 Mya; 6-9 phyla
matching law formulated1961 year
RLHF model size against base model1.3B InstructGPT preferred to 175B GPT-3 (100x fewer parameters)

Open contentions (4) open contention

Where sources or readings disagree. Both stay on the map.

Partners and parts published science

Built from / builds (8)

Sits inside / holds (3)

Trades with (valence) (17)

expressed by (11)

floor filled by (15)

Kinds (12)

Notes published science

Layers

The ways it takes in its world, where they meet, and how it acts back, each from L1 (where it meets the world) up.

operant conditioning: from an act and its consequence to a changed act (reader's view)

  1. L1: An act in the world meets a consequence: a cat presses a latch and reaches food; a lever press brings a pellet; a hot stove burns; a schedule sets when acts pay
  2. L2: The act is emitted, not elicited, and varies; the senses take in the consequence and the situation (discriminative stimuli), and a copy of the motor command registers the act from inside
  3. L3: Pathways carry the outcome to where the act is made: the dopamine-rich esophageal nerve En2 carries seaweed's reward in Aplysia; midbrain dopamine broadcasts to striatum, frontal cortex and amygdala; the inferior olive carries the reward contingency to the cerebellum
  4. L4: Act and outcome meet (credit assignment): an identified CPG neuron whose excitability rises, adenylyl cyclase as the convergence site, FoxP-positive motoneurons, mushroom-body synapses, the dorsomedial striatum encoding the action-outcome contingency
  5. L5: A prediction error sets how much is learned: dopamine neurons fire above baseline for more reward than predicted, stay at baseline for predicted reward, dip for less; in flies the plasticity rule must include reward expectation
  6. L6: The stored act-outcome association and the habit, gated by context: goal-directed action rests on contingency and value circuits; habits on sensorimotor cortex to posterior putamen; extinction is new inhibitory learning expressed in its context, not erasure
  7. L7: The act becomes more or less likely: responses followed by satisfaction recur, those followed by discomfort less; rates follow the schedule; choices are divided in proportion to the rewards received (matching)

the boardroom (shared floors)

  1. L5: Prediction error: both Pavlovian and operant learning involve prediction errors; most midbrain dopamine neurons fire above baseline for more reward than predicted, stay at baseline for fully predicted reward and dip for less; the signal matches the teaching term of temporal-difference learning; in flies the plasticity rule must carry reward expectation for matching to appear
  2. L6: The act-outcome association and the habit, and their gating: prefrontal cortex with dorsomedial striatum (anterior caudate in humans) encodes the contingency, orbitofrontal cortex and basolateral amygdala the outcome value; habits run from premotor and sensorimotor cortex to the posterior putamen; extinction adds context-bound inhibitory learning rather than erasing
  3. L7: The act made more or less likely, its rate set by the schedule, and choices divided among options in proportion to the rewards each returns (Herrnstein's matching law, seen in pigeons, people and flies)

way in: Appetitive outcome: food, seaweed, juice, a pellet (positive reinforcement)

  1. L1: Seaweed or seaweed juice presented after ingestion
  2. L3: The reward travels on the dopamine-rich esophageal nerve En2; in mammals, dopamine neurons respond to reward
  3. L4: Convergence with the act: adenylyl cyclase, the molecular site where the two signals meet

way in: Aversive outcome: heat, shock, a burn (punishment; escape and avoidance)

  1. L1: A punishing heat beam (infrared laser) on the fly's head; shock in avoidance experiments
  2. L4: The heat is linked to one turning direction with no guiding cue; the plasticity sits in the motoneurons

way in: Conditioned reinforcer: a clicker, tokens, coins, money

  1. L1: A clicker sounded immediately after the response, then food; tokens exchanged for primary reinforcers
  2. L4: The dopamine response transfers from the reward to the earliest reward-predicting stimulus

way in: Context: the room, the chamber, the situation continuously present

  1. L1: Walls, tables, the interior of the chamber
  2. L4: Extinction and renewal are context-bound: distinct corticostriatal, striatopallidal and striatohypothalamic ensembles and thalamic returns

way in: Discriminative stimulus: a light, a tone, an odour that sets the occasion

  1. L1: A light that comes on; the rattle of a food bag; odours that predict probabilistic reward
  2. L4: Odour-representing Kenyon cells sense dopamine; plasticity at Kenyon cell to output neuron synapses

way in: Instrumented act: a reflex size or a neuron's firing read by electrodes and rewarded

  1. L1: Reward given for modulations of neuronal firing not linked to overt behaviour
  2. L2: The instrument reads the act: reward based on reflex size measured by EMG
  3. L3: The reward contingency acts through the inferior olive on the cerebellum, then sensorimotor cortex, then the corticospinal tract
  4. L4: The spinal cord plasticity directly responsible for the reflex change; corticospinal tract and sensorimotor cortex required

way in: Internal state: deprivation and satiation

  1. L1: Blood sugar and appetite: deprivation makes sugar an effective reinforcer, satiation weakens it
  2. L4: Orbitofrontal and ventromedial prefrontal activity tracks hunger and pleasantness and falls after satiation

way in: Social consequence: praise, attention from another person

  1. L1: Praise given contingently on a behaviour

way in: The learner's own act (lever press, peck, turning, biting)

  1. L1: The act on the world: a cat presses a latch; a fly turns left or right; Aplysia bites
  2. L2: The act registered from inside: a copy of the motor command (efference copy)
  3. L3: The act is produced by its own motor circuit: the feeding central pattern generator in the buccal ganglia
  4. L4: The motor circuit itself is the port: an identified CPG neuron grows more excitable; aPKC in FoxP-positive wing-steering motoneurons

way in: Time since the last reinforcement (interval timing)

  1. L1: The interval between reinforcements: the pigeon waits 10, 15 or 20 s after eating before responding

way out: A spinal reflex made larger or smaller (H-reflex)

  1. L1: Walking: in people with incomplete spinal cord injury, a smaller soleus reflex came with faster, more symmetric walking
  2. L3: Corticospinal tract to the spinal cord, where the plasticity that changes the reflex sits
  3. L7: The reward criterion steers the reflex up or down

way out: Feeding (Aplysia biting, ingestion)

  1. L1: The radula protracts open and retracts closed around food
  2. L7: Conditioning raises the rate of ingestion-like motor patterns over yoked controls

way out: Flight steering (fly yaw torque)

  1. L1: Turning manoeuvres around the vertical body axis
  2. L7: The fly avoids the heated torque domain

way out: Neuronal firing changed without movement (neurofeedback, brain-machine interfaces)

  1. L1: Firing changed with no body movement; synchrony and oscillation proposed as the keys to BMI
  2. L7: Prefrontal neurons change their firing to meet the reward requirement

way out: Skeletal acts: pressing, pecking, pulling, turning, running

  1. L1: The act on the world: the lever, the key, the latch, the coin
  2. L7: Responses retained when they lead to a successful outcome

way out: Verbal behaviour (speech as operant behaviour)

  1. L1: Speech, learned as a form of motor learning
  2. L7: Skinner extended operant principles to language (mands and tacts)

Sources (20) published science

Operant conditioning (Wikipedia, via Wikimedia Enterprise snapshot)
Thorndike EL 1911. Animal Intelligence, chapter 5: Laws and Hypotheses for Behavior. Classics in the History of Psychology (behind its links)
Schultz W (2016) Dopamine reward prediction error coding. Dialogues Clin Neurosci 18:23-32. doi:10.31887/DCNS.2016.18.1/wschultz (PMC4826767, open) (behind its links)
Ehweiner A, Duch C, Brembs B 2024. Wings of Change: aPKC/FoxP-dependent plasticity in steering motor neurons underlies operant self-learning in Drosophila. F1000Research. doi:10.12688/f1000research.146347.2 (PMC11109550) (behind its links)
Skinner BF 1948. "Superstition" in the pigeon. J Exp Psychol 38:168-172. Classics in the History of Psychology (behind its links)
Bouton ME, Maren S, McNally GP 2021. Behavioral and neurobiological mechanisms of Pavlovian and instrumental extinction learning. Physiol Rev. (PMC8428921) (behind its links)
Rajagopalan et al. 2023. Reward expectations direct learning and drive operant matching in Drosophila. PNAS. doi:10.1073/pnas.2221415120 (PMC10523640) (behind its links)
Breland K, Breland M 1961. The misbehavior of organisms. Am Psychol 16:681-684. Classics in the History of Psychology (behind its links)
Hawkins RD, Byrne JH 2015. Associative Learning in Invertebrates. Cold Spring Harb Perspect Biol. doi:10.1101/cshperspect.a021709 (PMC4448622) (behind its links)
Thompson AK, Wolpaw JR 2014. Operant conditioning of spinal reflexes: from basic science to clinical therapy. Front Integr Neurosci. doi:10.3389/fnint.2014.00025 (PMC3957063) (behind its links)
Griffiths KR, Morris RW, Balleine BW 2014. Translational studies of goal-directed action as a framework for classifying deficits across psychiatric disorders. Front Syst Neurosci. doi:10.3389/fnsys.2014.00101 (PMC4033402) (behind its links)
Bronfman ZZ, Ginsburg S, Jablonka E. The Transition to Minimal Consciousness through the Evolution of Associative Learning. Frontiers in Psychology 7:1954 (2016). doi:10.3389/fpsyg.2016.01954 (behind its links)
Ouyang et al. 2022, Training language models to follow instructions with human feedback (arXiv:2203.02155) (behind its links)
Costa RM, Baxter DA, Byrne JH 2022. Neuronal population activity dynamics reveal a low-dimensional signature of operant learning in Aplysia. Commun Biol. doi:10.1038/s42003-022-03044-1 (PMC8786933) (behind its links)
Hogarth L 2020. Addiction is driven by excessive goal-directed drug choice under negative affect: translational critique of habit and compulsion theory. Neuropsychopharmacology. doi:10.1038/s41386-020-0600-8 (PMC7265389) (behind its links)
Motivation (Wikipedia, rev 1372798998, 2026-09-02, via Wikimedia Enterprise snapshot) (behind its links)
Sakurai Y, Song K, Tachibana S, Takahashi S 2014. Volitional enhancement of firing synchrony and oscillation by neuronal operant conditioning. Front Syst Neurosci. doi:10.3389/fnsys.2014.00011 (PMC3915778) (behind its links)
Classical conditioning (Wikipedia, rev 1380298957, 2026-10-14, via Wikimedia Enterprise snapshot 2026-11-01) (behind its links)
Learning (Wikipedia, via Wikimedia Enterprise) (behind its links)
Staddon JER, Cerutti DT. 2003. Operant Conditioning. Annu Rev Psychol. doi:10.1146/annurev.psych.54.101601.145124 (behind its links)