EVOPEDIA

← all systems · see it in The System

Large language model

To discriminate the difference in parameter scale, the research community has coined the term large language models (LLM) for the PLMs of significant size. Its basic function is to generate statistically likely continuations of word sequences (Shanahan 2022); today it is used inside dialogue systems and agents with tools, retrieval and system instructions.

Properties published scienceour proposal

Sourced fields, and our own readings.

born at2020 (GPT-3, 175B parameters; the survey names it and PaLM as early LLMs)
born precisionthe field draws no sharp line; by parameter scale
boundaryInside: the trained network (tokenizer, embeddings, transformer blocks, unembedding, weights) and its run over a context, from the prompt to the sampled token; in today's deployed form also the instruction, tool, retrieval and long-context arrangements that feed it. Outside: the people, software, corpus and hardware it exchanges with. Shanahan keeps 'the bare-bones LLM' apart from 'the systems in which they are embedded' (contention C-431).
nested levelBuilt from transformer components (token embedding, attention heads, MLP layers, unembedding communicating through a residual stream); embedded as the core computation unit of dialogue systems and LLM-based agents; runs on accelerators in datacentres; its medium is written language.
scaletechnological

Measures (17) published science

GPT-3 parameters175 billion parameters
GPT-3 training tokens300 billion tokens
Common Crawl size (GPT-3 corpus)nearly a trillion words
Transformer base: layers and widthN = 6 layers; dmodel = 512 layers; dimensions
BPE vocabulary (WMT 2014 En-De)about 37000 tokens
characteristic clock: one token per decoding step1 token per forward step
characteristic clock: Transformer (big) training run, En-Fr3.5 days on eight GPUs
characteristic clock: model generations1 generation (a model trained on the previous generation's output)
power-law range of loss scalingmore than seven orders of magnitude
Chinchilla: parameters and data70B parameters, 4x the data of Gopher (same compute) parameters; relative tokens
InstructGPT 1.3B preferred over GPT-3 175B100x fewer parameters, yet preferred
labelers (InstructGPT)about 40 contractors
context window (GPT-4 Turbo)up to 128k tokens
image-text pairs (CLIP pre-training)400 million pairs
carbon footprint range from DNN, datacentre and processor choice~100-1000 times
NLP community survey on LM understanding51% agree / 49% disagree (480 respondents) percent
ReAct gain on ALFWorld / WebShop34 / 10 absolute % success rate over imitation and RL baselines

Open contentions (3) open contention

Where sources or readings disagree. Both stay on the map.

Partners and parts published science

Built from / builds (9)

Sits inside / holds (2)

Trades with (valence) (60)

expressed by (10)

floor filled by (41)

Kinds (8)

wired (6)

Notes published science

Layers

The ways it takes in its world, where they meet, and how it acts back, each from L1 (where it meets the world) up.

electricity into computation (data centre)

  1. L1: Electrical energy drawn by training and inference
  2. L2: ML-oriented accelerators (GPUs, TPUs) inside datacentres doing the arithmetic

text in the context window (reader's view)

  1. L1: Text arriving as the prompt: the user's words, an application developer's system prompt, few-shot demonstrations, tool and retrieval results, placed in one input sequence (the context window)
  2. L2: Tokenizer: text cut into subword units from a fixed vocabulary (byte-pair encoding and kin)
  3. L3: Learned token embeddings: each token becomes a vector of the model's width; image and audio features are projected into the same space
  4. L4: The residual stream: the running sum of the embedding and every earlier layer's output, a channel every layer reads from and writes to
  5. L5: Attention heads and MLP (feed-forward) layers in a stack of identical blocks: heads move information between token positions (masked to the past in causal decoders); feed-forward layers act as key-value memories
  6. L6: Features: directions in activation space found by sparse autoencoders, many multilingual, multimodal and abstract (packed in superposition, so single 'neurons' are often polysemantic)
  7. L7: Next-token probabilities from the unembedding and softmax; a token is sampled and becomes text, a tool call or code; in an agent this floor is where plans and actions are produced

the boardroom (shared floors)

  1. L3: Token embeddings: every channel's tokens become vectors of one width; visual and audio features are projected into the same word-embedding space
  2. L4: The residual stream: one shared channel all layers read from and write to
  3. L5: Attention heads (moving information between positions, masked to the past) and feed-forward layers (key-value memories), in a stack of identical blocks
  4. L6: Features: concept-like directions in activation space, multilingual and multimodal, readable and steerable by interpretability methods
  5. L7: The output distribution over next tokens and the sampled token: what the model does to its context and, through outlets, to the world

training: the corpus into the weights

  1. L1: The corpus: webpages, books, conversational text and specialised data (code, science)
  2. L2: Mixing and processing: ready-made datasets combined, filtered, de-duplicated and tokenized
  3. L3: The language-modelling objective: predict each token from the ones before it
  4. L4: Weights updated: loss falls as a power law in model size, data and compute; factual knowledge is stored in the parameters
  5. L5: Fine-tuning on demonstrations, then reinforcement learning from human (or AI) preference rankings

way in: AI feedback (RLAIF / Constitutional AI)

  1. L1: Self-critiques, revisions and a model's choice between two samples, under a list of principles
  2. L2: A preference model trained on AI preferences, used as the reward signal

way in: Audio (speech, sounds, music)

  1. L1: General auditory information: speech, audio events and music
  2. L2: Speech and audio encoders joined to a text-based LLM

way in: Human feedback (RLHF)

  1. L1: Demonstrations of desired behaviour and rankings of model outputs by labelers
  2. L2: A reward model trained to predict which output labelers prefer; the model is tuned against it (PPO)

way in: Images

  1. L1: An input image given with the text
  2. L2: CLIP visual encoder (ViT-L/14), then a trainable projection into word-embedding space as visual tokens

way in: Own previous output (autoregression, reasoning traces)

  1. L1: Previously generated tokens, and in agents the model's own reasoning traces, appended to the context

way in: Retrieved passages

  1. L1: Non-parametric memory: a dense vector index of documents (Wikipedia in RAG)
  2. L2: A pre-trained neural retriever picks passages; the generator conditions on them

way in: System / developer instructions

  1. L1: System prompts: text from an application developer, placed ahead of the user's text
  2. L2: Same tokenizer; priority over other text comes from training, not from a separate door

way in: The user's message

  1. L1: Prompts written by people and submitted through an API or chat interface
  2. L2: Tokenizer: subword units from a fixed vocabulary (BPE)

way in: Tool results

  1. L1: Results of API calls (calculator, Q&A system, search engines, translation, calendar) and observations from an environment, read back into the context

way in: Training corpus (the slow channel)

  1. L1: Webpages, books, conversational text and specialised data
  2. L2: Mixed, processed and tokenized; then predicted token by token and the weights updated

way out: Code that runs

  1. L1: A runtime such as a Python interpreter executes them
  2. L7: Programs written as intermediate reasoning steps

way out: Text out

  1. L1: Text returned to the user by the dialogue system that wraps the model; some of it goes online, where later corpora are scraped
  2. L7: Sampling from the nucleus of the next-token distribution

way out: Tool / API calls

  1. L1: Tools run: calculator, Q&A system, search engines, translation system, calendar; in agents, actions in an environment
  2. L7: Deciding which API to call, when, and with what arguments

Sources (82) published science

Zhao et al., A Survey of Large Language Models (arXiv:2303.18223 v19; Frontiers of Computer Science 20(12) 2026; doi:10.1007/s11704-026-60308-3)
Vaswani et al. 2017, Attention Is All You Need (arXiv:1706.03762) (behind its links)
Yao et al. 2022, ReAct: Synergizing Reasoning and Acting in Language Models (arXiv:2210.03629) (behind its links)
Ouyang et al. 2022, Training language models to follow instructions with human feedback (arXiv:2203.02155) (behind its links)
Shumailov et al. 2023, The Curse of Recursion: Training on Generated Data Makes Models Forget (arXiv:2305.17493; later in Nature 2024) (behind its links)
Bai et al. 2022, Constitutional AI: Harmlessness from AI Feedback (arXiv:2212.08073) (behind its links)
Shanahan 2022, Talking About Large Language Models (arXiv:2212.03551) (behind its links)
Brown et al. 2020, Language Models are Few-Shot Learners (arXiv:2005.14165) (behind its links)
Kaplan et al. 2020, Scaling Laws for Neural Language Models (arXiv:2001.08361) (behind its links)
Hoffmann et al. 2022, Training Compute-Optimal Large Language Models (arXiv:2203.15556) (behind its links)
Radford et al. 2021, Learning Transferable Visual Models From Natural Language Supervision (CLIP) (arXiv:2103.00020) (behind its links)
Patterson et al. 2021, Carbon Emissions and Large Neural Network Training (arXiv:2104.10350) (behind its links)
Mitchell & Krakauer 2023, The debate over understanding in AI's large language models, PNAS 120(13) (doi:10.1073/pnas.2215907120; arXiv:2210.13966) (behind its links)
Sennrich, Haddow & Birch 2016, Neural Machine Translation of Rare Words with Subword Units (arXiv:1508.07909) (behind its links)
Elhage et al. 2021, A Mathematical Framework for Transformer Circuits (Transformer Circuits Thread, open write-up) (behind its links)
Geva et al. 2021, Transformer Feed-Forward Layers Are Key-Value Memories (arXiv:2012.14913) (behind its links)
Xi et al., The Rise and Potential of Large Language Model Based Agents: A Survey (arXiv:2309.07864v3) (behind its links)
Liu et al. 2023, Visual Instruction Tuning (LLaVA) (arXiv:2304.08485) (behind its links)
Tang et al. 2023, SALMONN: Towards Generic Hearing Abilities for Large Language Models (arXiv:2310.13289) (behind its links)
Olsson et al. 2022, In-context Learning and Induction Heads (arXiv:2209.11895) (behind its links)
Templeton et al. 2024, Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet (Transformer Circuits Thread, open write-up) (behind its links)
Wallace et al. 2024, The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions (arXiv:2404.13208) (behind its links)
Schick et al. 2023, Toolformer: Language Models Can Teach Themselves to Use Tools (arXiv:2302.04761) (behind its links)
Manning CD, Clark K, Hewitt J, Khandelwal U, Levy O 2020. Emergent linguistic structure in artificial neural networks trained by self-supervision. PNAS 117:30046-30054. doi:10.1073/pnas.1907367117 (behind its links)
Hu J, Floyd S, Jouravlev O, Fedorenko E, Gibson E. 2023. A fine-grained comparison of pragmatic language understanding in humans and language models. ACL 2023. arXiv:2212.06801 (behind its links)
Wang G, Wang W, Yang D, Ren J (2026) Generative AI, Cognitive Offloading, and Learner Agency in Higher Education: A Scoping Review. Behav Sci 16:1150. doi:10.3390/bs16071150 (behind its links)
Ruder S (2016) An overview of gradient descent optimization algorithms. arXiv:1609.04747 (behind its links)
Computer program (encyclopedia) (behind its links)
Mirza A et al. (2025) A framework for evaluating the chemical knowledge and reasoning abilities of large language models against the expertise of chemists. Nat Chem. doi:10.1038/s41557-025-01815-x (PMC12226332) (behind its links)
Mikolov T, Yih W, Zweig G. Linguistic regularities in continuous space word representations. NAACL-HLT 2013:746-751 (behind its links)
Sumers, Yao, Narasimhan & Griffiths, Cognitive Architectures for Language Agents (arXiv:2309.02427v3) (behind its links)
Shehabi A, Newkirk A, Smith SJ, et al. (2024). 2024 United States Data Center Energy Usage Report. Lawrence Berkeley National Laboratory LBNL-2001637 (behind its links)
Lakretz Y, Desbordes T, Hupkes D, Dehaene S (2021) Causal Transformers perform below chance on recursive nested constructions, unlike humans. arXiv:2110.07240 (behind its links)
Butlin et al. 2023, Consciousness in Artificial Intelligence: Insights from the Science of Consciousness (arXiv:2308.08708) (behind its links)
Nanda N, Chan L, Lieberum T, Smith J, Steinhardt J (2023) Progress measures for grokking via mechanistic interpretability. ICLR 2023. arXiv:2301.05217 (behind its links)
Algorithm (encyclopedia, Wikimedia Enterprise snapshot 2026-11-01) (behind its links)
Chen M et al. 2021. Evaluating large language models trained on code. arXiv:2107.03374 (behind its links)
Serapio-García G, Safdari M, Crepy C, et al. (2023) Personality traits in large language models. arXiv:2307.00184 (preprint) (behind its links)
Binz M, Schulz E. 2023. Using cognitive psychology to understand GPT-3. PNAS. doi:10.1073/pnas.2218523120 (behind its links)
Garg S, Tsipras D, Liang P, Valiant G (2022) What can transformers learn in-context? A case study of simple function classes. NeurIPS 2022; arXiv:2208.01066 (v3) (behind its links)
Kantamneni S, Tegmark M (2025) Language Models Use Trigonometry to Do Addition. arXiv:2502.00873 (behind its links)
Becker J, Rush N, Barnes B, Rein D (2025) Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity. arXiv:2507.09089 (behind its links)
Porter B, Machery E (2024) AI-generated poetry is indistinguishable from human-written poetry and is rated more favorably. Sci Rep. doi:10.1038/s41598-024-76900-1 (behind its links)
Salvi, Horta Ribeiro, Gallotti, West 2024. On the conversational persuasiveness of large language models: a randomized controlled trial. arXiv:2403.14380 (behind its links)
Artificial intelligence (Wikipedia, via the Wikimedia Enterprise snapshot of 2026-11-01) (behind its links)
Oddi L, Scardapane S, Sugimoto T, Genovese D (2026) Authorship attribution and aesthetic evaluation of AI poetry: a case study with Haiku. arXiv:2609.15511 (behind its links)
Costello, Pennycook, Rand 2024. Durably reducing conspiracy beliefs through dialogues with AI. Science. doi:10.1126/science.adq1814 (PMID 39264999) (behind its links)
Heiding F, Lermen S, Kao A, Schneier B, Vishwanath A (2024) Evaluating Large Language Models' Capability to Launch Fully Automated Spear Phishing Campaigns: Validated on Human Subjects. arXiv:2412.00586 (behind its links)
Lampinen AK, Dasgupta I et al. Language models show human-like content effects on reasoning tasks. arXiv:2207.07051v4 (behind its links)
Gholami A, Yao Z, Kim S, Hooper C, Mahoney MW, Keutzer K. AI and memory wall. arXiv:2403.14123 (2024); IEEE Micro (behind its links)
Stonebraker M, Pavlo A (2024) What Goes Around Comes Around... And Around... SIGMOD Record 53(2):21-37. doi:10.1145/3685980.3685984 (behind its links)
Li P, Yang J, Islam MA, Ren S (2023, v5 2025). Making AI Less 'Thirsty': Uncovering and Addressing the Secret Water Footprint of AI Models. arXiv:2304.03271 (behind its links)
Pearce H, Ahmad B, Tan B, Dolan-Gavitt B, Karri R. Asleep at the keyboard? Assessing the security of GitHub Copilot's code contributions. arXiv:2108.09293 (IEEE S&P 2022) (behind its links)
Ismail, Kurnia, R., Widyatama, F., Wibawa, I. M., Brata, Z. A., Ukasyah, Nelistiani, G. A. & Kim, H. (2025). Enhancing Security Operations Center: Wazuh Security Event Response with Retrieval-Augmented-Generation-Driven Copilot. Sensors 25(3):870. doi:10.3390/s25030870 (PMC11820992, CC BY) (behind its links)
Mezzi E, Massacci F, Tuma K (2025) Large Language Models Are Unreliable for Cyber Threat Intelligence. arXiv:2503.23175 (behind its links)
Deng G et al. (2024). PentestGPT: Evaluating and Harnessing Large Language Models for Automated Penetration Testing. USENIX Security 24 (arXiv 2308.06782v2) (behind its links)
Henriot C. 2025. The AI-Augmented Research Process: A Historian's Perspective. arXiv:2508.01779 (preprint) (behind its links)
Schroeder DT, Kunst JR, et al. 2026. How malicious AI swarms can threaten democracy. Science (doi:10.1126/science.adz1697); author version arXiv:2506.06299v4 (behind its links)
Pantelidou N, Leivada E, et al. (2026) Community size rather than grammatical complexity better predicts Large Language Model accuracy in a novel Wug Test. PLoS ONE. doi:10.1371/journal.pone.0343164 (PMC12978473) (behind its links)
Wendler C, Veselovsky V, Monea G, West R (2024) Do Llamas work in English? On the latent language of multilingual transformers. arXiv:2402.10588 (behind its links)
Mahowald K, Ivanova AA, Blank IA, Kanwisher N, Tenenbaum JB, Fedorenko E (2024) Dissociating language and thought in large language models. Trends in Cognitive Sciences; arXiv:2301.06627 (abstract read via the arXiv API) (behind its links)
Bender EM, Koller A. Climbing towards NLU: on meaning, form, and understanding in the age of data. ACL 2020:5185-5198. doi:10.18653/v1/2020.acl-main.463 (behind its links)
Soviany, Ionescu, Rota & Sebe 2021, Curriculum Learning: A Survey, arXiv:2101.10382 (behind its links)
Salazar Alva RG, Nuñez M, López C, Martín Arista J 2025. AI-Driven Generation of Old English: A Framework for Low-Resource Languages (arXiv:2507.20111) (behind its links)
Computer science (Wikipedia, via Wikimedia Enterprise snapshot 2026-11-01) (behind its links)
Wessels B 2025. Developing an AI framework for learning in higher education: a humanities perspective from English Literature. International Journal of Educational Technology in Higher Education. doi:10.1186/s41239-025-00562-w (PMC12580428) (behind its links)
Avigad, Mathematics and the formal turn (arXiv 2311.00007v2, 2023) (behind its links)
Romera-Paredes et al.: Mathematical discoveries from program search with large language models (Nature 2024, doi:10.1038/s41586-023-06924-6, PMC10794145, CC BY) (behind its links)
Zavitsanos, E., Mavroeidis, D., Spyropoulou, E. & Fergadiotis, M. (2024) ENTRANT: A Large Financial Dataset for Table Understanding. Scientific Data 11. doi:10.1038/s41597-024-03605-5 (behind its links)
Huh, M., Cheung, B., Wang, T., Isola, P. 2024. The Platonic Representation Hypothesis. arXiv 2405.07987 (behind its links)
Majumdar, Elkind & Pournaras 2026, Generative AI voting: fair collective choice is resilient to LLM biases and inconsistencies. EPJ Data Science. doi:10.1140/epjds/s13688-025-00612-3 (PMC12963128, CC BY) (behind its links)
Lin S, Hilton J, Evans O (2021) TruthfulQA: measuring how models mimic human falsehoods. arXiv:2109.07958 (preprint) (behind its links)
Herbold S, Hautli-Janisz A, Heuer U, Kikteva Z, Trautsch A (2023) A large-scale comparison of human-written versus ChatGPT-generated essays. Sci Rep 13:18617. doi:10.1038/s41598-023-45644-9 (PMC10616290, CC BY) (behind its links)
Lewis et al. 2020, Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (arXiv:2005.11401) (behind its links)
Holtzman et al. 2019, The Curious Case of Neural Text Degeneration (arXiv:1904.09751) (behind its links)
Gao et al. 2022, PAL: Program-aided Language Models (arXiv:2211.10435) (behind its links)
Rubenstein et al. 2023, AudioPaLM: A Large Language Model That Can Speak and Listen (arXiv:2306.12925) (behind its links)
Liu et al. 2023, Lost in the Middle: How Language Models Use Long Contexts (arXiv:2307.03172) (behind its links)
Bahdanau, Cho & Bengio 2014, Neural Machine Translation by Jointly Learning to Align and Translate (arXiv:1409.0473) (behind its links)
Wei et al. 2022, Emergent Abilities of Large Language Models (arXiv:2206.07682) (behind its links)
Schaeffer, Miranda & Koyejo 2023, Are Emergent Abilities of Large Language Models a Mirage? (arXiv:2304.15004) (behind its links)
Lindsay 2020, Attention in Psychology, Neuroscience, and Machine Learning, Front. Comput. Neurosci. 14:29 (doi:10.3389/fncom.2020.00029; PMC7177153) (behind its links)