← all systems · see it in The System
LLM-based agent
An agent harness is what turns a language model into an autonomous agent: the surrounding code that builds the model's context, mediates its tools, runs the loop, and persists state across a long-horizon run. CoALA describes such language agents by their memory (working and long-term), their action space (internal and external) and a decision procedure looping planning and execution.
Properties published scienceour proposal
Sourced fields, and our own readings.
| born at | 2022 (CoALA cites Ahn et al. 2022 and Huang et al. 2022; ReAct submitted 2022-10) |
| born precision | year |
| boundary | Inside: the model (its own walk) plus the harness around it: context assembly, memory stores the agent reads and writes, the decision loop, the tool interface, skills and instruction files. Outside: the people, tools, environments, other agents and knowledge sources it exchanges with. Where the agent ends is contested: Shanahan notes 'large language model' names both the model and the systems it sits in, and Voyager uses the model as a blackbox service (contention C-431). |
| nested level | Built from a large language model and a harness (memory, decision procedure, tool interface); a member of multi-agent systems, agent communities and human-agent teams; runs on the model's servers and the software it calls; its medium is the written record (messages, files, memory). |
| scale | technological |
Measures (16) published science
| Characteristic clock: decision cycle | one model call plus the selected action per cycle |
| Characteristic clock: Generative Agents plan grain | 5-15 minutes (game time) |
| Generative Agents reflection threshold | 150 summed importance score |
| Generative Agents community size | 25 agents |
| Reflexion long-term memory cap | 1-3 stored experiences |
| Reflexion HumanEval pass@1 | 91 (vs 80 for GPT-4) % |
| MemGPT warning token count | e.g. 70 % of context window |
| MemGPT flush eviction | e.g. 50 % of context window evicted at flush |
| Context window sizes (models listed by MemGPT) | Claude 2: 100k; GPT-4 Turbo: 128k tokens |
| Voyager skills retrieved per task | 5 skills |
| Voyager top-5 skill retrieval accuracy | 96.5 % |
| ACE context collapse example | 18,282 -> 122 tokens in one step; accuracy 66.7 -> 57.1 (baseline 63.7) tokens; % |
| ACE average gains | 10.6 (agents); 8.6 (domain benchmarks) % |
| SWE-agent file viewer ablation (% resolved) | 30 lines 14.3; 100 lines 18.0; full file 12.7; last 5 observations 18.0; full history 15.0 % resolved |
| Community skills with vulnerabilities | 26.1 % |
| Skill metadata pre-loaded per skill | a few dozen tokens |
Open contentions (2) open contention
Where sources or readings disagree. Both stay on the map.
- C-413 Where do thoughts come from?
booked: No theory is proven, so the map lists the main ones side by side and doesn't pick one. It shows where thinking is handled in the brain, not where thoughts start.
the other reading: ReAct: reasoning traces generated by the model itself; Reflexion: a separate Self-Reflection model writes the reflections; MemGPT: the harness's queue manager writes the recursive summary; Generative Agents: a user can speak as the agent's 'inner voice'; Jerry's view (his, labelled): thoughts arrive as patterns from outside one's own slice; not booked as a source - C-432 Do large language models understand what they say?
booked: The map describes how these models work in the field's own terms (learned features, next-word probabilities) and rules nothing true or false about understanding, belief or meaning. The competing views are listed side by side.
the other reading: Mitchell & Krakauer: whether models 'understand' is a heated, open debate; Butlin et al.: no current AI systems are conscious by indicator properties; no obvious technical barrier to systems that meet them; Xi et al.: many AI researchers set aside whether an agent 'actually' thinks; Shanahan: loaded words ('knows', 'believes', 'thinks') invite anthropomorphism
Partners and parts published science
Built from / builds (7)
- Large language model
- Agent harness
- Context window / working memory (main context)
- Long-term memory (episodic, semantic, procedural)
- Decision procedure (agent program)
- Agent-computer interface
- Skill library
Sits inside / holds (4)
Trades with (valence) (8)
- Human (Homo sapiens) via sys:written-record · LLM-based agent gain
- Software via sys:written-record · LLM-based agent gain
- Internet via sys:written-record · LLM-based agent gain
- Large language model via sys:written-record · LLM-based agent gain
- Long-term memory (episodic, semantic, procedural) via sys:written-record · LLM-based agent gain
- Artificial intelligence via sys:written-record · LLM-based agent gain
- Insider threat via sys:computer-network · LLM-based agent mixed
- Mathematical proof: the practice of showing that a statement follows from axioms and earlier theorems, how it is written, checked (by people and machines) and accepted, and the derivations it works on via sys:written-record · LLM-based agent gain
expressed by (27)
- Generation/Synthesis
- Recombination/Composition
- Scaffolding/Enablement (Meta-Tooling)
- Templating / Standardization
- Quality Improvement/Refinement
- Interoperability/Transduction (Bridging)
- Encapsulation & Modularity (Abstraction)
- Symbiosis/Co-Creation (Collaborative Specialization)
- Evolvability/Plasticity Engineering (Design for Future Variation)
- Distributed Autonomy/Decentralized Execution (Subsidiarity)
- Internal Coordination/Regulation
- Sensing/Scanning
- Hypothesis/Goal Selection
- Search/Foraging
- Uncertainty Reduction/Learning
- Multi-Scale Data Fusion / Triangulation
- Trail-Marking / Externalized Memory (Stigmergy)
- Frontier Outposts / Sensing Extension
- Social Learning / Public Information Use
- Physical/Virtual Barrier
- Self-Monitoring
- Validation of Identity/Intention
- Validation of Effectiveness
- Triage & Prioritized Allocation
- Quality Control & Error Correction
- Bridging Supply Gaps (Store & Dormancy)
- Scheduled Build-and-Dismantle (On-Demand Structure)
floor filled by (12)
- Large language model
- Human (Homo sapiens)
- Software
- Internet
- Agent harness
- Context window / working memory (main context)
- Long-term memory (episodic, semantic, procedural)
- Decision procedure (agent program)
- Agent-computer interface
- Skill library
- Artificial intelligence
- Insider threat
Kinds (6)
- Artificial intelligence
- Conversational agent / AI assistant
- Generative agent (behaviour simulacrum)
- Software-engineering agent
- Embodied LLM agent (robot)
- Open-world game agent
Notes published science
- Generative Agents lets a user command an agent by speaking as its inner voice: 'the user takes on the persona of the agent’s “inner voice”—this makes the agent more likely to treat the statement as a directive.' On a machine agent the inner channel can be written from outside, which puts the where-do-thoughts-come-from contention on the bench in a form you can test.
- Context collapse, measured: when an LLM rewrote its own working playbook as one monolithic summary, 'at step 60 the context contained 18,282 tokens and achieved an accuracy of 66.7, but at the very next step it collapsed to just 122 tokens, with accuracy dropping to 57.1—worse than the baseline accuracy of 63.7 without adaptation.' The fix ACE found was itemized delta updates in place of rewrites.
- Three open agent harnesses built on opposite philosophies (one subtracting scaffolding, one accreting it) 'converged toward one architectural middle form of five recurring elements: a commoditised loop, an append-only replayable session record, model quirks kept as data, progressive disclosure of context, and explicit extension seams.' The author says it is 'parallel discovery, diffusion, and literal reuse', not independent invention, and that none of them has 'a tamper-evident record an outside party can check without trusting the runtime.'
Layers
The ways it takes in its world, where they meet, and how it acts back, each from L1 (where it meets the world) up.
the agent's working record (reader's view)
- L1: The record and world the agent touches: users' requests, instructions, memory stores, tools and environments, other agents, files, clock
- L2: Harness intake: event parsers, retrieval functions, the agent-computer interface, progressive disclosure of skills
- L3: Assembled context / working memory: system instructions, working context and a FIFO message queue, filled under the context-length limit
- L4: The model call (the large language model, its own walk)
- L5: Evaluation and reflection on outputs and past episodes
- L6: Decision procedure: plan (propose and evaluate) then execute; outputs parsed into actions, replies and memory writes
the boardroom (shared floors)
- L3: The assembled context / working memory: system instructions, knowledge, tool definitions, memory, state and the user's request, orchestrated by an assembly function into the model's input; it carries the active goals from cycle to cycle
- L4: The model call: the LLM reads the assembled input and generates text (reasoning, a plan, an action string)
- L5: Evaluation and reflection: an Evaluator scores outputs; a Self-Reflection model writes verbal cues; reflections synthesise memories into higher-level inferences
- L6: The decision procedure: each cycle program code runs reasoning and retrieval to propose and evaluate alternatives (planning), then executes the selected action; the model's output is parsed into an action name and arguments
way in: External knowledge retrieval
- L1: External knowledge sources, e.g. a Wikipedia API (ReAct) or a document index
- L2: Knowledge retrieved by functions such as retrieval-augmented generation or from knowledge graphs, as one component of the assembled context
way in: Files and documents
- L1: Documents uploaded by a user (an upload alert is an event); repository files the agent works on
- L2: File viewer: a window onto the file (100 lines did better than 30 lines or the full file in SWE-agent's ablation)
way in: Human feedback and approval
- L1: Human feedback: a subjective signal; developers are told to start with humans always in the loop
- L2: A human-proxy agent solicits human input (human_input_mode='ALWAYS'); harnesses wire human-in-the-loop as middleware, with a human-approval interrupt
way in: Long-term memory read back in: episodic
- L1: The memory stream: a natural-language record of everything the agent perceived and did; in MemGPT, evicted messages kept indefinitely in recall storage
- L2: Retrieval model scoring each record by recency, importance and relevance; only the top-ranked records that fit the context window go into the prompt
way in: Long-term memory read back in: semantic
- L1: Semantic memory: facts about the world, including inferences the agent drew from its own failed episodes; in MemGPT, an archival database (e.g. uploaded Wikipedia documents)
- L2: A retrieval procedure reads from long-term memory into working memory; in MemGPT a function call, paginated so results do not overflow the context
way in: Other agents' messages
- L1: Messages from other agents, in full natural language
- L2: Auto-reply: on receiving a message from another agent, the agent invokes generate_reply; in a group chat a manager picks the next speaker and broadcasts
way in: Perception of a physical environment
- L1: Perceptual inputs (visual, audio, tactile) from a physical environment; also temperature, humidity and brightness
- L2: Pre-trained captioning or vision-language models convert them into textual observations
way in: Standing instruction files and skills (procedural memory)
- L1: Skill bundles of instructions, workflow guidance, scripts, reference documents and metadata; the AGENTS.md convention for standing instructions
- L2: Progressive disclosure: only a name, description and path sit in the prompt (a few dozen tokens per skill, pre-loaded at startup); the body is read on demand
way in: System and developer instructions
- L1: System prompts: text from an application developer; a profile of the agent's role written into the prompt
- L2: The system-instructions section, the first of three contiguous sections of the main context; its priority over other text is trained into the model (instruction hierarchy)
way in: Task feedback signals (rewards, tests, errors)
- L1: Binary environment feedback, heuristics for common failure cases, self-written unit tests; execution errors; linter alerts
- L2: Converted into verbal feedback, a textual summary added as context in the next episode
way in: The agent's own earlier steps and reflections (inner channel)
- L1: The trajectory history (short-term memory) and the reflections the agent wrote; plans and reflections fed back into the memory stream
- L2: Appended to the FIFO queue with function inputs and outputs; when evicted, folded into a recursive summary at the queue's head
way in: The agent's own state (context pressure, body bars)
- L1: Prompt tokens passing a warning count of the context window (e.g. 70%); for an embodied game agent its inventory, equipment, health and hunger bars
- L2: The queue manager inserts a system message warning of an impending eviction (a 'memory pressure' warning)
way in: The user's messages
- L1: The user's immediate request, sent as text (in dialogue the agent can 'accept instructions')
- L2: Event parser: a user message is one kind of event, converted to a plain text message and appended to main context
way in: Time and schedule triggers
- L1: Timed events that run on a regular schedule, letting the agent run without user intervention
- L2: The same event parser: the timed event becomes a plain text message in main context and triggers inference
way in: Tool and environment observations
- L1: Digital environments and tools (search, calculator, translator; code execution; web pages), whose results come back as observations
- L2: Observations transformed into text; for software work an agent-computer interface (search/navigation, file viewer, file editor, context management) shapes what is returned
way out: Code that runs
- L1: Programs act in the environment (Minecraft in Voyager; a sandbox or repository)
- L2: A user-proxy agent executes code suggested by the LLM
- L6: Code chosen as the action space
way out: Files and commits
- L1: Files written; the filesystem used as memory
- L2: The edit function, with a linter that alerts the agent to mistakes
- L6: File edits chosen as actions
way out: Messages to other agents
- L1: Natural-language messages read by other agents
- L2: A group-chat manager selects the next speaker and broadcasts
- L6: An agent's actions result in message passing for later conversations
way out: Replies to the user
- L1: Text delivered to the person
- L2: Function calls let the agent choose when to return a response to the user
- L6: External action: dialogue, or returning results to the user
way out: Scheduled follow-ups and chained steps
- L1: The next cycle starts (now, or at the next event)
- L2: Without the flag the processor yields until the next event or scheduled interrupt
- L6: The model asks for an immediate follow-up inference (request_heartbeat=true)
way out: Tool calls
- L1: Tools run against knowledge bases or environments
- L2: The agent-computer interface or function executor runs the command
- L6: The model output parsed into an action name and arguments
way out: Updates to the model's parameters
- L1: Model weights changed (the deeper system)
- L2: Fine-tuning (Voyager notes its blackbox queries bypass it)
- L6: Updating LLM parameters is the second form of procedural memory
way out: Writes to memory (the agent writing its own future input)
- L1: Reflections and plans stored in the memory stream, later read back in
- L2: Self-directed memory edits through function calls (working context, archival storage)
- L6: Learning actions: writing information to long-term memory
way out: Writes to skills and instructions (procedural memory)
- L1: New programs added to the skill library, indexed by the embedding of their description
- L2: A Curator integrates insights into structured, itemized context updates, offline (system prompts) and online (agent memory)
- L6: Updating agent code (procedural memory): prompt templates, code-based skills
Sources (25) published science
Xi et al., The Rise and Potential of Large Language Model Based Agents: A Survey (arXiv:2309.07864v3) (behind its links)
Packer et al. 2023, MemGPT: Towards LLMs as Operating Systems (arXiv:2310.08560v2) (behind its links)
Sumers, Yao, Narasimhan & Griffiths, Cognitive Architectures for Language Agents (arXiv:2309.02427v3) (behind its links)
Yang et al. 2024, SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering (arXiv:2405.15793v3) (behind its links)
Wang et al. 2023, Voyager: An Open-Ended Embodied Agent with Large Language Models (arXiv:2305.16291v2) (behind its links)
Wu et al. 2023, AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation (arXiv:2308.08155v2) (behind its links)
Park et al. 2023, Generative Agents: Interactive Simulacra of Human Behavior (arXiv:2304.03442v2) (behind its links)
Zhao et al., A Survey of Large Language Models (arXiv:2303.18223 v19; Frontiers of Computer Science 20(12) 2026; doi:10.1007/s11704-026-60308-3) (behind its links)
Intelligent agent (Wikipedia, via the Wikimedia Enterprise snapshot of 2026-11-01) (behind its links)
Shanahan 2022, Talking About Large Language Models (arXiv:2212.03551) (behind its links)
Shinn et al. 2023, Reflexion: Language Agents with Verbal Reinforcement Learning (arXiv:2303.11366v4) (behind its links)
Zhang et al., Agentic Context Engineering: Evolving Contexts for Self-Improving Language Models (arXiv:2510.04618v3) (behind its links)
Xu & Yan 2026, Agent Skills for Large Language Models: Architecture, Acquisition, Security, and the Path Forward (arXiv:2602.12430v4) (behind its links)
Du et al. 2023, Improving Factuality and Reasoning in Language Models through Multiagent Debate (arXiv:2305.14325) (behind its links)
Yao et al. 2022, ReAct: Synergizing Reasoning and Acting in Language Models (arXiv:2210.03629) (behind its links)
Mei et al. 2025, A Survey of Context Engineering for Large Language Models (arXiv:2507.13334v2) (behind its links)
Wallace et al. 2024, The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions (arXiv:2404.13208) (behind its links)
Lynch, Wright, Larson, Ritchie, Mindermann, Hubinger, Perez & Troy 2025, Agentic Misalignment: How LLMs Could Be Insider Threats (arXiv:2510.05179v2) (behind its links)
Wang et al., A Survey on Large Language Model based Autonomous Agents (arXiv:2308.11432v7) (behind its links)
Pan et al. 2026, Natural-Language Agent Harnesses (arXiv:2603.25723v2) (behind its links)
Liu et al. 2023, Lost in the Middle: How Language Models Use Long Contexts (arXiv:2307.03172) (behind its links)