← all systems · see it in The System
Database
An organized collection of data kept with a database management system (DBMS), 'the software that interacts with end users, applications, and the database itself to capture and analyze the data' (src:wp:database); the database, the DBMS and their applications together make a database system. The word is also used loosely (C-1397).
Properties published scienceour proposal
Sourced fields, and our own readings.
| born at | 1960s: the term 'data-base' in a 1962 SDC report; disks widely available mid-1960s; IDS and IMS (1966) |
| born precision | decade |
| boundary | Inside: the stored data with its structure and semantics ('the data, the structure, and the semantics'), and the DBMS that keeps it: storage engine and indexes, transaction manager and log, query processor and optimizer, catalog and schema, access control. Outside: the hardware and operating system it runs on, the network, the applications and people who use it, the world the data model, attackers, other databases (replicas, warehouses, federations). Whether 'database' names the data, the DBMS or the whole system is contested (C-1397). |
| nested level | Built from the storage engine, indexes, log, transaction manager, optimizer, schema and language; runs on the computer's memory and storage through the operating system; carries applications, websites, warehouses and science archives that run on it. |
| scale | technological |
Measures (21) published science
| mirrored disc pair mean time to failure (realistic analysis) | 800 years |
| characteristic clock: Dynamo service-level agreement (99.9th percentile) | 300 ms |
| characteristic clock: disc mean time to failure | about 1 year |
| disc repair time | about 1 hour |
| unmirrored eight-disc system: unavailable disc | about 8 per year |
| transactions that wait on a lock | 1 in 1000 transactions |
| deadlock rate scaling | square of multiprogramming degree; fourth power of transaction size deadlocks per second |
| cardinality misestimates, all systems tested (per-hand-off toll on the join chain) | factor of 1000 or more, routinely; errors grow exponentially with joins estimate / true size |
| Join Order Benchmark queries | 113 queries, 3 to 16 joins, average 8 joins per query |
| largest IMDB tables in the benchmark | 36 M and 15 M rows |
| learned index vs cache-optimized B-tree | up to 70% faster; an order of magnitude less memory speed, memory |
| divergent versions seen per request (shopping cart, 24 h) | 99.94% one version; 0.00057% two; 0.00047% three; 0.00009% four share of requests |
| NAR Molecular Biology Database Collection size (2025) | 2236 databases |
| NAR collection turnover in one year (2025) | 74 added; 226 discontinued URLs removed; 932 reviewed entries |
| unique molecular-biology databases in NAR Database Issues 1991-2016 | 1727 (in 3055 articles) databases |
| new molecular biology databases per year | over 100 per year |
| databases still available by citation quartile (debuted before 2013) | Q1 155 of 193 (80%); Q2 232 of 338 (69%); Q3 236 of 385 (61%) databases |
| vector search added to RDBMSs after ChatGPT | less than one year |
| Amazon platform scale | tens of millions of customers at peak; tens of thousands of servers customers, servers |
| industry size reached by about 1990 | a billion-dollar industry in about ten years US dollars |
| SQL standardized | ANSI 1986; ISO 1987 year |
Open contentions (4) open contention
Where sources or readings disagree. Both stay on the map.
- C-1389 What exactly does the CAP theorem say?
booked: The map uses the original statement (a network that can split can't guarantee both consistency and availability) and keeps the popular 'pick any two of three' alongside.
the other reading: CAP as 'any two of these guarantees at the same time, but not all three' (encyclopedia) - C-1396 Will relational databases stay on top?
booked: The map records the argument that they keep absorbing their rivals' ideas, and keeps other kinds of database as lasting alternatives alongside.
the other reading: non-relational models (key-value, document, graph, vector, blockchain) as lasting kinds of their own; the encyclopedia: 'In the 2000s, non-relational databases became popular' - C-1397 What is a database?
booked: The map uses the computing sense (organised data kept with management software) and keeps the looser everyday meanings alongside.
the other reading: 'database' used loosely for the DBMS, the database system or an application; outside IT, any collection of related data (a spreadsheet, a card index) - C-1420 What makes a database relational?
booked: The map uses the common meaning (rows and columns, usually with SQL) and keeps the strict reading (all of Codd's rules) alongside.
the other reading: only systems implementing all of Codd's rules are relational; 'Most other DBMS implementations usually called relational are actually SQL DBMSs'
Cross-scale patterns seen here (20) our proposal
Shapes that recur when the same kind of problem is solved at different scales, and how each shows in Database. Each links to every other system that shows it.
- borrowed-then-held (structural)
Two phases shown twice. (1) Bookkeeping history: on cards and tape 'The old master was never updated', partly because 'writing a new tape was easier'; the medium kept the history. With update in place on discs, the database holds it itself, in logs and compensating transactions. (2) Tuning: done by - causal ceiling (structural)
CAP: in a partition-prone network no replica can both answer at once and know the others' latest writes; 'you cannot achieve both safety and liveness in an unreliable distributed system'. A limit set by the system's structure, proved as a theorem. - same role, different builds (structural)
Keeping a business's history so it can be checked and undone: double-entry books on clay and paper, punched-card master runs, logging plus locking, and time-domain addressing ('two apparently different approaches to implementing the transaction concept'). Also: an email system 'performs many of the - same toolkit, independent builds (structural)
From Codd's one paper, separate teams built relational systems: INGRES at Berkeley ('similar to System R in a number of ways'), System R at IBM, Mimer SQL at Uppsala, and Oracle from IBM's papers. - The carry (structural)
Overflow passed on: a sort-merge join's 'sort can spill to disk if necessary' when memory is full; Dynamo passes a write meant for an absent replica on to the next node, which holds it with a hint and hands it back. The wait: latency judged at the 99.9th percentile, the slow tail. The lookahead: the - The Payment to Entropy (structural)
Signal form: each join hands an estimate to the next, and the error compounds: 'misestimates by a factor of 1000 or more', errors that 'grow exponentially' with the number of joins, systematically underestimated. Material form: replicas drift apart and need 'Anti-entropy using Merkle trees'; materia - the whole sets the local reference (structural)
The conceptual level 'unifies the various external views into a compatible global view': every view reads the one schema. In Dynamo the opposite face: each replica's vector clock reads true locally, and versions that do not close into one order are returned together for the application to reconcile. - trade one property for another (structural)
Consistency bought with availability (CAP), with kinds spread along the trade: Dynamo 'sacrifices consistency', other systems 'guarantee strong consistency and provide best effort availability', and large systems segment, each part at its own point. Also: materialized views trade storage for speed; - Umwelt: each system carves its own slice of one shared medium (structural)
Each database carves its own slice of the world: designers decide 'what information is needed in the database, and what can be left out', and each group of users gets its own external view of the one store. - who opens the door (structural)
Every input arrives through a port the DBMS installed ahead of time (APIs, the SQL processor, administrator interfaces). SQL injection uses a door opened by someone else: the web application's own trusted connection to the backend database. - barrier-gates-complexity (partial)
Access control and subschemas let one database serve many groups (payroll, HR) at once; the without-the-barrier case is not shown in the sources read. - holding / entrainment (partial)
The transaction manager decides which transactions live and die: deadlock victims 'must be aborted (using the log) and their locks freed'. The whole sets its parts' deaths; the parts are units of work, not replicators. - partner-becomes-part (partial)
Challenger data models become features of relational systems: vector DBMSs' indexes are 'a feature, not the foundation of a new system architecture', added to RDBMSs within a year; entity-relationship constructs 'retrofitted' until 'the difference between the two has become irrelevant'. A rival rath - red-queen (partial)
SQL injection: 'In spite of the development of different approaches to prevent SQL injection, it still remains an alarming threat'; defences and 'innovative attacks' both advance (survey title). Stalemate stated from one abstract only. - same shape, opposite sign (partial)
One mechanism, the replay of a transaction's threaded log records, run backward to undo ('one undoes each action is its log') or forward to redo ('redo all recent committed actions on the old state'). - seed (partial)
The relational model and SQL: a few operations over relations whose compositions express any query the navigational models could ('There is no loss of expressiveness'), and SQL 'has influenced database languages for other data models'. The combinatorics are named in the sources only loosely. - symmetry breaking (name the cut) (partial)
Codd's relations have no order ('without regard to any ordering between these domains'); the storage engine picks one (sorted arrays, B-trees). The symmetric logical level is cut at the physical level, where the chosen order sets which lookups are fast. - templating (partial)
SQL as the shared module: an ANSI and ISO standard 'supported (with varying degrees of conformance) by all mainstream commercial relational DBMSs'; ODBC as a database-independent API. But 'A database built with one DBMS is not portable to another DBMS', so the swap only partly works. - lightning (surface)
Codd's 1970 paper as the firing that reshaped the field: it 'inspired teams at various universities', IBM's System R and Oracle followed. A sudden idea that changed what came after; no stored difference measured. - runaway from an unstable balance (surface)
Deadlocks 'rise as the square of the degree of multiprogramming and as the fourth power of transaction size': a steep rise, but no balance point or self-feeding runaway is shown.
Valence: what it trades, and with whom (23) published science
Each exchange with a neighbour, read from both sides: who gains, who pays, and what carries it.
- Access control via sys:computer-hardware · Database gain
- Organization via sys:written-record · Database gain
- Human (Homo sapiens) via sys:sensory-environment · Database gain
- Science (as a human social and cultural system) via sys:written-record · Database gain
- Writing (the practice of marking language on a surface, and the script plus rules it works with) via sys:written-record · Database gain
- Software via sys:software · Database gain
- Internet via sys:electromagnetic-field · Database gain
- Large language model via sys:written-record · Database gain
- Computer via sys:computer-hardware · Database gain
- Operating system via sys:computer-hardware · Database gain
- Programming language via sys:written-record · Database mixed
- Cryptographic hash function via sys:written-record · Database gain
- Computer security via sys:written-record · Database loss
- Information security via sys:written-record · Database gain
- Computer hardware via sys:computer-hardware · Database gain
- Computer network via sys:electromagnetic-field · Database gain
- Cloud computing via sys:computer-hardware · Database gain
- Set theory: the study of sets (collections taken as single objects), membership, Cantor's infinities and the axioms, and the practice of the people and machines who work with them via sys:sensory-environment · Database gain
- Security information and event management (SIEM) via sys:written-record · Database gain
- Computer science (the field: the practice and the computation it studies) via sys:sensory-environment · Database gain
- Data structure via sys:computer-hardware · Database gain
- Metaphysics (the branch of philosophy: its practice and the questions about reality it works on) via sys:sensory-environment · Database gain
- Sorting algorithm via sys:computer-hardware · Database gain
Partners and parts published science
Built from / builds (12)
- Database management system (DBMS)
- Storage engine
- Indexes (B-tree, hash, bitmap)
- The log
- Transactions
- Query optimizer
- Schema and catalog
- SQL (and other database languages)
- Database model (relational, and others)
- Transactions competing for locks on shared data
- Attackers against database defenders (SQL injection)
- Security information and event management (SIEM)
Sits inside / holds (4)
expressed by (25)
- Generation/Synthesis
- Recombination/Composition
- Scaffolding/Enablement (Meta-Tooling)
- Templating / Standardization
- Interoperability/Transduction (Bridging)
- Encapsulation & Modularity (Abstraction)
- Evolvability/Plasticity Engineering (Design for Future Variation)
- Distributed Autonomy/Decentralized Execution (Subsidiarity)
- Internal Coordination/Regulation
- Search/Foraging
- Mapping/Modeling
- Navigation/Pathfinding
- Uncertainty Reduction/Learning
- Trail-Marking / Externalized Memory (Stigmergy)
- Offline Simulation & Replay-Guided Planning
- Quick Harm Avoidance
- Reactive Response
- Proactive Defense
- Physical/Virtual Barrier
- Self-Monitoring
- Validation of Identity/Intention
- Resilience & Repair
- Redundancy & Diversity
- Quality Control & Error Correction
- Bridging Supply Gaps (Store & Dormancy)
floor filled by (24)
- Human (Homo sapiens)
- Science (as a human social and cultural system)
- Software
- Operating system
- Cryptographic hash function
- Information security
- Organization
- Computer hardware
- Storage engine
- Indexes (B-tree, hash, bitmap)
- The log
- Transactions
- Query optimizer
- Schema and catalog
- SQL (and other database languages)
- Database model (relational, and others)
- Computer network
- vector clock
- membership gossip
- Cloud computing
- Set theory: the study of sets (collections taken as single objects), membership, Cantor's infinities and the axioms, and the practice of the people and machines who work with them
- Security information and event management (SIEM)
- Data structure
- Sorting algorithm
Kinds (25)
- Knowledge bases and logic
- Relational database
- Navigational database (hierarchical, network/CODASYL)
- Hierarchical database (IMS)
- Network database (CODASYL)
- Object database
- NoSQL database
- Key-value store
- Document-oriented database
- Graph database
- NewSQL
- XML database
- In-memory database
- Distributed database
- Cloud database
- Data warehouse
- Embedded database
- Federated database
- Temporal database
- Spatial database
- Array DBMS
- Vector database
- Blockchain (as a database)
- Biological database (molecular biology)
- Operational database
wired (2)
Notes published science
- Database transactions are older than computers: Jim Gray wrote that 'The transaction concept derives from contract law', and the database log follows the accountants' clay-tablet rule that 'one never alters the books' but adds a compensating entry instead (src:gray-1981).
- Biology starts more than 100 new online databases a year, but they die from lack of people, not disk space: storage costs 'are no longer rate-limiting factors', and in one year the NAR collection added 74 databases while removing 226 dead links (src:imker-2018; src:rigden-fernandez-2025).
- Every major database guesses wrong about its own data: tested on real movie data, all of them 'routinely' misjudged result sizes 'by a factor of 1000 or more', and the errors 'grow exponentially' with each table joined, a toll paid at every hand-off (src:leis-2015).
- Starting from set theory, a machine-checked proof that 2 + 2 = 4 is 'quite complex', yet the same database (set.mm) holds more than 12,000 theorems derived from ZFC, each checked by four verifiers written by four people in four languages. Sources: src:megill-wheeler-2019, src:wp:set-theory.
Layers
The ways it takes in its world, where they meet, and how it acts back, each from L1 (where it meets the world) up.
the boardroom (shared floors)
- L3: Transaction manager: atomicity, consistency, isolation, durability; locks, the log, commit and abort, recovery; replication
- L4: Query processor: the optimizer (cardinality estimates, cost model, join order, plan enumeration) and the executor; in self-driving DBMSs, the planning module
- L5: Conceptual level: the schema, keys and integrity constraints, the catalog, access control and security monitoring
- L6: External level: the views each group of users sees, and the languages and APIs (SQL, ODBC, JDBC) through which they reach it
- L7: The world the data model and the people and organizations who use it: applications, analysts, designers, administrators, scientific communities
the database as database people describe it: data coded as bits on storage up to the world the data model (reader's view)
- L1: Storage: the data coded into bits on disks, flash and main memory (and the stream of records arriving from the world through applications)
- L2: The internal (physical) level: the storage engine's pages, row or column layouts, indexes (B-trees, hashes, bitmaps), materialized views and the log on disk
- L3: Transactions: the transaction manager, locking and concurrency control, logging, commit and abort, crash recovery, replication (ACID)
- L4: Query processing: parser, the query optimizer (cardinality estimates, cost model, join order) and the executor
- L5: The conceptual (logical) level: the schema of relations, keys and integrity constraints, the catalog, and access control
- L6: The external level: views for each group of users, and the languages and APIs that reach them (SQL, ODBC, JDBC)
- L7: The world the data model and the people and organizations that use it: operations, reservations, inventories, libraries, science archives
the replicated database across machines: links between nodes up to reconciled versions (Dynamo-style key-value store)
- L1: Many commodity servers in several data centres, linked by networks; components fail all the time
- L2: Membership and failure detection by gossip; partitioning by consistent hashing
- L3: Replication with sloppy quorums and hinted handoff; anti-entropy with Merkle trees
- L4: Versioning and reconciliation: vector clocks, application-assisted conflict resolution; consistency traded for availability
way in: Access: credentials and privilege grants (DCL)
- L1: A person or program asks for access, giving a password
- L2: Privileges are looked up by individual, group or role
way in: Administration: the database administrator's settings and actions
- L1: The DBA decides: registering users, monitoring performance, tuning
- L2: Dedicated user interfaces for administrators
- L3: Configuration parameters (knobs) tuned statically or dynamically, such as main memory used
way in: Attacks: crafted input (SQL injection) arriving through an application
- L1: An attacker types crafted input into a web application's form
- L2: The web application passes it on to the backend database inside a query
- L4: The injected statement is executed as a query
way in: Deposits: data submitted by a community (scientific archives such as the PDB)
- L1: Structures measured by crystallography, NMR, electron microscopy and micro-electron diffraction in labs on every inhabited continent
way in: Queries: requests for data by content (DQL)
- L1: A question about the world is put as a declarative query: what data is wanted, not where it is
- L2: The query text is received by the language processor
- L3: Because queries are mathematical logic, they can be rewritten in provably correct ways
way in: Replicas: messages from other nodes (gossip, replica writes, Merkle-tree comparisons)
- L1: Messages from other servers over the network; routes flap and data centres fail
- L2: Gossip-based membership and failure detection; hinted writes held for absent replicas
way in: Schema: data definition from designers and administrators (DDL)
- L1: The designer's conceptual data model of the application domain (entities, relationships, attributes)
- L2: Data definition statements create, alter or drop tables and their relationships
- L3: The definitions are written into the catalog
- L4: Physical design: indexes and layout chosen for the expected workload, invisible to applications
way in: Self: statistics about its own data (cardinalities, histograms)
- L1: The stored data's own distribution: how many rows match, which values cluster together
- L2: Statistics gathered by a statistics tool: histograms, samples, column-group statistics
- L3: Cardinality estimates for each intermediate result
way in: Self: the workload over time (what queries arrive, how often)
- L1: The stream of queries and transactions arriving through the day
- L2: Queries classified (OLTP or OLAP) and clustered
- L3: Workload forecasting models (recurrent neural networks, LSTM) predict future demand
way in: Storage: the medium's own state (I/O completions, bad spots, failed discs, lost memory)
- L1: Discs and memory: a typical disc fails about once a year (bad spots, spindle, the path to the disc)
- L2: Fail-fast modules report: either they work or they fail and do nothing
way in: Time: the clock, the order of events, and the human day
- L1: Time passing in the world: users' days and nights shape the load
- L2: Time stamped into the data: valid-time and transaction-time in temporal databases; vector clocks for versions
way in: Writes: inserts, updates and deletes from applications and people (DML)
- L1: A fact in the world becomes a statement sent by an application or typed by a person: insert, update, delete
- L2: The statement arrives through the DBMS's API or language processor
way out: Abort and undo; reject what breaks the rules
- L1: The stored state is as if the transaction never ran
- L3: Undo each action in the transaction's log; free its locks
way out: Back up, and restore a previous state
- L1: The files are used to restore that state
- L3: Each desired database state is kept within dedicated backup files
way out: Commit: make a change durable on storage
- L1: Bits on durable storage
- L2: Log records and pages are written by the storage engine
- L3: The transaction commits: all its effects survive, or it aborts and none do
way out: Return results to the application and the person
- L1: Rows appear on a screen or flow into the application
- L4: The executor computes the query result along the chosen plan
- L6: Results are presented through the user's view, by an SQL tool or a website
way out: Self-driving actions on its own physical design
- L2: Actions: add or drop indexes and materialized views, row to columnar layouts, compress, repartition, move tiers, set knobs
- L4: The planner chooses an action sequence by estimated cost-benefit
way out: Send updates to replicas
- L1: Writes travel to the N hosts that hold the key; Merkle-tree comparisons run in the background
- L3: Updates of a replicated object are synchronized across the copies
way out: Triggers and alerts (active databases)
- L1: Security monitoring, alerting, statistics gathering, authorization
- L5: An event-driven rule fires on a condition inside or outside the database
way out: Write the audit trail
- L1: A record kept for a later forensic audit
- L5: Change and access logging records who accessed which attributes, what changed, and when