File format

A database is a folder:

shop.ndbc/
  nardb.toml            format version, page size, wrapped data key
  nardb.lock            single-owner lock
  wal.log               write-ahead log
  catalog.db            catalog document and per-table counters
  indexes.db            secondary indexes, across partitions
  schemas/s0001/p0.db   partition 0 of schema 1
  schemas/s0001/p1.db   ...

File names carry no meaning: schemas are named by ids such as s0001, and their names live inside the catalog. Partition files are created with their first row and removed when the schema is dropped.

Pages#

Every .db file is made of equal-sized pages. The page size is 4 KiB by default and can be any power of two from 4 to 64 KiB. All integers are little-endian.

Page 0 is the file header: magic (NARDB\r\n\x1a), format version, page size, feature flags (a file with an unknown flag is not opened), page count, the free-page list, a root page and a CRC-32C checksum.

In an encrypted database every page except page 0 is encrypted on disk (see Encryption); the layout below is the decrypted page.

Every other page starts with a 16-byte header: the page type and a CRC-32C checksum of the page. A page whose checksum does not match is not read; the engine returns ErrCorrupt and does not try to repair it.

The write-ahead log and transactions#

Every statement is one transaction. All pages a statement changes, in any file, are first kept inside the transaction. A commit happens in this order:

1. Full images of the changed pages and a commit record are written to wal.log and flushed to disk (fsync).
2. Only then does the change become visible.
3. When the log reaches 8 MiB, or when the database closes, the pages are copied into their files (checkpoint) and the log is emptied.

On open the log is scanned. Transactions that end with a commit record are restored; a half-written last transaction is dropped. Every log record has its own checksum and carries a salt that changes on every reset, so records from an earlier generation cannot mix with the new log.

As a result:

• After a sudden shutdown every committed statement comes back.
• When one statement changes the catalog, a partition and an index, all three are applied together or none is.
• A statement that fails has written nothing.

Cache#

Committed pages are kept in memory, least recently used first out. The default is 4096 pages, which is 16 MiB with 4 KiB pages. Reads look at the transaction's own changes first, then the cache, then the log, then the file.

The catalog#

Schemas, users, pool settings and the distribution map form a versioned document stored as a chain of overflow pages in catalog.db; the header's root page points to it. Catalog changes are written through transactions like everything else.

Tables#

• Each partition file holds a primary key tree (key → record) whose root is always page 1.
• The partition key decides which partition a row goes to: (value − start) / period. Without partitioning every row lives in p0.db.
• Secondary indexes live in indexes.db and span all partitions. The key is value ‖ primary key and the value is the row's partition number, so uniqueness is checked across partitions and an index lookup goes straight to the right file.
• When the primary key is not the partition key, a primary key lookup checks every partition file.

A record is a list of (field id, length, value) triples, and null fields are not written. As a result:

• the bytes of dropped fields are skipped when reading,
• fields added later read as their default in old records,
• widenings such as int8 → int64 and string → text need no rewrite.

Index keys use an order-preserving encoding: byte order equals value order.

Limits and known gaps#

• Every commit is flushed to disk (fsync), so many small statements are slow. Group commit does not exist yet.
• Leaves are not merged after deletes; empty pages will be collected by a future vacuum.
• Only one process can open a database at a time. The lock is taken with flock on unix systems. The Windows lock is not written yet, so databases do not open on Windows.
• Single-file .ndbc databases from older versions are converted to this layout when opened; the old file is kept with a .v1 suffix.