# KNOWN_DIVERGENCE.md — fields maple's writer intentionally does NOT match

This is the canonical list of bytes that differ between maple's
`captures/maple_acemo.pdb` and the AceMo reference
`data/acemo/PIONEER/rekordbox/export.pdb` **on purpose**, because they are
*history-dependent* artifacts of the particular reference file rather than rules
of the `.pdb` format.

## TL;DR

The AceMo reference `export.pdb` is **not a fresh from-scratch export — it is an
incrementally-edited rekordbox database.** rekordbox's DeviceSQL page manager
edits rows *in place*: when a row changes it appends a new copy, clears the old
slot's presence bit, and advances `used_size` past the dead bytes **without
compacting**. The reference therefore carries the residue of its entire edit
history.

maple writes a **fresh, fully-compacted** database (tightly 4-byte-aligned rows,
no dead rows, no leftover slots). That output is byte-correct for a *from-scratch*
rekordbox export, but it cannot — and should not — reproduce another file's edit
history. The fields below are the consequences of that, and the team has decided
**not to chase them**. They only become matchable against a fresh from-scratch
reference export (export to an empty USB, never edited afterward).

Everything NOT listed here is fair game and must match. The task-#7 diff should
filter out (and reference) this list, then drive the remaining bytes to zero.

## Evidence (tracks table, page 2 of the reference)

5 present rows, each with ~377–417 bytes of real content, yet:

| row id | ofs   | real content len | next ofs | gap   | gap bytes are…            |
|--------|-------|------------------|----------|-------|---------------------------|
| 1      | 428   | 377              | 856      | 51    | zero                      |
| 5      | 856   | 377              | 1324     | 91    | partial stale string      |
| 2      | 1324  | 417              | 1792     | 51    | zero                      |
| 3      | 1792  | 386              | 2656     | 478   | a whole dead track row    |
| 4      | 2656  | 404              | 3996*    | 936   | two whole dead track rows |

`used_size=3996`, `free_size=38`. The gap after id=3 holds a dead copy of a row
ending `"...Jelly Fish Jam.mp3"`; the gap after id=4 holds dead copies of
`"Myrtle Ave Party Track"` and `"Smooth Move"` rows. These are previous versions
of edited rows, left uncompacted. (Verified by ASCII-dumping the gap bytes.)

## The fields we do NOT match (and why)

1. **File / page count and `next_unused_page`.**
   Reference: 49 pages, `next_unused=53`. maple: 41 pages, `next_unused=41`.
   The reference is larger purely because dead-row bloat pushed live rows onto
   extra pages. A compacted export needs fewer pages.

2. **Overflow / data-page placement (tracks `last_page`).**
   Reference tracks table spans pages **1 → 48** (live track id=7 spilled onto
   overflow page 48 because dead rows "filled" page 2). maple spans **1 → 2**
   (all 6 tracks fit one compacted page). This is the *only* table-directory
   entry that differs — the other 19 tables match first/last page exactly. The
   divergence is entirely a function of the dead-row bloat above.

3. **File-header `sequence` value.**
   Reference: `353`. maple: an incrementing counter (reads the destination
   export's sequence + 1, default 1; see `writer._next_sequence`). rekordbox's
   value comes from its internal `master.db` transaction counter and is not
   reproducible from content. We match the *semantics* (monotonic increment),
   not the absolute value.

4. **Per-page `sequence` (page-header offset 16).**
   Reference pages carry assorted values (e.g. 26, 45, 320, 342, 347) — each a
   snapshot of the global counter at the moment that page was last rewritten.
   maple writes a clean per-build value. Same root cause as (3); not reproducible.

5. **`transaction_row_count` / `transaction_row_index` (page-header u2 pair).**
   Reference: nonzero per edited page (e.g. `1/5`, `1/8`, `2/0`). maple: `0/0`.
   These track the last in-place transaction on the page; meaningless for a
   freshly written, never-edited page.

6. **Dead-row offset arrays + `num_row_offsets` > `num_rows`.**
   Reference row groups list offsets for dead slots too: e.g. tracks page 2 has
   `nro=9, nrows=5` with offsets `[0,428,856,1324,1792,2224,2656,3112,3568]`
   (slots 0,5,7,8 dead); artists page 6 has `nro=9, nrows=1`. maple emits exactly
   `nro == nrows` with no dead slots. Reproducing dead offsets would mean
   reproducing the edit history.

7. **Live row in a non-zero slot.**
   Reference artists page 6 keeps its single live row in **slot 8** (slots 0–7
   are dead). maple places the single live row in **slot 0**. A from-scratch
   export also writes slot 0.

8. **`free_size` / `used_size` totals on data pages.**
   Reference `used+free` totals vary (4034, 4050, …) because `used_size` covers
   the dead-row region and the strange-block markers differ. maple's pages total
   a clean `4020` (= heap capacity for a single 36-byte row group). Direct
   consequence of (6) — dead bytes are counted as "used".

9. **Track row opaque id blob — `0x14`/`0x18`/`0x1a` (8 bytes/track).**
   One 8-byte LE value `opaque_track_id_lo | (opaque_seed << 32)`, both halves
   stored in rekordbox's internal master.db (a per-install `opaque_seed` + a
   per-track `opaque_track_id_lo`). Proven non-derivable from track metadata
   across every real export + an exhaustive negative hash search
   (FORMAT_SPEC.md s2). **Every real rekordbox export has BOTH halves non-zero**
   (this install's seed = `0x21b4a7ab`; `data/MAPLE` = `0xd6d3a567`); the
   "unanalyzed default" lo=0/seed=0x78f74a68 was a false lead — it came from
   maple's own earlier output (`data/usb`), not a real export. maple emits a
   plausible non-zero stand-in of the right shape: `opaque_seed = 0x21b4a7ab`
   (module const `TRACK_OPAQUE_SEED`, overridable per install) and a stable
   per-track `opaque_track_id_lo` mapped into the real Content-ID band
   (`OPAQUE_LO_MIN..MAX` = 0x08000000..0x0F900000, where every observed AceMo lo
   lands) via `crc32(id|file_path)`. This will NOT match rekordbox's master.db
   id byte-for-byte — unreproducible without its DB — but it is non-zero,
   in-range and deterministic, the safest shape for deck acceptance. Players
   treat the blob as opaque.

10. **Strange-page free-map extension + per-page seq on tracks/history.**
    On the tracks (type 0) and history (type 19) strange pages only, the
    free-space map carries an extra accounting word (body offset 16: `01 00`,
    offset 20: a u4 = the data page's used_size for history=320; tracks differs
    because of its dead-row/overflow state). maple writes the plain free-map.
    Tied to the same content/overflow state as items (1)/(2); not reproducible
    for a compacted writer. (The rest of every strange page — header, free-map
    head, 1004× `f8 ff ff 1f`, the data-page pointer word, and unknown7 — now
    matches the fresh reference byte-for-byte.)

11. **Table-directory `empty_candidate` (u32, table_dir entry +4) + header
    `next_unused_page`.** Reference: a nonzero per-table `empty_candidate`
    (e.g. 0x33, 0x04, 0x2e…); maple: 0. rekordcrate documents this field as
    "maybe links to a chain of empty pages if the database is ever garbage
    collected", and its fixtures show non-sequential free-pool page indices
    (47, 4, 49, 8, 50, 46, 42, 16…). It is free-list/GC allocator state, not
    derivable from content. `next_unused_page` (rekordcrate: "points past end of
    file", unused) is the same class — see also item (1).

> Canonical reference note: the diff target is now the FRESH export
> `data/acemo_fresh/PIONEER/rekordbox/export.pdb` (7 tracks). It is cleaner than
> the edited `data/acemo` (6 tracks) but still incrementally built per-file, so
> the divergences above still apply — just smaller (e.g. track 7's dead-row span
> is 464B not 3568B, but it still overflows to page 48).

## What IS reproducible (NOW matched against the fresh reference)

- The 20-table directory layout and page placement: an empty table is
  strange-page-only; a table gets a data page IFF it has rows (`CORE_TABLES` is
  now empty — the old genres data page was an edit-history artifact of the
  edited ref).
- `page_flags`: data pages are `0x34` for **tracks and history only**, `0x24`
  for every other table (the wider `{0,2,3,13,17,19}` was edited-ref noise);
  empty data pages `0x44`; strange pages `0x64`.
- Strange-page body: the full DeviceSQL free-space map — `u4(page_index)`,
  `u4(first_data_page or 0x03ffffff)`, the 12-byte const tail, 1004×
  `f8 ff ff 1f`, 20 zero bytes — plus header `free=used=0`, `txc=txi=0x1fff`,
  `next_page = page_index+1`, and `unknown7 = 1` for tracks/history else 0.
- Track row `index_shift` (`slot_index<<5`) and every other live-row content
  field except the opaque blob in divergence item (9).
- Three header fields `re` decoded (FORMAT_SPEC §4/§5/§6) as edit-history state
  whose FRESH-export value maple already writes — so they're matched on a fresh
  export and only diverge against edited refs:
  - file-header offset 16 (`_unnamed4`): fresh=`0` (maple writes 0); edited DBs
    accumulate a small nonzero with no structural meaning.
  - strange/data-page `unknown7` (0x26): fresh=`0` everywhere (maple writes 0);
    nonzero only on edited DBs' strange pages.
  - row-group trailing u2: it is the "rows-touched-in-last-transaction" bitmask.
    On a fresh export it equals the row-present mask, so maple writes
    `presence` (matches 12/14 groups in `data/acemo_fresh`; the 2 misses are
    edited pages — divergence). NOTE: `re`'s "keep constant 16" advice came from
    reading `data/usb`, which is maple's OWN output, not a real export — a real
    fresh export uses the presence value, not 16.

See `re/FORMAT_SPEC.md` for the value-construction rules and `re/README.md` for
the open RE questions.