the downfall has begun

maple-py/re/KNOWN_DIVERGENCE.md

@@ -0,0 +1,167 @@
+# KNOWN_DIVERGENCE.md — fields maple's writer intentionally does NOT match
+
+This is the canonical list of bytes that differ between maple's
+`captures/maple_acemo.pdb` and the AceMo reference
+`data/acemo/PIONEER/rekordbox/export.pdb` **on purpose**, because they are
+*history-dependent* artifacts of the particular reference file rather than rules
+of the `.pdb` format.
+
+## TL;DR
+
+The AceMo reference `export.pdb` is **not a fresh from-scratch export — it is an
+incrementally-edited rekordbox database.** rekordbox's DeviceSQL page manager
+edits rows *in place*: when a row changes it appends a new copy, clears the old
+slot's presence bit, and advances `used_size` past the dead bytes **without
+compacting**. The reference therefore carries the residue of its entire edit
+history.
+
+maple writes a **fresh, fully-compacted** database (tightly 4-byte-aligned rows,
+no dead rows, no leftover slots). That output is byte-correct for a *from-scratch*
+rekordbox export, but it cannot — and should not — reproduce another file's edit
+history. The fields below are the consequences of that, and the team has decided
+**not to chase them**. They only become matchable against a fresh from-scratch
+reference export (export to an empty USB, never edited afterward).
+
+Everything NOT listed here is fair game and must match. The task-#7 diff should
+filter out (and reference) this list, then drive the remaining bytes to zero.
+
+## Evidence (tracks table, page 2 of the reference)
+
+5 present rows, each with ~377–417 bytes of real content, yet:
+
+| row id | ofs   | real content len | next ofs | gap   | gap bytes are…            |
+|--------|-------|------------------|----------|-------|---------------------------|
+| 1      | 428   | 377              | 856      | 51    | zero                      |
+| 5      | 856   | 377              | 1324     | 91    | partial stale string      |
+| 2      | 1324  | 417              | 1792     | 51    | zero                      |
+| 3      | 1792  | 386              | 2656     | 478   | a whole dead track row    |
+| 4      | 2656  | 404              | 3996*    | 936   | two whole dead track rows |
+
+`used_size=3996`, `free_size=38`. The gap after id=3 holds a dead copy of a row
+ending `"...Jelly Fish Jam.mp3"`; the gap after id=4 holds dead copies of
+`"Myrtle Ave Party Track"` and `"Smooth Move"` rows. These are previous versions
+of edited rows, left uncompacted. (Verified by ASCII-dumping the gap bytes.)
+
+## The fields we do NOT match (and why)
+
+1. **File / page count and `next_unused_page`.**
+   Reference: 49 pages, `next_unused=53`. maple: 41 pages, `next_unused=41`.
+   The reference is larger purely because dead-row bloat pushed live rows onto
+   extra pages. A compacted export needs fewer pages.
+
+2. **Overflow / data-page placement (tracks `last_page`).**
+   Reference tracks table spans pages **1 → 48** (live track id=7 spilled onto
+   overflow page 48 because dead rows "filled" page 2). maple spans **1 → 2**
+   (all 6 tracks fit one compacted page). This is the *only* table-directory
+   entry that differs — the other 19 tables match first/last page exactly. The
+   divergence is entirely a function of the dead-row bloat above.
+
+3. **File-header `sequence` value.**
+   Reference: `353`. maple: an incrementing counter (reads the destination
+   export's sequence + 1, default 1; see `writer._next_sequence`). rekordbox's
+   value comes from its internal `master.db` transaction counter and is not
+   reproducible from content. We match the *semantics* (monotonic increment),
+   not the absolute value.
+
+4. **Per-page `sequence` (page-header offset 16).**
+   Reference pages carry assorted values (e.g. 26, 45, 320, 342, 347) — each a
+   snapshot of the global counter at the moment that page was last rewritten.
+   maple writes a clean per-build value. Same root cause as (3); not reproducible.
+
+5. **`transaction_row_count` / `transaction_row_index` (page-header u2 pair).**
+   Reference: nonzero per edited page (e.g. `1/5`, `1/8`, `2/0`). maple: `0/0`.
+   These track the last in-place transaction on the page; meaningless for a
+   freshly written, never-edited page.
+
+6. **Dead-row offset arrays + `num_row_offsets` > `num_rows`.**
+   Reference row groups list offsets for dead slots too: e.g. tracks page 2 has
+   `nro=9, nrows=5` with offsets `[0,428,856,1324,1792,2224,2656,3112,3568]`
+   (slots 0,5,7,8 dead); artists page 6 has `nro=9, nrows=1`. maple emits exactly
+   `nro == nrows` with no dead slots. Reproducing dead offsets would mean
+   reproducing the edit history.
+
+7. **Live row in a non-zero slot.**
+   Reference artists page 6 keeps its single live row in **slot 8** (slots 0–7
+   are dead). maple places the single live row in **slot 0**. A from-scratch
+   export also writes slot 0.
+
+8. **`free_size` / `used_size` totals on data pages.**
+   Reference `used+free` totals vary (4034, 4050, …) because `used_size` covers
+   the dead-row region and the strange-block markers differ. maple's pages total
+   a clean `4020` (= heap capacity for a single 36-byte row group). Direct
+   consequence of (6) — dead bytes are counted as "used".
+
+9. **Track row opaque id blob — `0x14`/`0x18`/`0x1a` (8 bytes/track).**
+   One 8-byte LE value `opaque_track_id_lo | (opaque_seed << 32)`, both halves
+   stored in rekordbox's internal master.db (a per-install `opaque_seed` + a
+   per-track `opaque_track_id_lo`). Proven non-derivable from track metadata
+   across every real export + an exhaustive negative hash search
+   (FORMAT_SPEC.md s2). **Every real rekordbox export has BOTH halves non-zero**
+   (this install's seed = `0x21b4a7ab`; `data/MAPLE` = `0xd6d3a567`); the
+   "unanalyzed default" lo=0/seed=0x78f74a68 was a false lead — it came from
+   maple's own earlier output (`data/usb`), not a real export. maple emits a
+   plausible non-zero stand-in of the right shape: `opaque_seed = 0x21b4a7ab`
+   (module const `TRACK_OPAQUE_SEED`, overridable per install) and a stable
+   per-track `opaque_track_id_lo` mapped into the real Content-ID band
+   (`OPAQUE_LO_MIN..MAX` = 0x08000000..0x0F900000, where every observed AceMo lo
+   lands) via `crc32(id|file_path)`. This will NOT match rekordbox's master.db
+   id byte-for-byte — unreproducible without its DB — but it is non-zero,
+   in-range and deterministic, the safest shape for deck acceptance. Players
+   treat the blob as opaque.
+
+10. **Strange-page free-map extension + per-page seq on tracks/history.**
+    On the tracks (type 0) and history (type 19) strange pages only, the
+    free-space map carries an extra accounting word (body offset 16: `01 00`,
+    offset 20: a u4 = the data page's used_size for history=320; tracks differs
+    because of its dead-row/overflow state). maple writes the plain free-map.
+    Tied to the same content/overflow state as items (1)/(2); not reproducible
+    for a compacted writer. (The rest of every strange page — header, free-map
+    head, 1004× `f8 ff ff 1f`, the data-page pointer word, and unknown7 — now
+    matches the fresh reference byte-for-byte.)
+
+11. **Table-directory `empty_candidate` (u32, table_dir entry +4) + header
+    `next_unused_page`.** Reference: a nonzero per-table `empty_candidate`
+    (e.g. 0x33, 0x04, 0x2e…); maple: 0. rekordcrate documents this field as
+    "maybe links to a chain of empty pages if the database is ever garbage
+    collected", and its fixtures show non-sequential free-pool page indices
+    (47, 4, 49, 8, 50, 46, 42, 16…). It is free-list/GC allocator state, not
+    derivable from content. `next_unused_page` (rekordcrate: "points past end of
+    file", unused) is the same class — see also item (1).
+
+> Canonical reference note: the diff target is now the FRESH export
+> `data/acemo_fresh/PIONEER/rekordbox/export.pdb` (7 tracks). It is cleaner than
+> the edited `data/acemo` (6 tracks) but still incrementally built per-file, so
+> the divergences above still apply — just smaller (e.g. track 7's dead-row span
+> is 464B not 3568B, but it still overflows to page 48).
+
+## What IS reproducible (NOW matched against the fresh reference)
+
+- The 20-table directory layout and page placement: an empty table is
+  strange-page-only; a table gets a data page IFF it has rows (`CORE_TABLES` is
+  now empty — the old genres data page was an edit-history artifact of the
+  edited ref).
+- `page_flags`: data pages are `0x34` for **tracks and history only**, `0x24`
+  for every other table (the wider `{0,2,3,13,17,19}` was edited-ref noise);
+  empty data pages `0x44`; strange pages `0x64`.
+- Strange-page body: the full DeviceSQL free-space map — `u4(page_index)`,
+  `u4(first_data_page or 0x03ffffff)`, the 12-byte const tail, 1004×
+  `f8 ff ff 1f`, 20 zero bytes — plus header `free=used=0`, `txc=txi=0x1fff`,
+  `next_page = page_index+1`, and `unknown7 = 1` for tracks/history else 0.
+- Track row `index_shift` (`slot_index<<5`) and every other live-row content
+  field except the opaque blob in divergence item (9).
+- Three header fields `re` decoded (FORMAT_SPEC §4/§5/§6) as edit-history state
+  whose FRESH-export value maple already writes — so they're matched on a fresh
+  export and only diverge against edited refs:
+  - file-header offset 16 (`_unnamed4`): fresh=`0` (maple writes 0); edited DBs
+    accumulate a small nonzero with no structural meaning.
+  - strange/data-page `unknown7` (0x26): fresh=`0` everywhere (maple writes 0);
+    nonzero only on edited DBs' strange pages.
+  - row-group trailing u2: it is the "rows-touched-in-last-transaction" bitmask.
+    On a fresh export it equals the row-present mask, so maple writes
+    `presence` (matches 12/14 groups in `data/acemo_fresh`; the 2 misses are
+    edited pages — divergence). NOTE: `re`'s "keep constant 16" advice came from
+    reading `data/usb`, which is maple's OWN output, not a real export — a real
+    fresh export uses the presence value, not 16.
+
+See `re/FORMAT_SPEC.md` for the value-construction rules and `re/README.md` for
+the open RE questions.