maple/maple-py/re/KNOWN_DIVERGENCE.md

KNOWN_DIVERGENCE.md — fields maple's writer intentionally does NOT match

This is the canonical list of bytes that differ between maple's captures/maple_acemo.pdb and the AceMo reference data/acemo/PIONEER/rekordbox/export.pdb on purpose, because they are history-dependent artifacts of the particular reference file rather than rules of the .pdb format.

TL;DR

The AceMo reference export.pdb is not a fresh from-scratch export — it is an incrementally-edited rekordbox database. rekordbox's DeviceSQL page manager edits rows in place: when a row changes it appends a new copy, clears the old slot's presence bit, and advances used_size past the dead bytes without compacting. The reference therefore carries the residue of its entire edit history.

maple writes a fresh, fully-compacted database (tightly 4-byte-aligned rows, no dead rows, no leftover slots). That output is byte-correct for a from-scratch rekordbox export, but it cannot — and should not — reproduce another file's edit history. The fields below are the consequences of that, and the team has decided not to chase them. They only become matchable against a fresh from-scratch reference export (export to an empty USB, never edited afterward).

Everything NOT listed here is fair game and must match. The task-#7 diff should filter out (and reference) this list, then drive the remaining bytes to zero.

Evidence (tracks table, page 2 of the reference)

5 present rows, each with ~377–417 bytes of real content, yet:

row id	ofs	real content len	next ofs	gap	gap bytes are…
1	428	377	856	51	zero
5	856	377	1324	91	partial stale string
2	1324	417	1792	51	zero
3	1792	386	2656	478	a whole dead track row
4	2656	404	3996*	936	two whole dead track rows

used_size=3996, free_size=38. The gap after id=3 holds a dead copy of a row ending "...Jelly Fish Jam.mp3"; the gap after id=4 holds dead copies of "Myrtle Ave Party Track" and "Smooth Move" rows. These are previous versions of edited rows, left uncompacted. (Verified by ASCII-dumping the gap bytes.)

The fields we do NOT match (and why)

1. File / page count and next_unused_page. Reference: 49 pages, next_unused=53. maple: 41 pages, next_unused=41. The reference is larger purely because dead-row bloat pushed live rows onto extra pages. A compacted export needs fewer pages.

2. Overflow / data-page placement (tracks last_page). Reference tracks table spans pages 1 → 48 (live track id=7 spilled onto overflow page 48 because dead rows "filled" page 2). maple spans 1 → 2 (all 6 tracks fit one compacted page). This is the only table-directory entry that differs — the other 19 tables match first/last page exactly. The divergence is entirely a function of the dead-row bloat above.

3. File-header sequence value. Reference: 353. maple: an incrementing counter (reads the destination export's sequence + 1, default 1; see writer._next_sequence). rekordbox's value comes from its internal master.db transaction counter and is not reproducible from content. We match the semantics (monotonic increment), not the absolute value.

4. Per-page sequence (page-header offset 16). Reference pages carry assorted values (e.g. 26, 45, 320, 342, 347) — each a snapshot of the global counter at the moment that page was last rewritten. maple writes a clean per-build value. Same root cause as (3); not reproducible.

5. transaction_row_count / transaction_row_index (page-header u2 pair). Reference: nonzero per edited page (e.g. 1/5, 1/8, 2/0). maple: 0/0. These track the last in-place transaction on the page; meaningless for a freshly written, never-edited page.

6. Dead-row offset arrays + num_row_offsets > num_rows. Reference row groups list offsets for dead slots too: e.g. tracks page 2 has nro=9, nrows=5 with offsets [0,428,856,1324,1792,2224,2656,3112,3568] (slots 0,5,7,8 dead); artists page 6 has nro=9, nrows=1. maple emits exactly nro == nrows with no dead slots. Reproducing dead offsets would mean reproducing the edit history.

7. Live row in a non-zero slot. Reference artists page 6 keeps its single live row in slot 8 (slots 0–7 are dead). maple places the single live row in slot 0. A from-scratch export also writes slot 0.

8. free_size / used_size totals on data pages. Reference used+free totals vary (4034, 4050, …) because used_size covers the dead-row region and the strange-block markers differ. maple's pages total a clean 4020 (= heap capacity for a single 36-byte row group). Direct consequence of (6) — dead bytes are counted as "used".

9. Track row opaque id blob — 0x14/0x18/0x1a (8 bytes/track). One 8-byte LE value opaque_track_id_lo | (opaque_seed << 32), both halves stored in rekordbox's internal master.db (a per-install opaque_seed + a per-track opaque_track_id_lo). Proven non-derivable from track metadata across every real export + an exhaustive negative hash search (FORMAT_SPEC.md s2). Every real rekordbox export has BOTH halves non-zero (this install's seed = 0x21b4a7ab; data/MAPLE = 0xd6d3a567); the "unanalyzed default" lo=0/seed=0x78f74a68 was a false lead — it came from maple's own earlier output (data/usb), not a real export. maple emits a plausible non-zero stand-in of the right shape: opaque_seed = 0x21b4a7ab (module const TRACK_OPAQUE_SEED, overridable per install) and a stable per-track opaque_track_id_lo mapped into the real Content-ID band (OPAQUE_LO_MIN..MAX = 0x08000000..0x0F900000, where every observed AceMo lo lands) via crc32(id|file_path). This will NOT match rekordbox's master.db id byte-for-byte — unreproducible without its DB — but it is non-zero, in-range and deterministic, the safest shape for deck acceptance. Players treat the blob as opaque.

10. Strange-page free-map extension + per-page seq on tracks/history. On the tracks (type 0) and history (type 19) strange pages only, the free-space map carries an extra accounting word (body offset 16: 01 00, offset 20: a u4 = the data page's used_size for history=320; tracks differs because of its dead-row/overflow state). maple writes the plain free-map. Tied to the same content/overflow state as items (1)/(2); not reproducible for a compacted writer. (The rest of every strange page — header, free-map head, 1004× f8 ff ff 1f, the data-page pointer word, and unknown7 — now matches the fresh reference byte-for-byte.)

11. Table-directory empty_candidate (u32, table_dir entry +4) + header next_unused_page. Reference: a nonzero per-table empty_candidate (e.g. 0x33, 0x04, 0x2e…); maple: 0. rekordcrate documents this field as "maybe links to a chain of empty pages if the database is ever garbage collected", and its fixtures show non-sequential free-pool page indices (47, 4, 49, 8, 50, 46, 42, 16…). It is free-list/GC allocator state, not derivable from content. next_unused_page (rekordcrate: "points past end of file", unused) is the same class — see also item (1).

Canonical reference note: the diff target is now the FRESH export data/acemo_fresh/PIONEER/rekordbox/export.pdb (7 tracks). It is cleaner than the edited data/acemo (6 tracks) but still incrementally built per-file, so the divergences above still apply — just smaller (e.g. track 7's dead-row span is 464B not 3568B, but it still overflows to page 48).

What IS reproducible (NOW matched against the fresh reference)

• The 20-table directory layout and page placement: an empty table is strange-page-only; a table gets a data page IFF it has rows (CORE_TABLES is now empty — the old genres data page was an edit-history artifact of the edited ref).
• page_flags: data pages are 0x34 for tracks and history only, 0x24 for every other table (the wider {0,2,3,13,17,19} was edited-ref noise); empty data pages 0x44; strange pages 0x64.
• Strange-page body: the full DeviceSQL free-space map — u4(page_index), u4(first_data_page or 0x03ffffff), the 12-byte const tail, 1004× f8 ff ff 1f, 20 zero bytes — plus header free=used=0, txc=txi=0x1fff, next_page = page_index+1, and unknown7 = 1 for tracks/history else 0.
• Track row index_shift (slot_index<<5) and every other live-row content field except the opaque blob in divergence item (9).
• Three header fields re decoded (FORMAT_SPEC §4/§5/§6) as edit-history state whose FRESH-export value maple already writes — so they're matched on a fresh export and only diverge against edited refs:
  • file-header offset 16 (_unnamed4): fresh=0 (maple writes 0); edited DBs accumulate a small nonzero with no structural meaning.
  • strange/data-page unknown7 (0x26): fresh=0 everywhere (maple writes 0); nonzero only on edited DBs' strange pages.
  • row-group trailing u2: it is the "rows-touched-in-last-transaction" bitmask. On a fresh export it equals the row-present mask, so maple writes presence (matches 12/14 groups in data/acemo_fresh; the 2 misses are edited pages — divergence). NOTE: re's "keep constant 16" advice came from reading data/usb, which is maple's OWN output, not a real export — a real fresh export uses the presence value, not 16.

See re/FORMAT_SPEC.md for the value-construction rules and re/README.md for the open RE questions.