168 lines
9.8 KiB
Markdown
168 lines
9.8 KiB
Markdown
# KNOWN_DIVERGENCE.md — fields maple's writer intentionally does NOT match
|
||
|
||
This is the canonical list of bytes that differ between maple's
|
||
`captures/maple_acemo.pdb` and the AceMo reference
|
||
`data/acemo/PIONEER/rekordbox/export.pdb` **on purpose**, because they are
|
||
*history-dependent* artifacts of the particular reference file rather than rules
|
||
of the `.pdb` format.
|
||
|
||
## TL;DR
|
||
|
||
The AceMo reference `export.pdb` is **not a fresh from-scratch export — it is an
|
||
incrementally-edited rekordbox database.** rekordbox's DeviceSQL page manager
|
||
edits rows *in place*: when a row changes it appends a new copy, clears the old
|
||
slot's presence bit, and advances `used_size` past the dead bytes **without
|
||
compacting**. The reference therefore carries the residue of its entire edit
|
||
history.
|
||
|
||
maple writes a **fresh, fully-compacted** database (tightly 4-byte-aligned rows,
|
||
no dead rows, no leftover slots). That output is byte-correct for a *from-scratch*
|
||
rekordbox export, but it cannot — and should not — reproduce another file's edit
|
||
history. The fields below are the consequences of that, and the team has decided
|
||
**not to chase them**. They only become matchable against a fresh from-scratch
|
||
reference export (export to an empty USB, never edited afterward).
|
||
|
||
Everything NOT listed here is fair game and must match. The task-#7 diff should
|
||
filter out (and reference) this list, then drive the remaining bytes to zero.
|
||
|
||
## Evidence (tracks table, page 2 of the reference)
|
||
|
||
5 present rows, each with ~377–417 bytes of real content, yet:
|
||
|
||
| row id | ofs | real content len | next ofs | gap | gap bytes are… |
|
||
|--------|-------|------------------|----------|-------|---------------------------|
|
||
| 1 | 428 | 377 | 856 | 51 | zero |
|
||
| 5 | 856 | 377 | 1324 | 91 | partial stale string |
|
||
| 2 | 1324 | 417 | 1792 | 51 | zero |
|
||
| 3 | 1792 | 386 | 2656 | 478 | a whole dead track row |
|
||
| 4 | 2656 | 404 | 3996* | 936 | two whole dead track rows |
|
||
|
||
`used_size=3996`, `free_size=38`. The gap after id=3 holds a dead copy of a row
|
||
ending `"...Jelly Fish Jam.mp3"`; the gap after id=4 holds dead copies of
|
||
`"Myrtle Ave Party Track"` and `"Smooth Move"` rows. These are previous versions
|
||
of edited rows, left uncompacted. (Verified by ASCII-dumping the gap bytes.)
|
||
|
||
## The fields we do NOT match (and why)
|
||
|
||
1. **File / page count and `next_unused_page`.**
|
||
Reference: 49 pages, `next_unused=53`. maple: 41 pages, `next_unused=41`.
|
||
The reference is larger purely because dead-row bloat pushed live rows onto
|
||
extra pages. A compacted export needs fewer pages.
|
||
|
||
2. **Overflow / data-page placement (tracks `last_page`).**
|
||
Reference tracks table spans pages **1 → 48** (live track id=7 spilled onto
|
||
overflow page 48 because dead rows "filled" page 2). maple spans **1 → 2**
|
||
(all 6 tracks fit one compacted page). This is the *only* table-directory
|
||
entry that differs — the other 19 tables match first/last page exactly. The
|
||
divergence is entirely a function of the dead-row bloat above.
|
||
|
||
3. **File-header `sequence` value.**
|
||
Reference: `353`. maple: an incrementing counter (reads the destination
|
||
export's sequence + 1, default 1; see `writer._next_sequence`). rekordbox's
|
||
value comes from its internal `master.db` transaction counter and is not
|
||
reproducible from content. We match the *semantics* (monotonic increment),
|
||
not the absolute value.
|
||
|
||
4. **Per-page `sequence` (page-header offset 16).**
|
||
Reference pages carry assorted values (e.g. 26, 45, 320, 342, 347) — each a
|
||
snapshot of the global counter at the moment that page was last rewritten.
|
||
maple writes a clean per-build value. Same root cause as (3); not reproducible.
|
||
|
||
5. **`transaction_row_count` / `transaction_row_index` (page-header u2 pair).**
|
||
Reference: nonzero per edited page (e.g. `1/5`, `1/8`, `2/0`). maple: `0/0`.
|
||
These track the last in-place transaction on the page; meaningless for a
|
||
freshly written, never-edited page.
|
||
|
||
6. **Dead-row offset arrays + `num_row_offsets` > `num_rows`.**
|
||
Reference row groups list offsets for dead slots too: e.g. tracks page 2 has
|
||
`nro=9, nrows=5` with offsets `[0,428,856,1324,1792,2224,2656,3112,3568]`
|
||
(slots 0,5,7,8 dead); artists page 6 has `nro=9, nrows=1`. maple emits exactly
|
||
`nro == nrows` with no dead slots. Reproducing dead offsets would mean
|
||
reproducing the edit history.
|
||
|
||
7. **Live row in a non-zero slot.**
|
||
Reference artists page 6 keeps its single live row in **slot 8** (slots 0–7
|
||
are dead). maple places the single live row in **slot 0**. A from-scratch
|
||
export also writes slot 0.
|
||
|
||
8. **`free_size` / `used_size` totals on data pages.**
|
||
Reference `used+free` totals vary (4034, 4050, …) because `used_size` covers
|
||
the dead-row region and the strange-block markers differ. maple's pages total
|
||
a clean `4020` (= heap capacity for a single 36-byte row group). Direct
|
||
consequence of (6) — dead bytes are counted as "used".
|
||
|
||
9. **Track row opaque id blob — `0x14`/`0x18`/`0x1a` (8 bytes/track).**
|
||
One 8-byte LE value `opaque_track_id_lo | (opaque_seed << 32)`, both halves
|
||
stored in rekordbox's internal master.db (a per-install `opaque_seed` + a
|
||
per-track `opaque_track_id_lo`). Proven non-derivable from track metadata
|
||
across every real export + an exhaustive negative hash search
|
||
(FORMAT_SPEC.md s2). **Every real rekordbox export has BOTH halves non-zero**
|
||
(this install's seed = `0x21b4a7ab`; `data/MAPLE` = `0xd6d3a567`); the
|
||
"unanalyzed default" lo=0/seed=0x78f74a68 was a false lead — it came from
|
||
maple's own earlier output (`data/usb`), not a real export. maple emits a
|
||
plausible non-zero stand-in of the right shape: `opaque_seed = 0x21b4a7ab`
|
||
(module const `TRACK_OPAQUE_SEED`, overridable per install) and a stable
|
||
per-track `opaque_track_id_lo` mapped into the real Content-ID band
|
||
(`OPAQUE_LO_MIN..MAX` = 0x08000000..0x0F900000, where every observed AceMo lo
|
||
lands) via `crc32(id|file_path)`. This will NOT match rekordbox's master.db
|
||
id byte-for-byte — unreproducible without its DB — but it is non-zero,
|
||
in-range and deterministic, the safest shape for deck acceptance. Players
|
||
treat the blob as opaque.
|
||
|
||
10. **Strange-page free-map extension + per-page seq on tracks/history.**
|
||
On the tracks (type 0) and history (type 19) strange pages only, the
|
||
free-space map carries an extra accounting word (body offset 16: `01 00`,
|
||
offset 20: a u4 = the data page's used_size for history=320; tracks differs
|
||
because of its dead-row/overflow state). maple writes the plain free-map.
|
||
Tied to the same content/overflow state as items (1)/(2); not reproducible
|
||
for a compacted writer. (The rest of every strange page — header, free-map
|
||
head, 1004× `f8 ff ff 1f`, the data-page pointer word, and unknown7 — now
|
||
matches the fresh reference byte-for-byte.)
|
||
|
||
11. **Table-directory `empty_candidate` (u32, table_dir entry +4) + header
|
||
`next_unused_page`.** Reference: a nonzero per-table `empty_candidate`
|
||
(e.g. 0x33, 0x04, 0x2e…); maple: 0. rekordcrate documents this field as
|
||
"maybe links to a chain of empty pages if the database is ever garbage
|
||
collected", and its fixtures show non-sequential free-pool page indices
|
||
(47, 4, 49, 8, 50, 46, 42, 16…). It is free-list/GC allocator state, not
|
||
derivable from content. `next_unused_page` (rekordcrate: "points past end of
|
||
file", unused) is the same class — see also item (1).
|
||
|
||
> Canonical reference note: the diff target is now the FRESH export
|
||
> `data/acemo_fresh/PIONEER/rekordbox/export.pdb` (7 tracks). It is cleaner than
|
||
> the edited `data/acemo` (6 tracks) but still incrementally built per-file, so
|
||
> the divergences above still apply — just smaller (e.g. track 7's dead-row span
|
||
> is 464B not 3568B, but it still overflows to page 48).
|
||
|
||
## What IS reproducible (NOW matched against the fresh reference)
|
||
|
||
- The 20-table directory layout and page placement: an empty table is
|
||
strange-page-only; a table gets a data page IFF it has rows (`CORE_TABLES` is
|
||
now empty — the old genres data page was an edit-history artifact of the
|
||
edited ref).
|
||
- `page_flags`: data pages are `0x34` for **tracks and history only**, `0x24`
|
||
for every other table (the wider `{0,2,3,13,17,19}` was edited-ref noise);
|
||
empty data pages `0x44`; strange pages `0x64`.
|
||
- Strange-page body: the full DeviceSQL free-space map — `u4(page_index)`,
|
||
`u4(first_data_page or 0x03ffffff)`, the 12-byte const tail, 1004×
|
||
`f8 ff ff 1f`, 20 zero bytes — plus header `free=used=0`, `txc=txi=0x1fff`,
|
||
`next_page = page_index+1`, and `unknown7 = 1` for tracks/history else 0.
|
||
- Track row `index_shift` (`slot_index<<5`) and every other live-row content
|
||
field except the opaque blob in divergence item (9).
|
||
- Three header fields `re` decoded (FORMAT_SPEC §4/§5/§6) as edit-history state
|
||
whose FRESH-export value maple already writes — so they're matched on a fresh
|
||
export and only diverge against edited refs:
|
||
- file-header offset 16 (`_unnamed4`): fresh=`0` (maple writes 0); edited DBs
|
||
accumulate a small nonzero with no structural meaning.
|
||
- strange/data-page `unknown7` (0x26): fresh=`0` everywhere (maple writes 0);
|
||
nonzero only on edited DBs' strange pages.
|
||
- row-group trailing u2: it is the "rows-touched-in-last-transaction" bitmask.
|
||
On a fresh export it equals the row-present mask, so maple writes
|
||
`presence` (matches 12/14 groups in `data/acemo_fresh`; the 2 misses are
|
||
edited pages — divergence). NOTE: `re`'s "keep constant 16" advice came from
|
||
reading `data/usb`, which is maple's OWN output, not a real export — a real
|
||
fresh export uses the presence value, not 16.
|
||
|
||
See `re/FORMAT_SPEC.md` for the value-construction rules and `re/README.md` for
|
||
the open RE questions.
|