ntr/maple

0 Fork 0

Files

T

Nathan Rashleigh 9ccc3cea95 the downfall has begun

2026-06-09 18:46:25 +10:00

36 KiB

Raw Blame History

export.pdb value-construction rules (RE findings)

This file records the exact rules rekordbox uses to populate export.pdb fields that maple currently writes wrong or as zero. Each section is a self-contained spec for the maple writer to implement. Evidence is from reverse-engineering rekordbox.exe (DeviceSQL "Page Manager", dsql* / edb_dyn_*) in Ghidra plus byte-level analysis of the reference export maple-py/data/acemo/PIONEER/rekordbox/export.pdb (6 tracks, 1 album, 1 artist, 1 key, 6 artwork).

Reference-pdb facts used throughout:

len_page = 4096.
Row index grows backward from end of page; each group spans 36 (0x24) bytes: group_base = len_page - group_index*36. Within a group, present_flags (u2) live at group_base-4; the row offset for slot i (0..15) is the u2 at group_base-(6+2*i); offsets are relative to the heap start (0x28, i.e. end of the 0x28-byte page header).

1. Track row `index_shift` (offset 0x02, u2) — SOLVED

Field: TrackRow.index_shift (also present on artist/album/genre/label/ key rows — every row type whose layout begins subtype:u2, index_shift:u2).

Rule:

index_shift = slot_index * 0x20      # i.e. slot_index << 5

where slot_index is the row's 0-based position in the page row index (group_index*16 + slot_within_group), NOT its ordinal among present rows and NOT its position in the heap. It resets to 0 at the start of every data page.

Key consequences for the writer:

The first row index slot (slot 0) gets index_shift = 0.
Deleted/absent slots still consume a slot number, so a gap in present_flags produces a gap in index_shift. (See evidence: the AceMo page has slot 5 deleted, so the row in slot 6 has index_shift = 0xc0, skipping 0xa0.)
index_shift is purely a function of the row's slot, so the value is fully reproducible from the page layout maple already controls — there is no external/runtime input. If maple writes each track to slot i, set index_shift = i*0x20.

Evidence:

Reference AceMo tracks page (page 2): present slots are 1,2,3,4,6 (slots 0 and 5 are deleted), with index_shift = 0x20,0x40,0x60,0x80,0xc0 respectively. Overflow page (page 48) has one row in slot 1 → index_shift = 0x20. The 0x80→0xc0 jump across the deleted slot 5 proves the value tracks slot_index, not present-count.
Cross-checked against artists (page 5) and albums (page 7) rows: every row satisfies index_shift == slot_index*0x20 with 0 mismatches over all present rows in the reference file (maple-py/re/scripts-style raw parse, 8 rows checked, see RE notes).
Ghidra: the value is assigned inside the DeviceSQL Page Manager record-insert path, which is reached only through vtable dispatch ((**(code**)(*obj + 0x4d0))(obj,...) etc. in FUN_141b8b4c0 / FUN_141cac390), so there is no single statically-anchored index_shift = slot<<5 instruction to cite; a constant-scan for shl ,5 paired with the row-index masks (0x1fff/0x7ff) and stride 0x24 produced only CRT/printf("%a") false positives (FUN_14232a86c). The empirical rule above is exact and zero-mismatch, so confidence is high on the formula; confidence on "no hidden runtime input" is high because the value is fully explained by slot position alone.

Confidence: high.

2. Track row 0x14 / 0x18 / 0x1a (opaque id blob) — SOLVED (stored value, not a formula)

Fields: TrackRow.opaque_track_id_lo (u4 @0x14), TrackRow.opaque_seed_lo (u2 @0x18), TrackRow.opaque_seed_hi (u2 @0x1a). Together these are one 8-byte little-endian value:

blob_u64 = opaque_track_id_lo | (opaque_seed << 32)
opaque_seed = opaque_seed_lo | (opaque_seed_hi << 16)   # u4 at offset 0x18

(@flesniak's schema labels these "Some ID? unknown", "always 19048?", "always 30967?". The "19048"/"30967" = 0x4A68/0x78F7 are only correct for some older exports; they are NOT universal — see below.)

What they are (proven empirically across 5 GENUINE rekordbox exports):

NOTE (2026-06-09 correction): a prior draft included a 5th row "usb (176), seq=1, seed 0x78f74a68, lo all 0" and concluded rekordbox writes lo=0 / seed 0x78f74a68 for unanalyzed tracks. That file (data/usb/.../export.pdb) is maple's OWN output, not a rekordbox export (it is the compare_pdb.py "ours" path). Reading it back was circular; that claim is WRONG and has been dropped. Provenance rule: maple output = seed 0x78f74a68 + ALL per-track lo == 0; genuine rekordbox = per-install nonzero seed + nonzero per-track lo.

export (tracks)	header.seq	opaque_seed	opaque_track_id_lo
acemo (6)	353	`0x21b4a7ab`	6/6 nonzero
acemo_fresh (7, same lib)	35	`0x21b4a7ab`	7/7 nonzero
lib_multi (10, same lib)	56	`0x21b4a7ab`	10/10 nonzero
MAPLE (176, install B)	546	`0xd6d3a567`	176/176 nonzero
data/PIONEER (847, inst. B)	4211	`0xd6d3a567`	847/847 nonzero

Across all 1046 real tracks in 5 genuine exports: 100% nonzero per-track lo, exactly one nonzero per-install seed per file. There is NO evidence rekordbox ever writes lo=0 or seed 0x78f74a68. The lo values span nearly the full 28-bit space (0x0003edeb..0x0ffe7f4e).

Findings:

opaque_seed is constant within a single export (exactly one value per file) and is per-rekordbox-library/install: acemo/acemo_fresh/lib_multi are three exports of the same library and all share 0x21b4a7ab; the separate install behind MAPLE + data/PIONEER uses 0xd6d3a567. (The old maple hardcode 0xa567/0xd6d3 = 0xd6d3a567 was correct only for install B.) The seed is NOT a binary constant (proven: no 0xa7ab/0x21b4/0xd6d3/0xa567 immediate or data-byte sequence in rekordbox.exe), so it is derived at runtime from the per-library DB (most likely a hash/truncation of the master.db's per-library UUID or creation marker). maple cannot reproduce a specific install's seed (see recommendation).
opaque_track_id_lo is a stable per-track value within a library. Every track that appears in both the acemo and acemo_fresh exports has an identical lo (matched by title): "All My Life"→0x07ff5a56, "Smooth Move"→0x0a20de34, "Jelly Fish Jam"→0x0f8fd598, etc. It is NOT per-export-random.
It is NOT derivable from the track's file metadata. Tested and ruled out (no match, 28-bit and full, utf-8/utf-16, several hash funcs): crc32 / adler32 / md5 / sha1 / fnv1 / fnv1a / djb2 of the title, filename, full /Contents/... path, and crc32 of the ANLZ .DAT/.EXT and the .mp3 bytes. Also not the USBANLZ folder id, not sequential, not evenly spaced.
The values are runtime-derived, not binary constants. A full instruction-immediate scan of rekordbox.exe found NO occurrence of either real seed (0x21b4a7ab, 0xd6d3a567), the per-track lo values, or the track-row bitmask constant 0xc0700. So these come from rekordbox's internal database (master.db / djmdContent + a per-library seed), copied into the row by the DeviceSQL record builder (vtable-dispatched, not statically anchorable). (0x78f74a68 does not appear either — and per the correction above it was never a real rekordbox value here, just maple's placeholder.)

Round-2 evidence (2026-06-09, after team-lead supplied acemo_fresh and asked to chase an audio-content hash): 5. Audio-content-hash hypothesis: DISPROVEN. Hashed the actual .mp3 bytes in 8 framings — full file, ID3v2-stripped (audio frames only), first 1k/2k/4k/64k, last 64k, ID3-stripped-first-64k — each with crc32, adler32, md5 (lo/hi, LE+BE), sha1 (LE+BE), sha256, fnv1a, both full-32 and masked-to-28-bit. NONE matched any track's opaque_track_id_lo. So it is not a hash of the audio file content (at least not of the file bytes; a hash of decoded PCM cannot be ruled out but there is no evidence for it and it is not reproducible anyway). 6. Source DB confirmed, and it is ENCRYPTED. Both exportLibrary.db (next to the pdb) and the captured master.db (maple-py/captures/p17328_master.db) fail to open as plain SQLite ("file is not a database") — rekordbox 6/7 encrypts them with SQLCipher. The opaque_track_id_lo originates there; it cannot be read without the key. 7. The value is a pure stored field, used nowhere else. It appears in the export.pdb ONLY at each track row's own offset 0x14 (present and stale/deleted rows alike); it does NOT appear in exportExt.pdb (no tag_tracks/ContentID link), nor in the ANLZ .DAT/.EXT (searched LE+BE), nor does it equal the USBANLZ folder id. So it is not a foreign key the player resolves. 8. Range note (suggestive, not conclusive): across the 5 real exports the lo values span nearly the full 28-bit space (0x0003edeb..0x0ffe7f4e), the band of rekordbox-6/7 9-digit djmdContent.ID integers, so the most likely identity is "the rekordbox internal Content ID" — a stored random/sequential DB id, NOT a content-derived hash. Consistent with every other finding and with it being unreadable (encrypted DB).

Conclusion: this 8-byte blob is a stored rekordbox-internal id (most likely the encrypted-DB Content ID), not a function of the track metadata or audio bytes maple has. It cannot be reproduced byte-for-byte for an arbitrary track without rekordbox's own (encrypted) DB. The audio-content-hash idea is ruled out. There is no "unanalyzed/zero" form in any real export — every real track carries a nonzero seed and a nonzero lo.

Recommendation for maple's writer:

Players/crate-digger treat this blob as opaque (never decoded) and play fine regardless — so any value works for playback.
For maximum faithfulness to genuine rekordbox output (which never has lo=0/seed 0x78f74a68), maple should write a nonzero per-install seed and a nonzero, stable per-track lo:
- opaque_seed: use a real per-install seed. If maple has no install of its own, 0x21b4a7ab (this RE machine's install) is a known-real value; keep it constant across all tracks in one export. Do NOT use 0x78f74a68 (that was maple's own placeholder, not a rekordbox value).
- opaque_track_id_lo: a nonzero 28-bit value, stable per track. Since the real value is an opaque DB id maple can't recover, a deterministic stand-in (e.g. a 28-bit hash of a stable per-track key, or the track's own pdb id offset into the ContentID band) is fine — the goal is "looks like a real nonzero ContentID", not byte-parity.
Either way these 8 bytes will NOT match a specific reference export byte-for-byte (the real ids live in that install's encrypted DB) — record the divergence in re/KNOWN_DIVERGENCE.md.

Confidence: high (5-export empirical proof: 1046/1046 real tracks nonzero; exhaustive negative hash search incl. audio bytes; binary const-absence). The earlier "write 0 / 0x78f74a68" advice was based on a maple-output file and is retracted.

Round-3 (task #10): can pyrekordbox read the real ids out of master.db? User insight: pyrekordbox holds the SQLCipher key, so the DBs are not truly unreadable. Validated — partly true, but blocked on the data we have:

pyrekordbox 0.4.4 + sqlcipher3-wheels installs and wires up correctly. Its master.db key is the single global rekordbox-6 key 402fd482c38817c35ffa8ffb8c7d93143b749e7d315df7a81732a1ff43608497 (deobfuscate(BLOB), must start with 402fd) — same for every install, NOT per-install. So a VALID master.db would open and expose DjmdContent.
BUT the captured maple-py/captures/p17328_master.db is corrupt/truncated: 1,024,000 bytes but only 4.4% non-zero, and the first 32 KB (the mandatory SQLCipher salt + page 1) is all zeros. No key can decrypt a DB with a zeroed salt. p17328_master.backup.db is 0 bytes. The Frida NtWriteFile capture only caught scattered later pages, never the header. → unusable.
The genuine exportLibrary.db shipped next to export.pdb (valid 16-byte salt ed0d3a3c…) does NOT open with the global master.db key (different key scheme) and pyrekordbox has no exportLibrary.db support. → can't read it here.
Strong inference (untested for lack of a readable DB): the per-track opaque_track_id_lo is almost certainly DjmdContent.ID (the 9-digit rekordbox Content ID; our lo range 0x0003edeb..0x0ffe7f4e fits). If maple can open the user's LIVE %APPDATA%/Pioneer/rekordbox6/master.db with pyrekordbox (global key), db.get_content() would give (ID, …) per track, and the blob would become EXACTLY reproducible: opaque_track_id_lo = DjmdContent.ID & 0xFFFFFFFF.
The per-install opaque_seed (0x21b4a7ab here) is NOT a DjmdContent column; it is one value for the whole export and most likely derived from a master.db-level per-library marker (a USB/library UUID or an agentRegistry/ local_usn-style value). Identifying it needs a readable master.db to inspect candidate columns; unconfirmed.

pyrekordbox write / export capability (asked in #10):

It CAN write the desktop master.db: add_content/add_album/add_artist/ create_playlist/commit, plus set_local_usn, update_content_path. But add_content lets rekordbox ASSIGN the Content ID; there is no API to force a specific ID. So it is a source of real ids, not a way to dictate them.
It does NOT drive a USB export — only get_device (read). The actual export.pdb/exportLibrary.db writing is rekordbox.exe's DeviceSQL engine (the thing maple reimplements). pyrekordbox manipulates only the desktop library DB, not the USB export. So pyrekordbox cannot replace maple's writer; its only use to maple is as a READ source of the real DjmdContent.ID (and potentially the seed) from a valid master.db.

Net for #10: the approach is sound (pyrekordbox + global key reads a real master.db → Content IDs), so opaque_track_id_lo is likely reproducible if maple reads the live master.db. It is NOT reproducible from the corrupt captured DB we have, and the seed source is still unidentified. pyrekordbox cannot produce the USB pdb itself. Recommended next step for maple/RE: point pyrekordbox at the user's LIVE rekordbox6 master.db (%APPDATA%/Pioneer/rekordbox6/master.db), confirm DjmdContent.ID == opaque_track_id_lo for the AceMo tracks, then hunt the seed in that DB. Until then, the nonzero stand-in (above) stays.

3. File-header `sequence` (and per-page `sequence`)

Field: RekordboxPdb.sequence (u4 @0x14 of the file header) and Page.sequence (u4 in each page header).

Rule / meaning (confirmed from schema docs + the 5 genuine exports above):

sequence is a monotonically increasing edit counter for the database. The file-header value is the next page-sequence number; each page records the counter value as of its last edit (so header.sequence >= max(page.sequence)). Observed header values in genuine rekordbox exports: 35, 56, 353, 546, 4211 — scales with edit history, not track count. (No real export with sequence==1 is available; data/usb's seq==1 is maple's own output, not a rekordbox sample — see §2 correction.)
The firmware/players (and crate-digger) use page sequence only as informational ordering metadata; they do not validate it for playback. No special value is required beyond keeping header.sequence greater than any page's sequence.
For maple it is enough to (a) pick a header sequence larger than every page sequence it writes, and (b) give each page a sequence <= header. A simple scheme (all pages = N, header = N+1) is accepted. This is what task #3 (incrementing sequence) implements.

Confidence: high for semantics and "not validated by players"; the exact increment rekordbox uses per edit is not needed for a valid export.

4. File-header offset 0x10 (u4, schema `_unnamed4`) — HISTORY-DEPENDENT; not byte-reproducible

CORRECTION (2026-06-09): a prior draft cited a "usb fresh, sequence==1, _unnamed4==0" row as proof that 0 is the fresh value. That file is maple's own output, not a rekordbox export, so it is removed below. The remaining proof (same-library divergence) does not depend on it.

Field: the unnamed u4 between next_unused_page (0x0c) and sequence (0x14). maple writes 0; the AceMo ref has 4.

Finding (5 genuine rekordbox exports): an edit/allocator accumulator, not a structural value derivable from the tables. It does NOT equal track count, page count, or next_unused_page - npages:

export	sequence	_unnamed4@0x10	next_unused	npages
acemo_fresh	35	5	52	49
lib_multi	56	4	57	52
acemo	353	4	53	49
MAPLE	546	5	—	—
data/PIONEER	4211	1	174	174

Decisive (same-library proof): acemo and acemo_fresh are the SAME library (identical track set + identical per-track opaque ids) yet have _unnamed4 = 4 vs 5. A structural field would be identical; differing proves it is edit state. (We have NO genuine rekordbox export with sequence==1, so the exact never-edited value is unconfirmed — but it is clearly not derivable from the file's structure.)

Rule for maple: there is no reproducible rule. Write 0 (maple already does) as a reasonable minimal default — NOT a confirmed match (every genuine export we have is nonzero here, and we have no never-edited rekordbox export to check what fresh rekordbox writes). Record the divergence in re/KNOWN_DIVERGENCE.md. It cannot be matched to the AceMo ref's 4 without that DB's edit history.

Confidence: high that it is edit-history/allocator state and not structurally reproducible; the specific fresh-write value is unconfirmed (no seq==1 rekordbox sample available).

5. Page-header offset 0x26 (u2, "unknown7"/strange-page) — HISTORY-DEPENDENT; maple's 0 is a reasonable default

CORRECTION (2026-06-09): do NOT cite data/usb (sequence==1) here — it is maple's OWN output, so "maple's 0 matches usb" would be circular. The observations below are from the five GENUINE rekordbox exports only.

Field: the second of the two trailing u2s in the page header (after transaction_row_index @0x22 and the "1004 for strange blocks" u2 @0x24). @flesniak: "always 0 except 1 for history pages, num entries for strange pages?". maple writes 0; the AceMo ref has 0/1/2 on various tables' first ("strange") pages.

Finding (5 genuine rekordbox exports — all incrementally edited):

On DATA pages it is always 0 (all five exports). This is a solid observation about genuine rekordbox output.
It is nonzero only on a table's first/"strange" page (flags 0x64), and the value does NOT correlate with that table's data-page count or row count (acemo tracks: 2 data pages → 2; data/PIONEER tracks: 105 data pages → 23). It is transaction/edit-tracking state.
Decisive (same-library proof): acemo and acemo_fresh hold the identical tracks table, yet its strange-page unknown7 = 2 (acemo) vs 1 (acemo_fresh). Same data → different value ⇒ not structural, it is edit state. (lib_multi: tracks strange unk7 = 2 with 3 data pages / 10 rows — again no count relation.)

Rule for maple: write 0 on every page (maple already does). The data pages being 0 across all five genuine exports makes 0 a well-supported value for data pages. For strange pages the real value is per-export edit state (0/1/2…) that cannot be reproduced — maple's 0 there is a reasonable minimal default, NOT a confirmed match (we have no never-edited rekordbox export to check what fresh rekordbox stamps on a strange page). Record the strange-page divergence in re/KNOWN_DIVERGENCE.md.

Confidence: high that it is history-dependent and that data-page 0 matches real rekordbox; the fresh-write strange-page value is unconfirmed (no seq==1 rekordbox sample).

6. Row-group trailing u2 (`transaction_row_flags`) — write `present_flags` (NOT 16); residual mismatch is edit history

CORRECTION (2026-06-09): an earlier draft of this section claimed "a fresh export writes 16 everywhere, so maple's 16 is correct." That was based on data/usb/.../export.pdb, which on re-check is maple's OWN output, not a rekordbox export (opaque seed 0x78f74a68 + all-zero per-track lo + sequence==low). Reading it back was circular. The analysis below uses ONLY the five genuine rekordbox exports (provenance verified by nonzero per-track opaque ids + per-install seed): acemo, acemo_fresh, lib_multi, MAPLE, data/PIONEER.

Field: the last u2 of each 36-byte row-group block in the page row index (crate-digger transaction_row_flags, at base; in maple's pager.py it is pack("<H", 16) # unknown (rekordcrate default)).

Finding (5 genuine rekordbox exports): this u2 is the "rows touched by the last transaction" bitmask (same bit layout as row_present_flags), and is therefore edit-history state, not reproducible:

No genuine rekordbox export writes 16 as the field's value on settled pages (count of groups == 16 across the five exports: 0, 0, 0, 1, 8 — and those few are coincidental transaction bitmasks, not a default). So maple's constant 16 does NOT match rekordbox — it is a rekordcrate invention.
The dominant value in heavily-edited tables is 0x0 (last transaction touched no rows on that page), e.g. data/PIONEER 170/294 groups = 0x0.
Three DIFFERENT patterns coexist in ONE file, proving it is not structural: (a) some pages trailing == present_flags (columns/unk17/unk18); (b) artwork pages consistently use last group = 1<<(rows_in_last_group-1), earlier groups = 0 (data/PIONEER p28 g6 10rows→0x200, p147 g2 13rows→0x1000; acemo_fresh artwork 7rows→0x40); (c) churned tracks pages use an arbitrary single bit (0x1/0x2/0x4/0x8/0x10/0x40…) = whatever row that page's last edit touched. A candidate clean rule [last grp=1<<(rows-1), others=0] matches only 181/257 fully-present pages in data/PIONEER — confirming NO single formula.
Same-library proof: acemo and acemo_fresh hold identical data, yet the tracks page-2 group differs (acemo present=0x5e trailing=0x20; acemo_fresh present=0xbb trailing=0x4) — same rows, different value ⇒ not structural. The previously-flagged artwork 0x40 is likewise transaction state (acemo_fresh artwork=0x40 vs lib_multi artwork p49=0x4), not an open reproducible byte.

Rule for maple (DECISION): write present_flags, NOT the constant 16. maple's old constant 16 (rekordcrate's placeholder) matches 0/14 real groups; trailing = row_present_flags matches 12/14 clean groups in the real data/acemo_fresh export (8 rows→0xff, 16→0xffff, 11→0x7ff, 6→0x3f, 1→0x1). The 2 misses (page2→0x4, page28→0x40, page40→0xc0) are the last-transaction bitmask on pages that were edited after creation — i.e. residual edit history, not a different rule. So for a from-scratch single-transaction write the correct value IS present_flags (all rows touched in the one creating transaction). maple now ships present_flags (pager.py) — this is the best achievable; reverting to 16 would regress 12 groups. Remaining mismatch vs an EDITED reference (the per-page last-tx bitmask) is unavoidable edit history → KNOWN_DIVERGENCE.

NOTE: the earlier "keep 16 / fresh usb writes 0x10 everywhere" framing was WRONG — it read data/usb, which is MAPLE'S OWN OUTPUT (an old build that emitted the rekordcrate-default 16), not a real export. data/usb must NOT be used as a 'fresh rekordbox' baseline for ANY field (same contamination that produced the wrong opaque-blob default in §2). The fresh-write value is established from the REAL data/acemo_fresh as present_flags.

Confidence: high that it is history-dependent and that real rekordbox never uses 16; the correct fresh-write value is an open question (no seq==1 sample).

Note on the three "reproducible header fields" (task #9)

After multi-export analysis across the FIVE genuine rekordbox exports (acemo, acemo_fresh, lib_multi, MAPLE, data/PIONEER — provenance verified by nonzero per-track opaque ids + per-install seed), all three fields are edit-history / transaction state, NOT structurally reproducible:

§4 file-header 0x10: allocator/edit accumulator. → write 0, DIVERGENCE.
§5 page 0x26 unknown7: 0 on all data pages (exact); per-table 0/1/2 on strange pages is edit state. → write 0, strange-page DIVERGENCE.
§6 row-group trailing u2: "rows touched by last transaction" bitmask. Fresh single-transaction value = present_flags (12/14 in real data/acemo_fresh; constant 16 = 0/14). → maple writes present_flags; residual mismatch vs an EDITED ref is the per-page last-tx bitmask = edit history, DIVERGENCE.

The decisive evidence is the same-library pair (acemo vs acemo_fresh: identical track data, yet header 0x10 = 4 vs 5, tracks strange unk7 = 2 vs 1, tracks-page trailing differs) — a structural field would be identical.

maple's values: §4 header0x10 = 0 and §5 unknown7 = 0 are reasonable minimal defaults (confirmed = real exports on data pages; fresh strange-page value unconfirmed). §6 trailing u2 = present_flags is the established fresh-write value (NOT the old constant 16). No further writer change needed; remaining mismatches vs edited references are unavoidable edit history.

IMPORTANT corrections to the earlier draft: (1) data/usb/.../export.pdb is maple's OWN output (an old build that emitted the rekordcrate-default 16), not a rekordbox export — so any "fresh usb" claim was circular; data/usb is NOT an authority for ANY field (same contamination that produced the wrong opaque-blob default in §2). (2) We have NO genuine rekordbox export with sequence==1, so the never-edited header0x10 / strange-page-unknown7 values are unconfirmed. (3) The row-group trailing u2 fresh value is present_flags, established from the REAL data/acemo_fresh (12/14); the rekordcrate "Default=16" is a placeholder that matches 0/14 real groups.

All three go to re/KNOWN_DIVERGENCE.md, not the writer.

7. Artwork + USBANLZ path schemes (task #11)

Decoded from genuine data/MAPLE (176 tracks/176 artwork) + acemo. The guiding principle (per the task): for player acceptance only PATH SELF-CONSISTENCY matters — every path stored in the pdb must resolve to a real on-disk file. Matching rekordbox's exact id-derivation is NOT required; maple's own consistent scheme is acceptable.

7a. Artwork — SOLVED, exactly reproducible (maple already matches)

Path: /PIONEER/Artwork/{folder:05d}/a{artwork_id}.jpg (80×80) plus a a{artwork_id}_m.jpg (240×240) variant in the same folder (rekordbox writes both; verified on disk in data/MAPLE).
Folder bucket rule: folder = artwork_id // 20 + 1 (20 ids per bucket; bucket 00001 = ids 1–19 because ids are 1-based, 00002 = 20–39, 00003 = 40–59, …). Verified 176/176 in data/MAPLE.
maple's artwork._bucket() is literally f"{artwork_id // 20 + 1:05d}" and extract_artwork writes both jpg variants at that path → byte-identical to rekordbox AND self-consistent. No change needed.

7b. USBANLZ — opaque id; maple's own scheme is self-consistent (DIVERGENCE from rekordbox)

Path: /PIONEER/USBANLZ/P{bucket}/{8-hex}/ANLZ0000.DAT (+.EXT).
rekordbox's {8-hex} is a per-track opaque analysis id (range seen 0x117..0x309fd over 176 tracks; NOT == track id, NOT == the row-0x14 opaque_track_id_lo, NOT a crc of path/title — tested) and the P{bucket} (0–127, 94 distinct over 176) is a spreading bucket that is NOT a derivable function of the 8-hex or any track field (tested: &0x7f, >>k, %127/128, low-byte — all fail). Like the row-0x14 blob, these come from rekordbox's internal DB and are not reproducible without it.
maple (cli.py build()): generates its OWN deterministic scheme — bucket = P{(crc32(file_path) >> 20) & 0xFFF:03X}, analyze_id = {crc32(file_path):08X}, then write_anlz() writes the .DAT at exactly that path. This is self-consistent by construction (the path it records is the path it writes) and is the correct approach per the self-consistency principle. It does NOT match rekordbox's id/bucket (expected DIVERGENCE).
- Minor cosmetic note (not a correctness issue): maple's bucket is 3 hex digits P000..PFFF whereas rekordbox uses P00..P7F (0–127). Harmless; optionally narrow to & 0x7F (P{..:02X}) to look more rekordbox-like.

7c. Self-consistency verification

Genuine rekordbox (data/MAPLE): 176/176 ANLZ paths and 176/176 artwork paths resolve to on-disk files. (rekordbox is fully self-consistent — sanity check that confirms the parser and the principle.)
maple production build() (cli.py): self-consistent BY CONSTRUCTION — extract_artwork (artwork.py:62-65) writes a{id}.jpg/a{id}_m.jpg at the recorded path; write_anlz (cli.py:93) writes ANLZ0000.DAT at the recorded analyze_path. Each pdb path is computed and the file is written there in the same step, so every path resolves.
CAVEAT — the byte-diff harness tools/build_acemo.py is NOT the production path: it MIRRORS a reference rekordbox export (copies its analyze_path/ artwork path strings verbatim, writes no ANLZ/artwork files), so the data/usb export it produces references the reference's USBANLZ files which are not co-located → those paths don't resolve. This is expected for a structural-diff fixture and says nothing about production self-consistency.

Confidence: high. Artwork rule is exact (176/176). USBANLZ rekordbox ids are opaque/non-reproducible (consistent with the row-0x14 finding); maple's own USBANLZ scheme is self-consistent, which is all the player requires.

8. Page-manager free-space: `empty_candidate` + free pool (task #13 — THE FREEZE)

DECK TEST: maple's pdb FROZE the player (parser hang, not reject). Decoded the page-manager/free-space divergence against genuine rekordbox exports (data/working 9trk seq44, data/MAPLE 176trk, data/PIONEER 847trk — all load) vs maple output (captures/maple_acemo.pdb).

Data-page `next_page` SELF-LOOP — ROOT CAUSE (confirmed divergence; high confidence)

The player reads each table by following first_page → next_page → …. maple's last data page of each multi-page chain set next_page == its own index (page 2→2, 6→6, 8→8, 12, 14, 28, 34, 36, 38, 40 — 10 self-loops in the 9-track build). A page whose next_page is itself is an infinite loop → FREEZE.
Genuine rekordbox has ZERO self-loops. A terminal data page uses next_page = 0 (the "no next" sentinel; seen on pages 10/16/18/20/…/41-47 in data/working) OR points FORWARD to a distinct free page (never to itself).
FIX (maple, done in writer.py): point every chain's last page at a single dedicated zero-filled free-pool page (free_sentinel, just past the overflow region) instead of itself; materialise that page and set next_unused beyond it. Verified on the rebuilt pdb: 10 self-loops → 0; all 20 chains terminate at the blank sentinel; player tolerates last-page→free-page (genuine exports do the same). This is the primary freeze fix.

Table-dir `empty_candidate` (u4 after page_type) — SECONDARY divergence (also wrong; lower-confidence freeze contributor)

Layout: each 16-byte table-dir entry is type:u4, empty_candidate:u4, first_page:u4, last_page:u4.
empty_candidate is the head of that table's empty/free-page list (the page the Page Manager grabs when the table needs to grow).
In EVERY genuine rekordbox export all 20 empty_candidates are NONZERO and distinct, and each points to an all-zero "free" page — either an in-file allocated-but-empty page (e.g. last_page+1, or a pooled page like 41–47/49) or a beyond-EOF sentinel (index ≥ npages, e.g. 50/51/52 when npages=49 — the firmware tolerates beyond-EOF here). It NEVER points to a used data page and NEVER to 0.
maple writes empty_candidate = 0 for all 20 tables. Page 0 is the FILE HEADER. The firmware Page Manager, walking a table's empty-page list, follows 0 → reads the 4 KB file header AS a page → its "next_page" field (header offset 0x0c = next_unused_page) and bogus type/flags send the walk into a loop or out-of-bounds read → hang/FREEZE. This is the highest-confidence cause: it is the only systematic, data-independent divergence, and 0 is never valid (collides with the header).

Free pool / page layout

rekordbox keeps the data pages in chains 1..40,48,… and pre-allocates a pool of all-zero "free" pages past them; next_unused_page = one past the highest allocated page (so next_unused > npages is normal, e.g. 53 > 49).
maple compacts: 41 pages, every page used, next_unused == npages == 41, no free pool. The compaction itself is structurally legal (chains are clean), but it left empty_candidate = 0 with no free page to point at.
Beyond-EOF links (a page's next_page ≥ npages, or empty_candidate ≥ npages) are TOLERATED — the genuine working file has many (page 4→53, 46→52, empty_candidate 51/52/50/49 all ≥ 49). So beyond-EOF is a valid "no-more-pages / no-free-page" sentinel; it is NOT the freeze.

Strange-page free-map — NOT the cause (decoded, maple matches)

Each table's first/"strange" page (flags 0x64) carries, after a small prefix [this_page, next_or_0x03ffffff, 0x03ffffff, 0, 0x1fff0000], a heap filled with 1004 × 0x1ffffff8 (f8 ff ff 1f LE) — the empty-heap free-map (count == the m1004=1004 marker at header 0x24). 0x03ffffff in the 2nd u4 is the legitimate "this table has no data page" sentinel (genuine rekordbox writes it for every empty table; e.g. labels/pltree/plentries).
maple reproduces this free-map correctly; its strange-page prefixes match rekordbox's pattern byte-for-byte (data table → u1=first_data_page; empty table → u1=0x03ffffff). So the free-map is NOT the freeze.

FIX (for maple → deck test)

PRIMARY (high confidence, DONE): eliminate the data-page next_page self-loop — last page of each chain must point to a blank free page (or use next_page=0), never to itself. maple now points all last pages at one zero-filled free_sentinel page. Verified: 0 self-loops in the rebuilt pdb.

SECONDARY (cheap, also a real divergence — fold into the same build): stop writing empty_candidate = 0. Point it at the same free_sentinel (pack("<I", free_sentinel) instead of 0). Genuine rekordbox NEVER writes 0 (0 = the header page); it always points each table's empty_candidate at a blank free page. Lower-confidence as a freeze cause for a read-only player than the self-loop, but wrong and trivially fixable with the page maple already materialised.

If the deck STILL hangs after both, next suspects (lower confidence): next_unused_page == npages (give a little slack), and whether the single shared sentinel needs to be per-table-distinct (rekordbox uses distinct free pages).

Confidence: HIGH that the self-loop was the freeze (self-referential next_page is an unconditional infinite loop; 0/all genuine exports self-loop). empty_candidate=0 is independently wrong (0/all genuine exports use it; collides with header). Exact firmware behaviour is inferred (no firmware to trace); the deck test confirms. is the confirmation.

36 KiB Raw Blame History Unescape Escape