610 lines
36 KiB
Markdown
610 lines
36 KiB
Markdown
# export.pdb value-construction rules (RE findings)
|
||
|
||
This file records the exact rules rekordbox uses to populate `export.pdb`
|
||
fields that maple currently writes wrong or as zero. Each section is a
|
||
self-contained spec for the `maple` writer to implement. Evidence is from
|
||
reverse-engineering `rekordbox.exe` (DeviceSQL "Page Manager", `dsql*` /
|
||
`edb_dyn_*`) in Ghidra plus byte-level analysis of the reference export
|
||
`maple-py/data/acemo/PIONEER/rekordbox/export.pdb` (6 tracks, 1 album,
|
||
1 artist, 1 key, 6 artwork).
|
||
|
||
Reference-pdb facts used throughout:
|
||
- `len_page` = 4096.
|
||
- Row index grows backward from end of page; each group spans 36 (0x24) bytes:
|
||
`group_base = len_page - group_index*36`. Within a group, `present_flags`
|
||
(u2) live at `group_base-4`; the row offset for slot `i` (0..15) is the u2 at
|
||
`group_base-(6+2*i)`; offsets are relative to the heap start (0x28, i.e. end
|
||
of the 0x28-byte page header).
|
||
|
||
---
|
||
|
||
## 1. Track row `index_shift` (offset 0x02, u2) — SOLVED
|
||
|
||
**Field:** `TrackRow.index_shift` (also present on artist/album/genre/label/
|
||
key rows — every row type whose layout begins `subtype:u2, index_shift:u2`).
|
||
|
||
**Rule:**
|
||
|
||
```
|
||
index_shift = slot_index * 0x20 # i.e. slot_index << 5
|
||
```
|
||
|
||
where `slot_index` is the row's **0-based position in the page row index**
|
||
(`group_index*16 + slot_within_group`), NOT its ordinal among present rows and
|
||
NOT its position in the heap. It resets to 0 at the start of every data page.
|
||
|
||
Key consequences for the writer:
|
||
- The first row index slot (slot 0) gets `index_shift = 0`.
|
||
- Deleted/absent slots still consume a slot number, so a gap in
|
||
`present_flags` produces a gap in `index_shift`. (See evidence: the AceMo
|
||
page has slot 5 deleted, so the row in slot 6 has `index_shift = 0xc0`,
|
||
skipping `0xa0`.)
|
||
- `index_shift` is purely a function of the row's slot, so the value is fully
|
||
reproducible from the page layout maple already controls — there is no
|
||
external/runtime input. If maple writes each track to slot `i`, set
|
||
`index_shift = i*0x20`.
|
||
|
||
**Evidence:**
|
||
- Reference AceMo `tracks` page (page 2): present slots are 1,2,3,4,6 (slots 0
|
||
and 5 are deleted), with `index_shift` = 0x20,0x40,0x60,0x80,0xc0
|
||
respectively. Overflow page (page 48) has one row in slot 1 →
|
||
`index_shift = 0x20`. The 0x80→0xc0 jump across the deleted slot 5 proves the
|
||
value tracks `slot_index`, not present-count.
|
||
- Cross-checked against `artists` (page 5) and `albums` (page 7) rows: every
|
||
row satisfies `index_shift == slot_index*0x20` with **0 mismatches** over all
|
||
present rows in the reference file
|
||
(`maple-py/re/scripts`-style raw parse, 8 rows checked, see RE notes).
|
||
- Ghidra: the value is assigned inside the DeviceSQL Page Manager record-insert
|
||
path, which is reached only through vtable dispatch
|
||
(`(**(code**)(*obj + 0x4d0))(obj,...)` etc. in `FUN_141b8b4c0` /
|
||
`FUN_141cac390`), so there is no single statically-anchored
|
||
`index_shift = slot<<5` instruction to cite; a constant-scan for `shl ,5`
|
||
paired with the row-index masks (0x1fff/0x7ff) and stride 0x24 produced only
|
||
CRT/`printf("%a")` false positives (`FUN_14232a86c`). The empirical rule
|
||
above is exact and zero-mismatch, so confidence is **high** on the formula;
|
||
confidence on "no hidden runtime input" is high because the value is fully
|
||
explained by slot position alone.
|
||
|
||
**Confidence:** high.
|
||
|
||
---
|
||
|
||
## 2. Track row 0x14 / 0x18 / 0x1a (opaque id blob) — SOLVED (stored value, not a formula)
|
||
|
||
**Fields:** `TrackRow.opaque_track_id_lo` (u4 @0x14), `TrackRow.opaque_seed_lo`
|
||
(u2 @0x18), `TrackRow.opaque_seed_hi` (u2 @0x1a). Together these are one
|
||
8-byte little-endian value:
|
||
|
||
```
|
||
blob_u64 = opaque_track_id_lo | (opaque_seed << 32)
|
||
opaque_seed = opaque_seed_lo | (opaque_seed_hi << 16) # u4 at offset 0x18
|
||
```
|
||
|
||
(@flesniak's schema labels these "Some ID? unknown", "always 19048?",
|
||
"always 30967?". The "19048"/"30967" = 0x4A68/0x78F7 are only correct for some
|
||
older exports; they are NOT universal — see below.)
|
||
|
||
**What they are (proven empirically across 5 GENUINE rekordbox exports):**
|
||
|
||
> NOTE (2026-06-09 correction): a prior draft included a 5th row
|
||
> "usb (176), seq=1, seed 0x78f74a68, lo all 0" and concluded rekordbox writes
|
||
> lo=0 / seed 0x78f74a68 for unanalyzed tracks. **That file
|
||
> (`data/usb/.../export.pdb`) is maple's OWN output, not a rekordbox export**
|
||
> (it is the compare_pdb.py "ours" path). Reading it back was circular; that
|
||
> claim is WRONG and has been dropped. Provenance rule: maple output = seed
|
||
> `0x78f74a68` + ALL per-track lo == 0; genuine rekordbox = per-install nonzero
|
||
> seed + nonzero per-track lo.
|
||
|
||
| export (tracks) | header.seq | opaque_seed | opaque_track_id_lo |
|
||
|----------------------------|-----------:|---------------|-------------------------------|
|
||
| acemo (6) | 353 | `0x21b4a7ab` | 6/6 nonzero |
|
||
| acemo_fresh (7, same lib) | 35 | `0x21b4a7ab` | 7/7 nonzero |
|
||
| lib_multi (10, same lib) | 56 | `0x21b4a7ab` | 10/10 nonzero |
|
||
| MAPLE (176, install B) | 546 | `0xd6d3a567` | 176/176 nonzero |
|
||
| data/PIONEER (847, inst. B)| 4211 | `0xd6d3a567` | 847/847 nonzero |
|
||
|
||
Across **all 1046 real tracks in 5 genuine exports: 100% nonzero per-track lo,
|
||
exactly one nonzero per-install seed per file. There is NO evidence rekordbox
|
||
ever writes lo=0 or seed `0x78f74a68`.** The lo values span nearly the full
|
||
28-bit space (`0x0003edeb`..`0x0ffe7f4e`).
|
||
|
||
Findings:
|
||
1. **`opaque_seed` is constant within a single export** (exactly one value per
|
||
file) and is **per-rekordbox-library/install**: acemo/acemo_fresh/lib_multi
|
||
are three exports of the same library and all share `0x21b4a7ab`; the separate
|
||
install behind MAPLE + data/PIONEER uses `0xd6d3a567`. (The old maple hardcode
|
||
`0xa567`/`0xd6d3` = `0xd6d3a567` was correct only for install B.) The seed is
|
||
NOT a binary constant (proven: no `0xa7ab`/`0x21b4`/`0xd6d3`/`0xa567`
|
||
immediate or data-byte sequence in rekordbox.exe), so it is derived at runtime
|
||
from the per-library DB (most likely a hash/truncation of the master.db's
|
||
per-library UUID or creation marker). maple cannot reproduce a specific
|
||
install's seed (see recommendation).
|
||
2. **`opaque_track_id_lo` is a stable per-track value within a library.** Every
|
||
track that appears in both the acemo and acemo_fresh exports has an
|
||
**identical** lo (matched by title): "All My Life"→`0x07ff5a56`,
|
||
"Smooth Move"→`0x0a20de34`, "Jelly Fish Jam"→`0x0f8fd598`, etc. It is NOT
|
||
per-export-random.
|
||
3. **It is NOT derivable from the track's file metadata.** Tested and ruled out
|
||
(no match, 28-bit and full, utf-8/utf-16, several hash funcs): crc32 / adler32
|
||
/ md5 / sha1 / fnv1 / fnv1a / djb2 of the title, filename, full
|
||
`/Contents/...` path, and crc32 of the ANLZ `.DAT`/`.EXT` and the `.mp3`
|
||
bytes. Also not the USBANLZ folder id, not sequential, not evenly spaced.
|
||
4. **The values are runtime-derived, not binary constants.** A full
|
||
instruction-immediate scan of `rekordbox.exe` found NO occurrence of either
|
||
real seed (`0x21b4a7ab`, `0xd6d3a567`), the per-track lo values, or the
|
||
track-row `bitmask` constant `0xc0700`. So these come from rekordbox's
|
||
internal database (master.db / `djmdContent` + a per-library seed), copied
|
||
into the row by the DeviceSQL record builder (vtable-dispatched, not
|
||
statically anchorable). (`0x78f74a68` does not appear either — and per the
|
||
correction above it was never a real rekordbox value here, just maple's
|
||
placeholder.)
|
||
|
||
**Round-2 evidence (2026-06-09, after team-lead supplied acemo_fresh and asked
|
||
to chase an audio-content hash):**
|
||
5. **Audio-content-hash hypothesis: DISPROVEN.** Hashed the actual `.mp3`
|
||
bytes in 8 framings — full file, ID3v2-stripped (audio frames only),
|
||
first 1k/2k/4k/64k, last 64k, ID3-stripped-first-64k — each with crc32,
|
||
adler32, md5 (lo/hi, LE+BE), sha1 (LE+BE), sha256, fnv1a, both full-32 and
|
||
masked-to-28-bit. NONE matched any track's `opaque_track_id_lo`. So it is
|
||
not a hash of the audio file content (at least not of the file bytes; a hash
|
||
of decoded PCM cannot be ruled out but there is no evidence for it and it is
|
||
not reproducible anyway).
|
||
6. **Source DB confirmed, and it is ENCRYPTED.** Both `exportLibrary.db` (next
|
||
to the pdb) and the captured `master.db`
|
||
(`maple-py/captures/p17328_master.db`) fail to open as plain SQLite
|
||
("file is not a database") — rekordbox 6/7 encrypts them with SQLCipher.
|
||
The `opaque_track_id_lo` originates there; it cannot be read without the key.
|
||
7. **The value is a pure stored field, used nowhere else.** It appears in the
|
||
export.pdb ONLY at each track row's own offset 0x14 (present and stale/deleted
|
||
rows alike); it does NOT appear in exportExt.pdb (no tag_tracks/ContentID
|
||
link), nor in the ANLZ `.DAT`/`.EXT` (searched LE+BE), nor does it equal the
|
||
USBANLZ folder id. So it is not a foreign key the player resolves.
|
||
8. **Range note (suggestive, not conclusive):** across the 5 real exports the
|
||
lo values span nearly the full 28-bit space (`0x0003edeb`..`0x0ffe7f4e`),
|
||
the band of rekordbox-6/7 9-digit `djmdContent.ID` integers, so the most
|
||
likely identity is "the rekordbox internal Content ID" — a stored
|
||
random/sequential DB id, NOT a content-derived hash. Consistent with every
|
||
other finding and with it being unreadable (encrypted DB).
|
||
|
||
**Conclusion:** this 8-byte blob is a **stored rekordbox-internal id (most
|
||
likely the encrypted-DB Content ID), not a function of the track metadata or
|
||
audio bytes maple has. It cannot be reproduced byte-for-byte** for an arbitrary
|
||
track without rekordbox's own (encrypted) DB. The audio-content-hash idea is
|
||
ruled out. There is **no "unanalyzed/zero" form** in any real export — every
|
||
real track carries a nonzero seed and a nonzero lo.
|
||
|
||
**Recommendation for maple's writer:**
|
||
- Players/crate-digger treat this blob as opaque (never decoded) and play fine
|
||
regardless — so any value works for playback.
|
||
- For maximum faithfulness to genuine rekordbox output (which never has
|
||
lo=0/seed `0x78f74a68`), maple should write a **nonzero per-install seed and a
|
||
nonzero, stable per-track lo**:
|
||
- `opaque_seed`: use a real per-install seed. If maple has no install of its
|
||
own, `0x21b4a7ab` (this RE machine's install) is a known-real value;
|
||
keep it constant across all tracks in one export. Do NOT use `0x78f74a68`
|
||
(that was maple's own placeholder, not a rekordbox value).
|
||
- `opaque_track_id_lo`: a nonzero 28-bit value, stable per track. Since the
|
||
real value is an opaque DB id maple can't recover, a deterministic stand-in
|
||
(e.g. a 28-bit hash of a stable per-track key, or the track's own pdb id
|
||
offset into the ContentID band) is fine — the goal is "looks like a real
|
||
nonzero ContentID", not byte-parity.
|
||
- Either way these 8 bytes will NOT match a specific reference export
|
||
byte-for-byte (the real ids live in that install's encrypted DB) — record the
|
||
divergence in re/KNOWN_DIVERGENCE.md.
|
||
|
||
**Confidence:** high (5-export empirical proof: 1046/1046 real tracks nonzero;
|
||
exhaustive negative hash search incl. audio bytes; binary const-absence). The
|
||
earlier "write 0 / 0x78f74a68" advice was based on a maple-output file and is
|
||
retracted.
|
||
|
||
**Round-3 (task #10): can pyrekordbox read the real ids out of master.db?**
|
||
User insight: pyrekordbox holds the SQLCipher key, so the DBs are not truly
|
||
unreadable. Validated — partly true, but blocked on the data we have:
|
||
- pyrekordbox 0.4.4 + `sqlcipher3-wheels` installs and wires up correctly. Its
|
||
master.db key is the single **global** rekordbox-6 key
|
||
`402fd482c38817c35ffa8ffb8c7d93143b749e7d315df7a81732a1ff43608497`
|
||
(`deobfuscate(BLOB)`, must start with `402fd`) — same for every install, NOT
|
||
per-install. So a VALID master.db *would* open and expose `DjmdContent`.
|
||
- BUT the captured `maple-py/captures/p17328_master.db` is **corrupt/truncated**:
|
||
1,024,000 bytes but only 4.4% non-zero, and the first 32 KB (the mandatory
|
||
SQLCipher salt + page 1) is **all zeros**. No key can decrypt a DB with a
|
||
zeroed salt. `p17328_master.backup.db` is 0 bytes. The Frida NtWriteFile
|
||
capture only caught scattered later pages, never the header. → unusable.
|
||
- The genuine `exportLibrary.db` shipped next to export.pdb (valid 16-byte salt
|
||
`ed0d3a3c…`) does NOT open with the global master.db key (different key
|
||
scheme) and pyrekordbox has **no exportLibrary.db support**. → can't read it
|
||
here.
|
||
- Strong inference (untested for lack of a readable DB): the per-track
|
||
`opaque_track_id_lo` is almost certainly `DjmdContent.ID` (the 9-digit
|
||
rekordbox Content ID; our lo range 0x0003edeb..0x0ffe7f4e fits). If maple can
|
||
open the user's LIVE `%APPDATA%/Pioneer/rekordbox6/master.db` with pyrekordbox
|
||
(global key), `db.get_content()` would give `(ID, …)` per track, and the blob
|
||
would become EXACTLY reproducible: `opaque_track_id_lo = DjmdContent.ID & 0xFFFFFFFF`.
|
||
- The per-install `opaque_seed` (0x21b4a7ab here) is NOT a DjmdContent column;
|
||
it is one value for the whole export and most likely derived from a
|
||
master.db-level per-library marker (a USB/library UUID or an `agentRegistry`/
|
||
`local_usn`-style value). Identifying it needs a readable master.db to inspect
|
||
candidate columns; unconfirmed.
|
||
|
||
**pyrekordbox write / export capability (asked in #10):**
|
||
- It CAN write the desktop master.db: `add_content`/`add_album`/`add_artist`/
|
||
`create_playlist`/`commit`, plus `set_local_usn`, `update_content_path`. But
|
||
`add_content` lets rekordbox ASSIGN the Content ID; there is no API to force a
|
||
specific ID. So it is a source of real ids, not a way to dictate them.
|
||
- It does NOT drive a USB export — only `get_device` (read). The actual
|
||
`export.pdb`/`exportLibrary.db` writing is rekordbox.exe's DeviceSQL engine
|
||
(the thing maple reimplements). pyrekordbox manipulates only the desktop
|
||
library DB, not the USB export. So pyrekordbox cannot replace maple's writer;
|
||
its only use to maple is as a READ source of the real `DjmdContent.ID` (and
|
||
potentially the seed) from a valid master.db.
|
||
|
||
**Net for #10:** the approach is sound (pyrekordbox + global key reads a real
|
||
master.db → Content IDs), so `opaque_track_id_lo` is likely reproducible *if
|
||
maple reads the live master.db*. It is NOT reproducible from the corrupt
|
||
captured DB we have, and the seed source is still unidentified. pyrekordbox
|
||
cannot produce the USB pdb itself. Recommended next step for maple/RE: point
|
||
pyrekordbox at the user's LIVE rekordbox6 master.db
|
||
(`%APPDATA%/Pioneer/rekordbox6/master.db`), confirm
|
||
`DjmdContent.ID == opaque_track_id_lo` for the AceMo tracks, then hunt the seed
|
||
in that DB. Until then, the nonzero stand-in (above) stays.
|
||
|
||
---
|
||
|
||
## 3. File-header `sequence` (and per-page `sequence`)
|
||
|
||
**Field:** `RekordboxPdb.sequence` (u4 @0x14 of the file header) and
|
||
`Page.sequence` (u4 in each page header).
|
||
|
||
**Rule / meaning (confirmed from schema docs + the 5 genuine exports above):**
|
||
- `sequence` is a monotonically increasing **edit counter** for the database.
|
||
The file-header value is the **next** page-sequence number; each page records
|
||
the counter value as of its last edit (so `header.sequence >=
|
||
max(page.sequence)`). Observed header values in genuine rekordbox exports:
|
||
35, 56, 353, 546, 4211 — scales with edit history, not track count. (No real
|
||
export with sequence==1 is available; `data/usb`'s seq==1 is maple's own
|
||
output, not a rekordbox sample — see §2 correction.)
|
||
- The firmware/players (and crate-digger) use page `sequence` only as
|
||
informational ordering metadata; **they do not validate it** for playback.
|
||
No special value is required beyond keeping `header.sequence` greater than any
|
||
page's `sequence`.
|
||
- For maple it is enough to (a) pick a header `sequence` larger than every page
|
||
`sequence` it writes, and (b) give each page a `sequence` <= header. A simple
|
||
scheme (all pages = N, header = N+1) is accepted. This is what task #3
|
||
(incrementing sequence) implements.
|
||
|
||
**Confidence:** high for semantics and "not validated by players"; the exact
|
||
increment rekordbox uses per edit is not needed for a valid export.
|
||
|
||
---
|
||
|
||
## 4. File-header offset 0x10 (u4, schema `_unnamed4`) — HISTORY-DEPENDENT; not byte-reproducible
|
||
|
||
> CORRECTION (2026-06-09): a prior draft cited a "usb fresh, sequence==1,
|
||
> _unnamed4==0" row as proof that 0 is the fresh value. That file is maple's own
|
||
> output, not a rekordbox export, so it is removed below. The remaining proof
|
||
> (same-library divergence) does not depend on it.
|
||
|
||
**Field:** the unnamed u4 between `next_unused_page` (0x0c) and `sequence`
|
||
(0x14). maple writes 0; the AceMo ref has 4.
|
||
|
||
**Finding (5 genuine rekordbox exports):** an **edit/allocator accumulator**,
|
||
not a structural value derivable from the tables. It does NOT equal track count,
|
||
page count, or `next_unused_page - npages`:
|
||
|
||
| export | sequence | _unnamed4@0x10 | next_unused | npages |
|
||
|---------------|---------:|---------------:|------------:|-------:|
|
||
| acemo_fresh | 35 | 5 | 52 | 49 |
|
||
| lib_multi | 56 | 4 | 57 | 52 |
|
||
| acemo | 353 | 4 | 53 | 49 |
|
||
| MAPLE | 546 | 5 | — | — |
|
||
| data/PIONEER | 4211 | 1 | 174 | 174 |
|
||
|
||
**Decisive (same-library proof):** acemo and acemo_fresh are the SAME library
|
||
(identical track set + identical per-track opaque ids) yet have `_unnamed4` =
|
||
4 vs 5. A structural field would be identical; differing proves it is edit
|
||
state. (We have NO genuine rekordbox export with `sequence==1`, so the exact
|
||
never-edited value is unconfirmed — but it is clearly not derivable from the
|
||
file's structure.)
|
||
|
||
**Rule for maple:** there is no reproducible rule. Write **0** (maple already
|
||
does) as a **reasonable minimal default — NOT a confirmed match** (every genuine
|
||
export we have is nonzero here, and we have no never-edited rekordbox export to
|
||
check what fresh rekordbox writes). Record the divergence in
|
||
re/KNOWN_DIVERGENCE.md. It cannot be matched to the AceMo ref's `4` without that
|
||
DB's edit history.
|
||
|
||
**Confidence:** high that it is edit-history/allocator state and not
|
||
structurally reproducible; the specific fresh-write value is unconfirmed (no
|
||
seq==1 rekordbox sample available).
|
||
|
||
---
|
||
|
||
## 5. Page-header offset 0x26 (u2, "unknown7"/strange-page) — HISTORY-DEPENDENT; maple's 0 is a reasonable default
|
||
|
||
> CORRECTION (2026-06-09): do NOT cite `data/usb` (sequence==1) here — it is
|
||
> maple's OWN output, so "maple's 0 matches usb" would be circular. The
|
||
> observations below are from the five GENUINE rekordbox exports only.
|
||
|
||
**Field:** the second of the two trailing u2s in the page header (after
|
||
`transaction_row_index` @0x22 and the "1004 for strange blocks" u2 @0x24).
|
||
@flesniak: "always 0 except 1 for history pages, num entries for strange
|
||
pages?". maple writes 0; the AceMo ref has 0/1/2 on various tables' first
|
||
("strange") pages.
|
||
|
||
**Finding (5 genuine rekordbox exports — all incrementally edited):**
|
||
- On DATA pages it is **always 0** (all five exports). This is a solid
|
||
observation about genuine rekordbox output.
|
||
- It is nonzero only on a table's first/"strange" page (flags 0x64), and the
|
||
value does NOT correlate with that table's data-page count or row count
|
||
(acemo tracks: 2 data pages → 2; data/PIONEER tracks: 105 data pages → 23).
|
||
It is transaction/edit-tracking state.
|
||
- **Decisive (same-library proof):** acemo and acemo_fresh hold the identical
|
||
tracks table, yet its strange-page unknown7 = 2 (acemo) vs 1 (acemo_fresh).
|
||
Same data → different value ⇒ not structural, it is edit state. (lib_multi:
|
||
tracks strange unk7 = 2 with 3 data pages / 10 rows — again no count relation.)
|
||
|
||
**Rule for maple:** write **0** on every page (maple already does). The data
|
||
pages being 0 across all five genuine exports makes 0 a well-supported value
|
||
*for data pages*. For strange pages the real value is per-export edit state
|
||
(0/1/2…) that cannot be reproduced — maple's 0 there is a **reasonable minimal
|
||
default, NOT a confirmed match** (we have no never-edited rekordbox export to
|
||
check what fresh rekordbox stamps on a strange page). Record the strange-page
|
||
divergence in re/KNOWN_DIVERGENCE.md.
|
||
|
||
**Confidence:** high that it is history-dependent and that data-page 0 matches
|
||
real rekordbox; the fresh-write strange-page value is unconfirmed (no seq==1
|
||
rekordbox sample).
|
||
|
||
---
|
||
|
||
## 6. Row-group trailing u2 (`transaction_row_flags`) — write `present_flags` (NOT 16); residual mismatch is edit history
|
||
|
||
> CORRECTION (2026-06-09): an earlier draft of this section claimed "a fresh
|
||
> export writes 16 everywhere, so maple's 16 is correct." That was based on
|
||
> `data/usb/.../export.pdb`, which on re-check is **maple's OWN output**, not a
|
||
> rekordbox export (opaque seed `0x78f74a68` + all-zero per-track lo +
|
||
> `sequence`==low). Reading it back was circular. The analysis below uses ONLY
|
||
> the five genuine rekordbox exports (provenance verified by nonzero per-track
|
||
> opaque ids + per-install seed): acemo, acemo_fresh, lib_multi, MAPLE,
|
||
> data/PIONEER.
|
||
|
||
**Field:** the last u2 of each 36-byte row-group block in the page row index
|
||
(crate-digger `transaction_row_flags`, at `base`; in maple's `pager.py` it is
|
||
`pack("<H", 16) # unknown (rekordcrate default)`).
|
||
|
||
**Finding (5 genuine rekordbox exports):** this u2 is the
|
||
**"rows touched by the last transaction" bitmask** (same bit layout as
|
||
`row_present_flags`), and is therefore **edit-history state, not reproducible**:
|
||
- **No genuine rekordbox export writes 16 as the field's value** on settled
|
||
pages (count of groups == 16 across the five exports: 0, 0, 0, 1, 8 — and
|
||
those few are coincidental transaction bitmasks, not a default). So maple's
|
||
constant **16 does NOT match rekordbox** — it is a rekordcrate invention.
|
||
- The dominant value in heavily-edited tables is **0x0** (last transaction
|
||
touched no rows on that page), e.g. data/PIONEER 170/294 groups = 0x0.
|
||
- Three DIFFERENT patterns coexist in ONE file, proving it is not structural:
|
||
(a) some pages `trailing == present_flags` (columns/unk17/unk18); (b) artwork
|
||
pages consistently use `last group = 1<<(rows_in_last_group-1)`, earlier groups
|
||
= 0 (data/PIONEER p28 g6 10rows→0x200, p147 g2 13rows→0x1000; acemo_fresh
|
||
artwork 7rows→0x40); (c) churned tracks pages use an arbitrary single bit
|
||
(0x1/0x2/0x4/0x8/0x10/0x40…) = whatever row that page's last edit touched.
|
||
A candidate clean rule `[last grp=1<<(rows-1), others=0]` matches only
|
||
**181/257** fully-present pages in data/PIONEER — confirming NO single formula.
|
||
- **Same-library proof:** acemo and acemo_fresh hold identical data, yet the
|
||
tracks page-2 group differs (acemo present=0x5e trailing=0x20; acemo_fresh
|
||
present=0xbb trailing=0x4) — same rows, different value ⇒ not structural.
|
||
The previously-flagged artwork `0x40` is likewise transaction state
|
||
(acemo_fresh artwork=0x40 vs lib_multi artwork p49=0x4), not an open
|
||
reproducible byte.
|
||
|
||
**Rule for maple (DECISION): write `present_flags`, NOT the constant 16.**
|
||
maple's old constant 16 (rekordcrate's placeholder) matches **0/14** real groups;
|
||
`trailing = row_present_flags` matches **12/14** clean groups in the real
|
||
`data/acemo_fresh` export (8 rows→0xff, 16→0xffff, 11→0x7ff, 6→0x3f, 1→0x1). The
|
||
2 misses (page2→0x4, page28→0x40, page40→0xc0) are the last-transaction bitmask
|
||
on pages that were edited after creation — i.e. residual edit history, not a
|
||
different rule. So for a from-scratch single-transaction write the correct value
|
||
IS `present_flags` (all rows touched in the one creating transaction). maple now
|
||
ships `present_flags` (pager.py) — this is the best achievable; reverting to 16
|
||
would regress 12 groups. Remaining mismatch vs an EDITED reference (the per-page
|
||
last-tx bitmask) is unavoidable edit history → KNOWN_DIVERGENCE.
|
||
|
||
> NOTE: the earlier "keep 16 / fresh usb writes 0x10 everywhere" framing was
|
||
> WRONG — it read `data/usb`, which is MAPLE'S OWN OUTPUT (an old build that
|
||
> emitted the rekordcrate-default 16), not a real export. `data/usb` must NOT be
|
||
> used as a 'fresh rekordbox' baseline for ANY field (same contamination that
|
||
> produced the wrong opaque-blob default in §2). The fresh-write value is
|
||
> established from the REAL `data/acemo_fresh` as `present_flags`.
|
||
|
||
**Confidence:** high that it is history-dependent and that real rekordbox never
|
||
uses 16; the correct fresh-write value is an open question (no seq==1 sample).
|
||
|
||
---
|
||
|
||
## Note on the three "reproducible header fields" (task #9)
|
||
|
||
After multi-export analysis across the FIVE genuine rekordbox exports (acemo,
|
||
acemo_fresh, lib_multi, MAPLE, data/PIONEER — provenance verified by nonzero
|
||
per-track opaque ids + per-install seed), all three fields are
|
||
**edit-history / transaction state, NOT structurally reproducible**:
|
||
- §4 file-header 0x10: allocator/edit accumulator. → write 0, DIVERGENCE.
|
||
- §5 page 0x26 unknown7: 0 on all data pages (exact); per-table 0/1/2 on strange
|
||
pages is edit state. → write 0, strange-page DIVERGENCE.
|
||
- §6 row-group trailing u2: "rows touched by last transaction" bitmask. Fresh
|
||
single-transaction value = `present_flags` (12/14 in real data/acemo_fresh;
|
||
constant 16 = 0/14). → maple writes `present_flags`; residual mismatch vs an
|
||
EDITED ref is the per-page last-tx bitmask = edit history, DIVERGENCE.
|
||
|
||
The decisive evidence is the **same-library pair** (acemo vs acemo_fresh:
|
||
identical track data, yet header 0x10 = 4 vs 5, tracks strange unk7 = 2 vs 1,
|
||
tracks-page trailing differs) — a structural field would be identical.
|
||
|
||
maple's values: §4 header0x10 = 0 and §5 unknown7 = 0 are reasonable minimal
|
||
defaults (confirmed = real exports on data pages; fresh strange-page value
|
||
unconfirmed). §6 trailing u2 = `present_flags` is the established fresh-write
|
||
value (NOT the old constant 16). No further writer change needed; remaining
|
||
mismatches vs edited references are unavoidable edit history.
|
||
|
||
IMPORTANT corrections to the earlier draft: (1) `data/usb/.../export.pdb` is
|
||
maple's OWN output (an old build that emitted the rekordcrate-default 16), not a
|
||
rekordbox export — so any "fresh usb" claim was circular; `data/usb` is NOT an
|
||
authority for ANY field (same contamination that produced the wrong opaque-blob
|
||
default in §2). (2) We have NO genuine rekordbox export with sequence==1, so the
|
||
never-edited header0x10 / strange-page-unknown7 values are unconfirmed. (3) The
|
||
row-group trailing u2 fresh value is `present_flags`, established from the REAL
|
||
data/acemo_fresh (12/14); the rekordcrate "Default=16" is a placeholder that
|
||
matches 0/14 real groups.
|
||
|
||
All three go to re/KNOWN_DIVERGENCE.md, not the writer.
|
||
|
||
---
|
||
|
||
## 7. Artwork + USBANLZ path schemes (task #11)
|
||
|
||
Decoded from genuine `data/MAPLE` (176 tracks/176 artwork) + acemo. The guiding
|
||
principle (per the task): for **player acceptance only PATH SELF-CONSISTENCY
|
||
matters** — every path stored in the pdb must resolve to a real on-disk file.
|
||
Matching rekordbox's exact id-derivation is NOT required; maple's own consistent
|
||
scheme is acceptable.
|
||
|
||
### 7a. Artwork — SOLVED, exactly reproducible (maple already matches)
|
||
- Path: `/PIONEER/Artwork/{folder:05d}/a{artwork_id}.jpg` (80×80) plus a
|
||
`a{artwork_id}_m.jpg` (240×240) variant in the same folder (rekordbox writes
|
||
both; verified on disk in data/MAPLE).
|
||
- **Folder bucket rule: `folder = artwork_id // 20 + 1`** (20 ids per bucket;
|
||
bucket 00001 = ids 1–19 because ids are 1-based, 00002 = 20–39, 00003 = 40–59,
|
||
…). Verified **176/176** in data/MAPLE.
|
||
- maple's `artwork._bucket()` is literally `f"{artwork_id // 20 + 1:05d}"` and
|
||
`extract_artwork` writes both jpg variants at that path → **byte-identical to
|
||
rekordbox AND self-consistent**. No change needed.
|
||
|
||
### 7b. USBANLZ — opaque id; maple's own scheme is self-consistent (DIVERGENCE from rekordbox)
|
||
- Path: `/PIONEER/USBANLZ/P{bucket}/{8-hex}/ANLZ0000.DAT` (+`.EXT`).
|
||
- rekordbox's `{8-hex}` is a per-track **opaque analysis id** (range seen
|
||
0x117..0x309fd over 176 tracks; NOT == track id, NOT == the row-0x14
|
||
`opaque_track_id_lo`, NOT a crc of path/title — tested) and the `P{bucket}`
|
||
(0–127, 94 distinct over 176) is a spreading bucket that is **NOT a derivable
|
||
function** of the 8-hex or any track field (tested: `&0x7f`, `>>k`, `%127/128`,
|
||
low-byte — all fail). Like the row-0x14 blob, these come from rekordbox's
|
||
internal DB and are **not reproducible** without it.
|
||
- maple (cli.py `build()`): generates its OWN deterministic scheme —
|
||
`bucket = P{(crc32(file_path) >> 20) & 0xFFF:03X}`,
|
||
`analyze_id = {crc32(file_path):08X}`, then `write_anlz()` writes the .DAT at
|
||
exactly that path. This is **self-consistent by construction** (the path it
|
||
records is the path it writes) and is the correct approach per the
|
||
self-consistency principle. It does NOT match rekordbox's id/bucket (expected
|
||
DIVERGENCE).
|
||
- Minor cosmetic note (not a correctness issue): maple's bucket is 3 hex
|
||
digits `P000..PFFF` whereas rekordbox uses `P00..P7F` (0–127). Harmless;
|
||
optionally narrow to `& 0x7F` (`P{..:02X}`) to look more rekordbox-like.
|
||
|
||
### 7c. Self-consistency verification
|
||
- **Genuine rekordbox (data/MAPLE): 176/176 ANLZ paths and 176/176 artwork
|
||
paths resolve to on-disk files.** (rekordbox is fully self-consistent — sanity
|
||
check that confirms the parser and the principle.)
|
||
- **maple production `build()` (cli.py): self-consistent BY CONSTRUCTION** —
|
||
`extract_artwork` (artwork.py:62-65) writes `a{id}.jpg`/`a{id}_m.jpg` at the
|
||
recorded path; `write_anlz` (cli.py:93) writes `ANLZ0000.DAT` at the recorded
|
||
`analyze_path`. Each pdb path is computed and the file is written there in the
|
||
same step, so every path resolves.
|
||
- CAVEAT — the byte-diff harness `tools/build_acemo.py` is NOT the production
|
||
path: it MIRRORS a reference rekordbox export (copies its `analyze_path`/
|
||
artwork `path` strings verbatim, writes no ANLZ/artwork files), so the
|
||
`data/usb` export it produces references the reference's USBANLZ files which
|
||
are not co-located → those paths don't resolve. This is expected for a
|
||
structural-diff fixture and says nothing about production self-consistency.
|
||
|
||
**Confidence:** high. Artwork rule is exact (176/176). USBANLZ rekordbox ids are
|
||
opaque/non-reproducible (consistent with the row-0x14 finding); maple's own
|
||
USBANLZ scheme is self-consistent, which is all the player requires.
|
||
|
||
---
|
||
|
||
## 8. Page-manager free-space: `empty_candidate` + free pool (task #13 — THE FREEZE)
|
||
|
||
DECK TEST: maple's pdb FROZE the player (parser hang, not reject). Decoded the
|
||
page-manager/free-space divergence against genuine rekordbox exports
|
||
(`data/working` 9trk seq44, `data/MAPLE` 176trk, `data/PIONEER` 847trk — all
|
||
load) vs maple output (`captures/maple_acemo.pdb`).
|
||
|
||
### Data-page `next_page` SELF-LOOP — ROOT CAUSE (confirmed divergence; high confidence)
|
||
- The player reads each table by following `first_page → next_page → …`. maple's
|
||
**last data page of each multi-page chain set `next_page == its own index`**
|
||
(page 2→2, 6→6, 8→8, 12, 14, 28, 34, 36, 38, 40 — 10 self-loops in the 9-track
|
||
build). A page whose `next_page` is itself is an **infinite loop → FREEZE**.
|
||
- **Genuine rekordbox has ZERO self-loops.** A terminal data page uses
|
||
`next_page = 0` (the "no next" sentinel; seen on pages 10/16/18/20/…/41-47 in
|
||
`data/working`) OR points FORWARD to a distinct free page (never to itself).
|
||
- FIX (maple, done in writer.py): point every chain's last page at a single
|
||
dedicated zero-filled free-pool page (`free_sentinel`, just past the overflow
|
||
region) instead of itself; materialise that page and set `next_unused` beyond
|
||
it. Verified on the rebuilt pdb: 10 self-loops → 0; all 20 chains terminate at
|
||
the blank sentinel; player tolerates last-page→free-page (genuine exports do
|
||
the same). This is the primary freeze fix.
|
||
|
||
### Table-dir `empty_candidate` (u4 after page_type) — SECONDARY divergence (also wrong; lower-confidence freeze contributor)
|
||
- Layout: each 16-byte table-dir entry is `type:u4, empty_candidate:u4,
|
||
first_page:u4, last_page:u4`.
|
||
- `empty_candidate` is the head of that table's **empty/free-page list** (the
|
||
page the Page Manager grabs when the table needs to grow).
|
||
- **In EVERY genuine rekordbox export all 20 `empty_candidate`s are NONZERO and
|
||
distinct**, and each points to an **all-zero "free" page** — either an
|
||
in-file allocated-but-empty page (e.g. `last_page+1`, or a pooled page like
|
||
41–47/49) or a **beyond-EOF sentinel** (index ≥ npages, e.g. 50/51/52 when
|
||
npages=49 — the firmware tolerates beyond-EOF here). It NEVER points to a
|
||
used data page and NEVER to 0.
|
||
- **maple writes `empty_candidate = 0` for all 20 tables.** Page 0 is the FILE
|
||
HEADER. The firmware Page Manager, walking a table's empty-page list, follows
|
||
`0` → reads the 4 KB file header AS a page → its "next_page" field (header
|
||
offset 0x0c = `next_unused_page`) and bogus type/flags send the walk into a
|
||
loop or out-of-bounds read → **hang/FREEZE**. This is the highest-confidence
|
||
cause: it is the only systematic, data-independent divergence, and `0` is
|
||
never valid (collides with the header).
|
||
|
||
### Free pool / page layout
|
||
- rekordbox keeps the data pages in chains `1..40,48,…` and pre-allocates a
|
||
pool of all-zero "free" pages past them; `next_unused_page` = one past the
|
||
highest allocated page (so `next_unused > npages` is normal, e.g. 53 > 49).
|
||
- maple **compacts**: 41 pages, every page used, `next_unused == npages == 41`,
|
||
no free pool. The compaction itself is structurally legal (chains are clean),
|
||
but it left `empty_candidate = 0` with no free page to point at.
|
||
- Beyond-EOF links (a page's `next_page` ≥ npages, or `empty_candidate` ≥
|
||
npages) are TOLERATED — the genuine `working` file has many (page 4→53,
|
||
46→52, empty_candidate 51/52/50/49 all ≥ 49). So beyond-EOF is a valid
|
||
"no-more-pages / no-free-page" sentinel; it is NOT the freeze.
|
||
|
||
### Strange-page free-map — NOT the cause (decoded, maple matches)
|
||
- Each table's first/"strange" page (flags 0x64) carries, after a small prefix
|
||
`[this_page, next_or_0x03ffffff, 0x03ffffff, 0, 0x1fff0000]`, a heap filled
|
||
with **1004 × `0x1ffffff8`** (`f8 ff ff 1f` LE) — the empty-heap free-map
|
||
(count == the `m1004`=1004 marker at header 0x24). `0x03ffffff` in the 2nd u4
|
||
is the legitimate **"this table has no data page"** sentinel (genuine
|
||
rekordbox writes it for every empty table; e.g. labels/pltree/plentries).
|
||
- maple reproduces this free-map correctly; its strange-page prefixes match
|
||
rekordbox's pattern byte-for-byte (data table → u1=first_data_page; empty
|
||
table → u1=0x03ffffff). **So the free-map is NOT the freeze.**
|
||
|
||
### FIX (for maple → deck test)
|
||
PRIMARY (high confidence, DONE): **eliminate the data-page `next_page` self-loop**
|
||
— last page of each chain must point to a blank free page (or use `next_page=0`),
|
||
never to itself. maple now points all last pages at one zero-filled `free_sentinel`
|
||
page. Verified: 0 self-loops in the rebuilt pdb.
|
||
|
||
SECONDARY (cheap, also a real divergence — fold into the same build): **stop
|
||
writing `empty_candidate = 0`.** Point it at the same `free_sentinel`
|
||
(`pack("<I", free_sentinel)` instead of 0). Genuine rekordbox NEVER writes 0
|
||
(0 = the header page); it always points each table's empty_candidate at a blank
|
||
free page. Lower-confidence as a freeze cause for a read-only player than the
|
||
self-loop, but wrong and trivially fixable with the page maple already
|
||
materialised.
|
||
|
||
If the deck STILL hangs after both, next suspects (lower confidence):
|
||
`next_unused_page == npages` (give a little slack), and whether the single shared
|
||
sentinel needs to be per-table-distinct (rekordbox uses distinct free pages).
|
||
|
||
**Confidence:** HIGH that the self-loop was the freeze (self-referential
|
||
next_page is an unconditional infinite loop; 0/all genuine exports self-loop).
|
||
empty_candidate=0 is independently wrong (0/all genuine exports use it; collides
|
||
with header). Exact firmware behaviour is inferred (no firmware to trace); the
|
||
deck test confirms.
|
||
is the confirmation.
|