ntr/maple
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
9ee5a702660639545213db4d7a93ea0eb100ac9a
maple/maple-py/re/REFERENCE_COMPARISON.md
T

29 KiB
Raw Blame History

REFERENCE_COMPARISON.md — pyrekordbox & rekordcrate vs maple

What the two mature open-source rekordbox libraries can decode/encode, how that lines up with maple's writer and our RE findings, and the concrete GAPS maple must close for a full from-scratch USB export.

Scope note (coordination): the per-format library-capability comparison and the from-scratch-export gaps are mine (task #12). The master.db SQLCipher key mechanics and opaque Content ID identity are owned by re (#10); the exact USBANLZ/artwork on-disk path schemes by re (#11). Where those are cited below they are tagged [see re #10] / [see re #11] and should be treated as re's authority, not re-derived here.

0. One-screen capability matrix

Format pyrekordbox rekordcrate maple
export.pdb parse only (via the pdb parser, read) parse + WRITE (BinWrite on rows/pages, full roundtrip) WRITE only (from-scratch, the whole point)
exportExt.pdb not supported not supported not written
ANLZ (.DAT/.EXT/.2EX) parse only ("creating … is planned") parse + partial write (binrw BinWrite derived on waveform/cue structs; not a whole-file builder) WRITE (minimal: PMAI + PPTH only)
artwork (.jpg thumbs) reads ImagePath from db only n/a (just the pdb Artwork row path string) WRITE (extracts embedded art)
master.db (db6/7) READ + WRITE (SQLCipher unlock) not supported not used
mysetting (*SETTING.DAT) READ + WRITE (build from scratch) parse only (setting module) not written
rekordbox XML READ + WRITE parse only (xml module) not used

The headline: no existing library writes a complete from-scratch USB export. rekordcrate writes export.pdb rows but not ANLZ/exportExt and has no metadata source; pyrekordbox writes master.db + mysetting + XML but not the USB export.pdb/ANLZ. maple is filling a real gap. The reference libs are most valuable to maple as (a) a read-back oracle (parse maple's output and diff) and (b) for pyrekordbox, a source of the real opaque Content IDs from master.db.

1. export.pdb

What rekordcrate does (the closest reference writer)

pdb/mod.rs decodes and writes every row type maple emits: Track, Artist, Album, Artwork, Color, Genre, Key, Label, PlaylistTreeNode, PlaylistEntry, ColumnEntry, HistoryPlaylist, HistoryEntry (pub enum Row, lines ~1064-1103). It has a genuine page serializer: write_page_contents (mod.rs:212), write_options_and_get_row_offset (mod.rs:463), per-row-type alignment (align_by, mod.rs:1121-1133), and a Track BinWrite impl (mod.rs:934).

Where rekordcrate CONFIRMS maple's layout (high-value confirmations)

  • Track fixed prefix = 0x5e bytes, then a 21×u2 (0x2a) offset block. rekordcrate asserts start_of_string_section - base_position == 0x5e and seeks 0x2a for 21 string offsets (mod.rs ~986-996). maple's TRACK_FIXED_PREFIX = 0x5E, TRACK_HEADER_LEN = 0x5E + 21*2 (rows.py:8-10) and its 21-string list match field-for-field, including the exact string order (isrc, …, analyze_path, analyze_date, comment, title, unknown_string8, filename, file_path). Independent confirmation of maple's track row.
  • Field order of the 0x5e prefix is identical between rekordcrate's Track::write_options and maple's serialize_track: unknown1/index_shift/ bitmask/sample_rate/composer_id/file_size/unknown2(=opaque lo @0x14)/ unknown3+unknown4(=seed @0x18/0x1a)/artwork_id/key_id/orig_artist_id/ label_id/remixer_id/bitrate/track_number/tempo/genre_id/album_id/artist_id/ id/disc/play_count/year/sample_depth/duration/unknown5/color/rating/unknown6/ unknown7.
  • DeviceSQL string encoding. rekordcrate pdb/string.rs roundtrips the same three forms maple's strings.py writes: short-ASCII ((len<<1)|1), long-ASCII (0x40), long-UTF16LE (0x90). Its tests encode "I ❤ Rust" as 0x90 and a long string as 0x40 — exactly maple's selection rule. The empty string is 0x03 in both. Confirms strings.py.
  • Row-group / page-index mechanics. rekordcrate's RowGroup (mod.rs:451, 2102) uses the same 16-rows-per-group, backward-growing offset index, and presence bitmask maple builds in pager.py. Header (mod.rs:162) carries the same 13-bit num_row_offsets / 11-bit num_rows packing maple writes (pager.py:131).

Where rekordcrate caught a maple BUG — next_page chain termination (the FREEZE)

rekordcrate pdb/mod.rs documents Page::next_page (lines 262-265) precisely:

"Index of the next page with the same page type. If this page is the last one of that type, the page index stored in the field will point past the end of the file."

maple's writer originally set the last page of each table chain to next_page = its own page_index (a self-pointer). That is exactly what rekordcrate says must NOT happen — the last page must point forward, past the allocated pages. A self-referencing next_page makes a firmware page-walk that terminates on "next_page past EOF" loop forever on the last page → the deck freezes. This is the root cause being fixed in task #13: writer.py now points every last page at a dedicated free-pool sentinel page just past the overflow region (free_sentinel, materialised zero-filled) and sets next_unused_page beyond it — which removes the self-loop. (rekordcrate's own read_pages, mod.rs:185-196, terminates the walk on page_index == table.last_page from the directory, so its parser tolerates a self-pointer — which is why maple's read-back oracle never flagged it. The firmware uses the chain-walk, so only a real deck exposed the freeze. Lesson: a clean rekordcrate parse is NOT proof the chain is firmware-valid.)

Verified against a real export — the EXACT rule (not just "past EOF"). I decoded data/PIONEER/rekordbox/export.pdb (847 tracks, 174 pages, next_unused=177). rekordcrate's "points past the end of file" is a simplification; the real, consistent rule across all 20 tables is: **each table's last/only page sets next_page == that table's empty_candidate**, where empty_candidate(table-dir entry +4) is a *distinct, nonzero* page index< next_unused— the head of that table's free-page chain. Evidence: every last page'snext_pageequalled its own dirempty_candidate(type0 173→174=ec; type1 4→53=ec; type19 40→41=ec; …), and those targets (174,53,158,163,54,49,42,46,170,20,22,175,176,165,30,32, 43,44,45,41) are all< next_unused. Strange-only tables already match (next_page = page_idx+1 = their ec; free-map data_ptr=0x03ffffff) — maple's build_strange_page` does this correctly.

So maple's current #13 fix (a single shared sentinel for every data table + empty_candidate=0 in the directory) removes the freeze but still diverges from real rekordbox, which uses a distinct per-table empty_candidate that the directory entry also records (nonzero). If firmware ever cross-checks next_page against the directory's empty_candidate, the shared sentinel + zero ec could still misbehave. Recommended (reported to maple): give each data table its own free page, set next_page = that page AND write that index into the table-dir empty_candidate, keep them all < next_unused. This is straightforwardly reproducible (unlike the value of empty_candidate on an edited ref, which is GC state — here we only need a valid distinct free page, which a from-scratch writer fully controls).

Where rekordcrate is WRONG and maple is RIGHT — page row-count header (>255 rows)

Audited because rekordcrate's Page header (mod.rs:277-377) reads the row count as num_rows_small: u8 @0x18 plus num_rows_large: u16 @0x22 with a calculate_num_rows rule (large if large>small && large!=0x1fff else small). maple instead writes the canonical crate-digger .ksy bitfield at 0x18: 13-bit num_row_offsets + 11-bit num_rows, then page_flags @0x1b (pager.py:131). These are two different readings of the same bytes — so I verified against a real export (data/PIONEER, which has 284-row playlist-entry pages):

  • Real page 18 bytes @0x18 = 1c 81 23 24. The 24-bit LE value 0x23811c decodes (kaitai/maple way) to num_row_offsets=284, num_rows=284 — correct.
  • maple's packer reproduces 1c 81 23 exactly for n=284 (and a6 c0 14 for 166, 7e c0 0f for 126 — all matching real pages byte-for-byte).
  • rekordcrate's reading of the same bytes: num_rows_small = 0x1c = 28 (just the low byte) and num_rows_large @0x22 = 283 (which is really the transaction_row_index edit field), giving calculate_num_rows = 283 — i.e. rekordcrate would report 283 rows on a 284-row page, and 28 if the transaction field were 0. Its small/large model only coincidentally agrees when n <= 255.
  • Conclusion: maple's header encoding is correct and firmware-faithful for pages up to 2047 rows (the 11-bit max); rekordcrate's num_rows_small/ num_rows_large is an older/divergent decode that breaks for n > 255. This matters because maple can emit >255-row pages (playlist_entries fits ~284/ page, colors ~394). No maple change needed — this is a maple-is-right result.

Where rekordcrate's row alignment diverges (maple matches real rekordbox)

rekordcrate aligns each row by its Rust struct's natural align_of_val (mod.rs:1114-1136), which would 2-align u16-only row types. maple aligns every row to 4 (pager._align4). Checked real exports: every row in data/PIONEER (3567/3567) and rb_ref (101/101) is 4-aligned. So maple's universal 4-alignment matches rekordbox; rekordcrate's per-type scheme is the divergent one (its own code comments admit "we should survey all the different kind of rows and ensure they're aligned properly", mod.rs:1110). No maple change needed.

Where maple's RE goes BEYOND rekordcrate (we know more than the reference)

These are fields rekordcrate leaves as opaque caller-supplied "unknown"s but maple's RE solved/characterised — important because it means rekordcrate cannot be used to validate these; maple's FORMAT_SPEC is the authority:

  • index_shift (row +0x02). rekordcrate calls it index_shift but writes whatever the caller stored — it does not know the value (mod.rs:104-105 "Unknown field, called index_shift by flesniak"). maple derives it as slot_index << 5 and patches it at page-build time (pager.py:142-145, FORMAT_SPEC.md §1). maple is correct where rekordcrate is agnostic.
  • The 8-byte opaque blob (0x14 lo / 0x18+0x1a seed). rekordcrate: unknown2: u32, unknown3: u16 ("always 19048?"), unknown4: u16 ("always 30967?") — three blind unknowns it round-trips verbatim (mod.rs:115-119). maple's RE (FORMAT_SPEC.md §2) proved this is one LE u64 = opaque_track_id_lo | (opaque_seed<<32), a per-track Content ID + per-install seed stored in master.db, non-derivable from metadata, and that the "19048/30967" constants are NOT universal (they only hold for some old exports). maple emits a non-zero stand-in; the real values live in master.db [see re #10]. maple's understanding is strictly deeper than rekordcrate's.
  • bitmask (0x06). rekordcrate: opaque "unknown" round-tripped. maple writes a fixed 788224 (0xC0700) — itself not yet RE'd to a formula, but maple at least pins a known-good value.
  • Strange/"first" pages. rekordcrate leaves the strange-page heap zero (it round-trips parsed input); maple constructs the full DeviceSQL free-space map from scratch byte-for-byte (pager.build_strange_page, verified vs data/acemo_fresh and the real strange pages in data/PIONEER), including the f8 ff ff 1f×1004 map, the 0x03ffffff no-data sentinel, and unknown7=1 for tracks/history. re confirms maple is strictly more faithful than rekordcrate on strange pages. This is from-scratch generation rekordcrate never had to do.
  • RowGroup trailing u2. rekordcrate writes a constant 16 placeholder (Default::default, mod.rs:454) and does not decode it; re confirms genuine rekordbox never uses 16 there — it is the "rows touched by the last transaction" bitmask (FORMAT_SPEC §6). maple writes present_flags (the single-transaction value), closer to real output; either way it is edit-history state (KNOWN_DIVERGENCE §6), not a rekordcrate-validatable field.

Where rekordcrate (and our RE) AGREE the field is unreproducible — with a caveat

rekordcrate documents empty_candidate ("maybe links to a chain of empty pages if the database is ever garbage collected") and next_unused_page ("points past end of file") as allocator/GC state — matching KNOWN_DIVERGENCE.md items 11 and 1. The historical VALUE on an edited reference is indeed unreproducible. BUT my data-page audit (see the FREEZE section above) found empty_candidate is load-bearing structure, not just GC trivia: in data/PIONEER every table's last page has next_page == that table's empty_candidate, a distinct free page < next_unused. So a from-scratch writer must still emit a valid distinct empty_candidate per table (which it fully controls) even though it can't match an edited ref's specific value. Confirms these are correctly on maple's divergence list, not bugs.

GAP — none structural for export.pdb

maple writes a complete, players-accepted export.pdb. Remaining export.pdb divergences from a reference file are all edit-history artifacts (KNOWN_DIVERGENCE.md) that neither library can reproduce from content either. The one content-level gap is the opaque blob (see §4).

2. exportExt.pdb — GAP (nobody writes it; maybe nobody needs to)

  • pyrekordbox: no support.
  • rekordcrate: no support (its pdb module is export.pdb only).
  • maple: does not emit it.

Format facts (Deep Symmetry export analysis; reverse-engineered by Dominik Stolz, crate-digger issue #11): exportExt.pdb is a second, smaller DeviceSQL file beside export.pdb holding only two table types — Tags (type 03) and Tag Tracks (type 04) — i.e. the "My Tag" categorisation feature and its tag→track associations. It uses the same page/row container format as export.pdb.

Assessment / recommendation: exportExt.pdb is optional. It only carries the My-Tag feature; players that don't find it simply show no custom tags, and tracks/playlists from export.pdb play normally. maple has no tag data to populate anyway. Do not block a v1 export on this. If maple ever supports tags, the existing pager.py/PageBuilder already produces the container format — only two new row serialisers (tag row, tag-track row) and a second write_pdb-style call writing the 2-table directory would be needed. Confidence the file is skippable: high (it post-dates the core format and is absent from older exports that still play).

3. ANLZ (.DAT / .EXT / .2EX) — the biggest real GAP

What the references decode

  • pyrekordbox (pyrekordbox.anlz): full parser for PMAI + all section tags — PPTH, PQTZ/PQT2 (beat grids), PCOB/PCO2 (cues/loops), PVBR, PSSI (phrase/song-structure, XOR-masked), PWAV/PWV2 (previews), PWV3/4/5 (detailed + colour waveforms), PWV6/7/PWVC (.2EX). Writing is "planned … but the full structure has to be understood first" — i.e. read-only today.
  • rekordcrate (anlz.rs): parses the same tag set; magic constants present in source: PMAI, PPTH, PQTZ, PVBR, PWAV, PWV2, PCOB, PCO2, PCPT, PCP2, PSSI, PWV3/4/5. It derives BinWrite on the waveform-column structs (WaveformPreviewColumn etc., anlz.rs:368-428) and has a write_encrypted for the XOR-masked PSSI (anlz.rs:902) — so it has piece-level write code, but no whole-file ANLZ builder / CLI to emit a valid .DAT/.EXT from track audio.

What maple does

anlz.py writes a minimal ANLZ0000.DAT: a 28-byte PMAI header + a single PPTH section (u4 len + UTF-16BE path). No beat grid, no waveform, no cues, no .EXT, no .2EX. Rationale in the module docstring: firmware reads PPTH to confirm which audio file the analysis belongs to, and a track whose analyze_path points at nothing can be hidden from the library, so PPTH alone makes the track "appear analysed enough to surface."

CONFIRMATIONS

  • maple's PMAI/PPTH layout (4-cc + u4 len_header + u4 len_tag; PPTH = u4 len_path + UTF-16BE, big-endian throughout) matches the format both references parse. The big-endian-for-ANLZ-vs-little-endian-for-pdb split that anlz.py calls out is corroborated by both libraries.

GAPS maple must close for full faithfulness

  1. No .EXT file. Modern CDJs/XDJs (NXS2+) read ANLZ0000.EXT for colour waveforms (PWV4/5), extended cues (PCO2), and phrase data (PSSI). Without it maple tracks show no colour waveform and no phrase/structure on newer players. The analyze_path in the pdb points at the .DAT; the player derives the .EXT sibling. maple writes neither beyond the .DAT PPTH stub.
  2. No beat grid (PQTZ). Without a beat grid, beat-sync / quantize / loop features have nothing to lock to; the track plays but degrades the CDJ experience. This is the highest-value ANLZ section to add after PPTH.
  3. No waveform (PWAV/PWV3-5). Cosmetic but conspicuous — tracks show a flat or absent waveform. pyrekordbox can parse these to learn the exact byte layout, and rekordcrate's BinWrite waveform-column structs are a concrete reference for the encoding, but maple must generate the waveform data itself (decode audio → downsample → colour-band), which neither library provides.
  4. Generation, not just layout, is the hard part. The byte format is fully documented (both libs + Deep Symmetry); what's missing everywhere is the DSP that produces beat positions and waveform columns from PCM. That DSP is maple's to build — the references only validate the container.

Recommendation: keep the PPTH-only .DAT as the working baseline (it is what gets tracks visible), and treat PQTZ beat grid + a .EXT with PWV5 colour waveform as the next ANLZ milestones. Use pyrekordbox's parser as the round-trip oracle (write a section, parse it back, compare). Confidence the PPTH-only stub is sufficient for visibility/playback: medium-high (matches maple's own deck testing rationale); confidence it's sufficient for a complete experience: low — beat grid/waveform are expected by users.

4. master.db (db6/7) + the opaque Content IDs — GAP maple can close via pyrekordbox

What pyrekordbox does (this is its core strength) — VALIDATED by re #10

  • Reads and writes the encrypted master.db. The db is SQLCipher4 encrypted with a single GLOBAL key, identical on every install (not machine/license-derived). re validated the concrete key (FORMAT_SPEC §2 Round-3): 402fd482c38817c35ffa8ffb8c7d93143b749e7d315df7a81732a1ff43608497. pyrekordbox 0.4.4 + sqlcipher3-wheels installs and wires up correctly, so a valid master.db opens and exposes DjmdContent [see re #10].
  • Exposes DjmdContent (one row per track) with: ID (the rekordbox internal Content ID), FolderPath, FileNameL, AnalysisDataPath (the /PIONEER/USBANLZ/... analysis path), ImagePath (artwork), and FK ids to Artist/Album/Genre/Label/Key/Composer; plus djmdCue, djmdPlaylist/ djmdSongPlaylist, djmdHistory, djmdMixerParam, djmdDevice, etc.

Reality check from re #10 — currently BLOCKED on data, not on method:

  • The captured maple-py/captures/p17328_master.db is corrupt/truncated (zeroed SQLCipher salt + page 1) — no key can open it. The genuine exportLibrary.db shipped beside export.pdb uses a different key scheme that the global master.db key does not open, and pyrekordbox has no exportLibrary.db support. So the IDs have not yet been read on this RE machine — the path works only against a live %APPDATA%/Pioneer/rekordbox6/master.db.
  • opaque_track_id_lo == DjmdContent.ID & 0xFFFFFFFF is a strong inference, still UNTESTED (the lo range fits the 9-digit Content-ID band), not a confirmed equality.
  • The per-install seed (0x21b4a7ab) is NOT a DjmdContent column and its source is still unidentified (likely a library/USB UUID or agentRegistry/local_usn-style marker; needs a readable master.db to find).
  • pyrekordbox add_content lets rekordbox assign the Content ID — there is no API to force a specific ID — and it does not drive a USB export (get_device is read-only). So pyrekordbox is a read source of real IDs, not a writer/exporter and not a way to dictate IDs.

Why this matters for maple (ties to FORMAT_SPEC §2 / KNOWN_DIVERGENCE #9)

maple's RE proved the export.pdb track blob at 0x14/0x18 is the rekordbox Content ID (per-track, 28-bit band) plus a per-install seed, both stored in master.db and not derivable from the audio/metadata maple has. maple currently emits a non-zero stand-in (crc32(id|file_path) lo + TRACK_OPAQUE_SEED = 0x21b4a7ab). Therefore:

  • GAP (closable in principle, not yet proven): real Content IDs. If maple runs on a machine with rekordbox installed, it can use pyrekordbox + the global key to open the LIVE %APPDATA%/Pioneer/rekordbox6/master.db and read DjmdContent.ID per track, then write those into the pdb blob instead of the stand-in. re validated the mechanism (global key, DjmdContent schema) but could NOT confirm ID == opaque_track_id_lo here because the only captured master.db is corrupt — so this is a strong-but-untested lead, not a closed item. Next step (re #10): point pyrekordbox at a live master.db and confirm the equality for the AceMo tracks. The per-install seed 0x21b4a7ab source is still unidentified, so maple keeps the configurable TRACK_OPAQUE_SEED either way.
  • Caveats: (a) works only with a rekordbox install present and a readable master.db (not the corrupt capture), and only for tracks rekordbox already knows; (b) pyrekordbox add_content cannot force a chosen ID (rekordbox assigns it) and does not drive a USB export, so it can supply IDs but cannot replace maple's writer. Pure from-scratch (no rekordbox install) stays a documented divergence (KNOWN_DIVERGENCE #9), with maple's non-zero stand-in.

rekordcrate

No master.db support at all — it is a device-export (USB) library, not a library-database library. So it cannot help with Content IDs.

5. mysettings (*SETTING.DAT) — GAP (small, easy) maple doesn't write

  • pyrekordbox: reads and writes/builds DEVSETTING.DAT, MYSETTING.DAT (40B, CDJ play settings), MYSETTING2.DAT (40B), and DJMMYSETTING.DAT (52B, DJM mixer) — dict-style get/set + save. (DEVSETTING payload still partly ununderstood.)
  • rekordcrate: setting module parses these (read-only).
  • maple: does not write any of them.

These live in PIONEER/ on the USB alongside rekordbox/. They are optional preference files — absent, the player just uses its own defaults; tracks and playlists are unaffected. Recommendation: low priority. If desired, pyrekordbox can build default ones in a few lines (it's read/write), so maple could shell out to / import pyrekordbox to drop standard MYSETTING.DAT + MYSETTING2.DAT rather than re-implementing the 40-byte structs. Confidence they're optional for playback: high.

6. Overall directory layout + path schemes (cross-check) [see re #11 / FORMAT_SPEC §7]

Both references and Deep Symmetry agree on the USB tree maple targets: /PIONEER/rekordbox/export.pdb (+ optional exportExt.pdb), /PIONEER/USBANLZ/<...>/ANLZ0000.{DAT,EXT,2EX}, /PIONEER/Artwork/<...>/a<id>.jpg, plus optional /PIONEER/*SETTING.DAT. maple writes PIONEER/rekordbox/export.pdb (cli.py:156) and PIONEER/USBANLZ/<bucket>/<id>/ANLZ0000.DAT (cli.py:92).

re resolved both path schemes (#11, FORMAT_SPEC §7):

  • Artwork — EXACTLY reproducible, and maple already matches. Real rekordbox uses /PIONEER/Artwork/{id//20+1:05d}/a{id}.jpg (+ a a{id}_m.jpg 240px thumb), 20 ids per bucket folder (bucket 1 = ids 1-19). re verified 176/176 in data/MAPLE, and maple's artwork._bucket() already implements this rule. Reproducible, confirmed — no gap.

  • USBANLZ — NOT reproducible; maple's own scheme is correct-by-construction. Real rekordbox uses /PIONEER/USBANLZ/P{bucket}/{8hex}/ANLZ0000.DAT where the 8-hex is an opaque per-track analysis id (NOT the track id, NOT the row 0x14 lo, NOT any crc of the path — re ruled these out) and the P-bucket (0-127) is not a derivable function of it. Like the 0x14 blob, it is DB-sourced and non-reproducible. maple synthesises its own P{(crc32(path)>>20)&0xFFF:03X}/{crc32(path):08X} and writes the .DAT there, so the pdb's analyze_path and the on-disk file always agree. This is fine: the player only requires every pdb path to resolve to a real on-disk file, not to match rekordbox's id scheme.

Gotcha for the gaps below (from re): tools/build_acemo.py MIRRORS a reference export's paths verbatim and writes no ANLZ/artwork files (it's a byte-diff fixture builder). The production cli.py build() is the one that actually writes each ANLZ/artwork file at the recorded path. Don't conflate the two when reasoning about what maple emits on disk.

7. Bottom line — prioritised GAP list for a full from-scratch export

  1. (high, lead — untested) Real opaque Content IDs via pyrekordbox/master.db — mechanism validated by re (global SQLCipher key 402fd482… + Rekordbox6Database reads DjmdContent), but DjmdContent.ID == opaque_track_id_lo is a strong INFERENCE, NOT validated (the only captured master.db is corrupt — zeroed salt — so re never confirmed the equality on real data). Would close KNOWN_DIVERGENCE #9 if confirmed against a readable live %APPDATA%/Pioneer/rekordbox6/master.db. [see re #10 / FORMAT_SPEC §2 Round-3]
  2. (OPEN — Ghidra trace pending) Per-install opaque_seed source. The seed (0x21b4a7ab / 0xd6d3a567) is NOT a DjmdContent column and re has not located it — likely a per-library UUID/agent/usn marker or a computed value. It needs a readable master.db to inspect candidate rows, OR a Ghidra trace of rekordbox.exe (which re can now attempt — the Ghidra MCP just came online), so it is not permanently blocked. Until resolved, maple keeps the configurable TRACK_OPAQUE_SEED stand-in. This is the one genuinely open question touching the opaque blob. [see re #10 — OPEN; FORMAT_SPEC §2 Round-3]
  3. (high) ANLZ beat grid (PQTZ) + an .EXT colour waveform (PWV5) — needs DSP maple must build; references validate the byte format only. Biggest functional gap on modern players.
  4. (structural — in progress) Per-table empty_candidate / last-page next_page — the FREEZE fix (#13). maple must point each table's last page at a distinct free page recorded in the dir's empty_candidate, not a shared sentinel with empty_candidate=0. See the FREEZE section in §1.
  5. (low) mysettings — optional; pyrekordbox can build them.
  6. (low/skip) exportExt.pdb — optional My-Tag data; only if maple adds tags.
  7. (reference, not a gap) Use rekordcrate as a read-back oracle — parse maple's export.pdb with rekordcrate's CLI to catch container regressions; use pyrekordbox's anlz parser as the oracle for any new ANLZ sections. Caveat (proven this session): rekordcrate's parser tolerates things the firmware rejects (the next_page self-loop) and is itself wrong on some fields (>255-row count, per-type alignment) — so it is a regression oracle, NOT a correctness authority. Validate load-critical fields against REAL export bytes (e.g. data/PIONEER), not against rekordcrate.

Headline confirmation for the team: rekordcrate's writer independently validates maple's export.pdb track-row layout, DeviceSQL string encoding, and page/row-group container down to the 0x5e/0x2a offsets — but it is agnostic exactly where maple's RE added knowledge (index_shift formula, the opaque-id semantics). So maple's FORMAT_SPEC is the authority for those fields, and the remaining real work is ANLZ generation and pulling the real Content IDs from master.db — neither of which any existing library does end-to-end.

Reference in new issue View Git Blame Copy Permalink