maple/maple-py/re/REFERENCE_COMPARISON.md

# REFERENCE_COMPARISON.md — pyrekordbox & rekordcrate vs maple

What the two mature open-source rekordbox libraries can decode/encode, how
that lines up with maple's writer and our RE findings, and the concrete GAPS
maple must close for a full from-scratch USB export.

- **pyrekordbox** (Python, dylanljones) — https://pyrekordbox.readthedocs.io
- **rekordcrate** (Rust, Holzhaus) — https://holzhaus.github.io/rekordcrate/
  local source at `/mnt/c/Users/ntr/code/rekordcrate/src`
- **maple** — `maple-py/src/maple/*`, this `re/` dir.

Scope note (coordination): the per-format **library-capability** comparison and
the from-scratch-export gaps are mine (task #12). The master.db SQLCipher key
mechanics and opaque Content ID identity are owned by `re` (#10); the exact
USBANLZ/artwork on-disk path schemes by `re` (#11). Where those are cited below
they are tagged `[see re #10]` / `[see re #11]` and should be treated as
`re`'s authority, not re-derived here.

---

## 0. One-screen capability matrix

| Format | pyrekordbox | rekordcrate | maple |
|---|---|---|---|
| **export.pdb** | parse only (via the `pdb` parser, read) | **parse + WRITE** (`BinWrite` on rows/pages, full roundtrip) | **WRITE only** (from-scratch, the whole point) |
| **exportExt.pdb** | not supported | not supported | not written |
| **ANLZ (.DAT/.EXT/.2EX)** | **parse only** ("creating … is planned") | parse + *partial* write (binrw `BinWrite` derived on waveform/cue structs; not a whole-file builder) | **WRITE** (minimal: PMAI + PPTH only) |
| **artwork (.jpg thumbs)** | reads `ImagePath` from db only | n/a (just the pdb `Artwork` row path string) | **WRITE** (extracts embedded art) |
| **master.db (db6/7)** | **READ + WRITE** (SQLCipher unlock) | not supported | not used |
| **mysetting (\*SETTING.DAT)** | **READ + WRITE** (build from scratch) | parse only (`setting` module) | not written |
| **rekordbox XML** | **READ + WRITE** | parse only (`xml` module) | not used |

The headline: **no existing library writes a complete from-scratch USB export.**
rekordcrate writes `export.pdb` rows but not ANLZ/exportExt and has no
metadata source; pyrekordbox writes master.db + mysetting + XML but not the USB
`export.pdb`/ANLZ. maple is filling a real gap. The reference libs are most
valuable to maple as (a) a **read-back oracle** (parse maple's output and diff)
and (b) for pyrekordbox, a **source of the real opaque Content IDs** from
master.db.

---

## 1. export.pdb

### What rekordcrate does (the closest reference writer)
`pdb/mod.rs` decodes **and writes** every row type maple emits: Track, Artist,
Album, Artwork, Color, Genre, Key, Label, PlaylistTreeNode, PlaylistEntry,
ColumnEntry, HistoryPlaylist, HistoryEntry (`pub enum Row`, lines ~1064-1103).
It has a genuine page serializer: `write_page_contents` (mod.rs:212),
`write_options_and_get_row_offset` (mod.rs:463), per-row-type alignment
(`align_by`, mod.rs:1121-1133), and a `Track` `BinWrite` impl (mod.rs:934).

### Where rekordcrate CONFIRMS maple's layout (high-value confirmations)
- **Track fixed prefix = 0x5e bytes, then a 21×u2 (0x2a) offset block.**
  rekordcrate asserts `start_of_string_section - base_position == 0x5e` and
  seeks `0x2a` for 21 string offsets (mod.rs ~986-996). maple's
  `TRACK_FIXED_PREFIX = 0x5E`, `TRACK_HEADER_LEN = 0x5E + 21*2`
  (`rows.py:8-10`) and its 21-string list match field-for-field, including the
  exact string order (isrc, …, analyze_path, analyze_date, comment, title,
  unknown_string8, filename, file_path). **Independent confirmation of maple's
  track row.**
- **Field order of the 0x5e prefix** is identical between rekordcrate's
  `Track::write_options` and maple's `serialize_track`: unknown1/index_shift/
  bitmask/sample_rate/composer_id/file_size/`unknown2`(=opaque lo @0x14)/
  `unknown3`+`unknown4`(=seed @0x18/0x1a)/artwork_id/key_id/orig_artist_id/
  label_id/remixer_id/bitrate/track_number/tempo/genre_id/album_id/artist_id/
  id/disc/play_count/year/sample_depth/duration/unknown5/color/rating/unknown6/
  unknown7.
- **DeviceSQL string encoding.** rekordcrate `pdb/string.rs` roundtrips the
  same three forms maple's `strings.py` writes: short-ASCII
  (`(len<<1)|1`), long-ASCII (`0x40`), long-UTF16LE (`0x90`). Its tests encode
  `"I ❤ Rust"` as 0x90 and a long string as 0x40 — exactly maple's selection
  rule. The empty string is `0x03` in both. **Confirms strings.py.**
- **Row-group / page-index mechanics.** rekordcrate's `RowGroup` (mod.rs:451,
  2102) uses the same 16-rows-per-group, backward-growing offset index, and
  presence bitmask maple builds in `pager.py`. `Header` (mod.rs:162) carries
  the same 13-bit num_row_offsets / 11-bit num_rows packing maple writes
  (`pager.py:131`).

### Where rekordcrate caught a maple BUG — `next_page` chain termination (the FREEZE)
rekordcrate `pdb/mod.rs` documents `Page::next_page` (lines 262-265) precisely:
> *"Index of the next page with the same page type. If this page is the last one
> of that type, the page index stored in the field will point **past the end of
> the file**."*

maple's writer originally set the **last** page of each table chain to
`next_page = its own page_index` (a self-pointer). That is exactly what
rekordcrate says must NOT happen — the last page must point *forward, past the
allocated pages*. A self-referencing `next_page` makes a firmware page-walk that
terminates on "next_page past EOF" **loop forever on the last page → the deck
freezes.** This is the root cause being fixed in task #13: `writer.py` now points
every last page at a dedicated free-pool sentinel page just past the overflow
region (`free_sentinel`, materialised zero-filled) and sets `next_unused_page`
beyond it — which removes the self-loop.
(rekordcrate's own `read_pages`, mod.rs:185-196, terminates the walk on
`page_index == table.last_page` from the directory, so its *parser* tolerates a
self-pointer — which is why maple's read-back oracle never flagged it. The
*firmware* uses the chain-walk, so only a real deck exposed the freeze.
Lesson: a clean rekordcrate parse is NOT proof the chain is firmware-valid.)

> **Verified against a real export — the EXACT rule (not just "past EOF").**
> I decoded `data/PIONEER/rekordbox/export.pdb` (847 tracks, 174 pages,
> `next_unused=177`). rekordcrate's "points past the end of file" is a
> simplification; the real, consistent rule across **all 20 tables** is:
> **each table's last/only page sets `next_page == that table's
> `empty_candidate`**, where `empty_candidate` (table-dir entry +4) is a
> *distinct, nonzero* page index `< next_unused` — the head of that table's
> free-page chain. Evidence: every last page's `next_page` equalled its own
> dir `empty_candidate` (type0 173→174=ec; type1 4→53=ec; type19 40→41=ec; …),
> and those targets (174,53,158,163,54,49,42,46,170,20,22,175,176,165,30,32,
> 43,44,45,41) are all `< next_unused`. Strange-only tables already match
> (next_page = page_idx+1 = their ec; free-map `data_ptr=0x03ffffff`) — maple's
> `build_strange_page` does this correctly.
>
> So maple's current #13 fix (a **single shared sentinel** for every data
> table + `empty_candidate=0` in the directory) removes the freeze but still
> **diverges** from real rekordbox, which uses a **distinct per-table
> `empty_candidate`** that the directory entry also records (nonzero). If
> firmware ever cross-checks `next_page` against the directory's
> `empty_candidate`, the shared sentinel + zero ec could still misbehave.
> Recommended (reported to `maple`): give each data table its own free page,
> set `next_page = that page` AND write that index into the table-dir
> `empty_candidate`, keep them all `< next_unused`. This is straightforwardly
> reproducible (unlike the *value* of `empty_candidate` on an edited ref, which
> is GC state — here we only need a valid distinct free page, which a
> from-scratch writer fully controls).

### Where rekordcrate is WRONG and maple is RIGHT — page row-count header (>255 rows)
Audited because rekordcrate's `Page` header (mod.rs:277-377) reads the row count
as **`num_rows_small: u8` @0x18** plus **`num_rows_large: u16` @0x22** with a
`calculate_num_rows` rule (`large if large>small && large!=0x1fff else small`).
maple instead writes the **canonical crate-digger .ksy bitfield** at 0x18:
13-bit `num_row_offsets` + 11-bit `num_rows`, then `page_flags` @0x1b
(`pager.py:131`). These are two different readings of the same bytes — so I
**verified against a real export** (`data/PIONEER`, which has 284-row
playlist-entry pages):
- Real page 18 bytes @0x18 = `1c 81 23 24`. The 24-bit LE value `0x23811c`
  decodes (kaitai/maple way) to `num_row_offsets=284, num_rows=284` — **correct**.
- maple's packer reproduces `1c 81 23` **exactly** for n=284 (and `a6 c0 14` for
  166, `7e c0 0f` for 126 — all matching real pages byte-for-byte).
- rekordcrate's reading of the *same* bytes: `num_rows_small = 0x1c = 28` (just
  the low byte) and `num_rows_large @0x22 = 283` (which is really the
  `transaction_row_index` edit field), giving `calculate_num_rows = 283` — i.e.
  rekordcrate would report **283 rows on a 284-row page, and 28 if the
  transaction field were 0.** Its small/large model only coincidentally agrees
  when `n <= 255`.
- **Conclusion:** maple's header encoding is correct and firmware-faithful for
  pages up to 2047 rows (the 11-bit max); rekordcrate's `num_rows_small`/
  `num_rows_large` is an older/divergent decode that breaks for `n > 255`. This
  matters because maple *can* emit >255-row pages (playlist_entries fits ~284/
  page, colors ~394). No maple change needed — this is a maple-is-right result.

### Where rekordcrate's row alignment diverges (maple matches real rekordbox)
rekordcrate aligns each row by its Rust struct's natural `align_of_val`
(mod.rs:1114-1136), which would 2-align u16-only row types. maple aligns **every
row to 4** (`pager._align4`). Checked real exports: **every row in
`data/PIONEER` (3567/3567) and `rb_ref` (101/101) is 4-aligned.** So maple's
universal 4-alignment matches rekordbox; rekordcrate's per-type scheme is the
divergent one (its own code comments admit "we should survey all the different
kind of rows and ensure they're aligned properly", mod.rs:1110). No maple change
needed.

### Where maple's RE goes BEYOND rekordcrate (we know more than the reference)
These are fields rekordcrate leaves as opaque caller-supplied "unknown"s but
maple's RE solved/characterised — important because it means **rekordcrate
cannot be used to validate these; maple's FORMAT_SPEC is the authority**:

- **`index_shift` (row +0x02).** rekordcrate calls it `index_shift` but writes
  whatever the caller stored — it does **not** know the value (mod.rs:104-105
  "Unknown field, called `index_shift` by flesniak"). maple **derives** it as
  `slot_index << 5` and patches it at page-build time (`pager.py:142-145`,
  FORMAT_SPEC.md §1). maple is correct where rekordcrate is agnostic.
- **The 8-byte opaque blob (0x14 lo / 0x18+0x1a seed).** rekordcrate:
  `unknown2: u32`, `unknown3: u16` ("always 19048?"), `unknown4: u16`
  ("always 30967?") — three blind unknowns it round-trips verbatim
  (mod.rs:115-119). maple's RE (FORMAT_SPEC.md §2) proved this is one LE u64 =
  `opaque_track_id_lo | (opaque_seed<<32)`, a per-track Content ID + per-install
  seed stored in master.db, **non-derivable** from metadata, and that the
  "19048/30967" constants are NOT universal (they only hold for some old
  exports). maple emits a non-zero stand-in; the real values live in master.db
  `[see re #10]`. maple's understanding is strictly deeper than rekordcrate's.
- **`bitmask` (0x06).** rekordcrate: opaque "unknown" round-tripped. maple
  writes a fixed `788224` (0xC0700) — itself not yet RE'd to a formula, but
  maple at least pins a known-good value.
- **Strange/"first" pages.** rekordcrate leaves the strange-page heap **zero**
  (it round-trips parsed input); maple **constructs** the full DeviceSQL
  free-space map from scratch byte-for-byte (`pager.build_strange_page`,
  verified vs `data/acemo_fresh` and the real strange pages in data/PIONEER),
  including the `f8 ff ff 1f`×1004 map, the `0x03ffffff` no-data sentinel, and
  `unknown7=1` for tracks/history. `re` confirms maple is **strictly more
  faithful than rekordcrate on strange pages.** This is from-scratch generation
  rekordcrate never had to do.
- **RowGroup trailing u2.** rekordcrate writes a constant `16` placeholder
  (`Default::default`, mod.rs:454) and does **not** decode it; `re` confirms
  genuine rekordbox **never** uses 16 there — it is the "rows touched by the
  last transaction" bitmask (FORMAT_SPEC §6). maple writes `present_flags` (the
  single-transaction value), closer to real output; either way it is
  edit-history state (KNOWN_DIVERGENCE §6), not a rekordcrate-validatable field.

### Where rekordcrate (and our RE) AGREE the field is unreproducible — with a caveat
rekordcrate documents `empty_candidate` ("maybe links to a chain of empty
pages if the database is ever garbage collected") and `next_unused_page`
("points past end of file") as allocator/GC state — matching
KNOWN_DIVERGENCE.md items 11 and 1. The **historical VALUE** on an edited
reference is indeed unreproducible. **BUT** my data-page audit (see the FREEZE
section above) found `empty_candidate` is **load-bearing structure**, not just
GC trivia: in data/PIONEER every table's last page has
`next_page == that table's empty_candidate`, a distinct free page `< next_unused`.
So a from-scratch writer must still emit a *valid* distinct `empty_candidate`
per table (which it fully controls) even though it can't match an edited ref's
specific value. **Confirms** these are correctly on
maple's divergence list, not bugs.

### GAP — none structural for export.pdb
maple writes a complete, players-accepted export.pdb. Remaining export.pdb
divergences from a reference file are all **edit-history artifacts**
(KNOWN_DIVERGENCE.md) that neither library can reproduce from content either.
The one content-level gap is the opaque blob (see §4).

---

## 2. exportExt.pdb  — GAP (nobody writes it; maybe nobody needs to)

- **pyrekordbox:** no support.
- **rekordcrate:** no support (its `pdb` module is export.pdb only).
- **maple:** does not emit it.

Format facts (Deep Symmetry export analysis; reverse-engineered by Dominik
Stolz, crate-digger issue #11): `exportExt.pdb` is a *second, smaller*
DeviceSQL file beside `export.pdb` holding only two table types — **Tags
(type 03)** and **Tag Tracks (type 04)** — i.e. the "My Tag" categorisation
feature and its tag→track associations. It uses the same page/row container
format as export.pdb.

**Assessment / recommendation:** exportExt.pdb is **optional**. It only carries
the My-Tag feature; players that don't find it simply show no custom tags, and
tracks/playlists from export.pdb play normally. maple has no tag data to
populate anyway. **Do not block a v1 export on this.** If maple ever supports
tags, the existing `pager.py`/`PageBuilder` already produces the container
format — only two new row serialisers (tag row, tag-track row) and a second
`write_pdb`-style call writing the 2-table directory would be needed.
Confidence the file is skippable: high (it post-dates the core format and is
absent from older exports that still play).

---

## 3. ANLZ (.DAT / .EXT / .2EX)  — the biggest real GAP

### What the references decode
- **pyrekordbox** (`pyrekordbox.anlz`): full **parser** for PMAI + all section
  tags — PPTH, PQTZ/PQT2 (beat grids), PCOB/PCO2 (cues/loops), PVBR, PSSI
  (phrase/song-structure, XOR-masked), PWAV/PWV2 (previews), PWV3/4/5
  (detailed + colour waveforms), PWV6/7/PWVC (.2EX). Writing is **"planned …
  but the full structure has to be understood first"** — i.e. **read-only
  today.**
- **rekordcrate** (`anlz.rs`): parses the same tag set; magic constants present
  in source: PMAI, PPTH, PQTZ, PVBR, PWAV, PWV2, PCOB, PCO2, PCPT, PCP2, PSSI,
  PWV3/4/5. It derives `BinWrite` on the waveform-column structs
  (`WaveformPreviewColumn` etc., anlz.rs:368-428) and has a `write_encrypted`
  for the XOR-masked PSSI (anlz.rs:902) — so it has *piece-level* write code,
  but **no whole-file ANLZ builder / CLI to emit a valid .DAT/.EXT from track
  audio.**

### What maple does
`anlz.py` writes a **minimal** `ANLZ0000.DAT`: a 28-byte PMAI header + a single
**PPTH** section (u4 len + UTF-16BE path). No beat grid, no waveform, no cues,
no `.EXT`, no `.2EX`. Rationale in the module docstring: firmware reads PPTH to
confirm which audio file the analysis belongs to, and a track whose
`analyze_path` points at *nothing* can be hidden from the library, so PPTH
alone makes the track "appear analysed enough to surface."

### CONFIRMATIONS
- maple's PMAI/PPTH layout (4-cc + u4 len_header + u4 len_tag; PPTH = u4
  len_path + UTF-16**BE**, big-endian throughout) matches the format both
  references parse. The big-endian-for-ANLZ-vs-little-endian-for-pdb split that
  `anlz.py` calls out is corroborated by both libraries.

### GAPS maple must close for full faithfulness
1. **No `.EXT` file.** Modern CDJs/XDJs (NXS2+) read `ANLZ0000.EXT` for colour
   waveforms (PWV4/5), extended cues (PCO2), and phrase data (PSSI). Without it
   maple tracks show **no colour waveform and no phrase/structure** on newer
   players. The `analyze_path` in the pdb points at the `.DAT`; the player
   derives the `.EXT` sibling. maple writes neither beyond the `.DAT` PPTH stub.
2. **No beat grid (PQTZ).** Without a beat grid, beat-sync / quantize / loop
   features have nothing to lock to; the track plays but degrades the CDJ
   experience. This is the highest-value ANLZ section to add after PPTH.
3. **No waveform (PWAV/PWV3-5).** Cosmetic but conspicuous — tracks show a flat
   or absent waveform. pyrekordbox can *parse* these to learn the exact byte
   layout, and rekordcrate's `BinWrite` waveform-column structs are a concrete
   reference for the encoding, but **maple must generate the waveform data
   itself** (decode audio → downsample → colour-band), which neither library
   provides.
4. **Generation, not just layout, is the hard part.** The byte format is fully
   documented (both libs + Deep Symmetry); what's missing everywhere is the
   DSP that *produces* beat positions and waveform columns from PCM. That DSP
   is maple's to build — the references only validate the container.

**Recommendation:** keep the PPTH-only `.DAT` as the working baseline (it is
what gets tracks visible), and treat PQTZ beat grid + a `.EXT` with PWV5 colour
waveform as the next ANLZ milestones. Use pyrekordbox's parser as the
round-trip oracle (write a section, parse it back, compare). Confidence the
PPTH-only stub is sufficient for *visibility/playback*: medium-high (matches
maple's own deck testing rationale); confidence it's sufficient for a *complete*
experience: low — beat grid/waveform are expected by users.

---

## 4. master.db (db6/7) + the opaque Content IDs  — GAP maple can close via pyrekordbox

### What pyrekordbox does (this is its core strength) — VALIDATED by re #10
- Reads **and writes** the encrypted `master.db`. The db is **SQLCipher4**
  encrypted with a **single GLOBAL key, identical on every install** (not
  machine/license-derived). `re` validated the concrete key (FORMAT_SPEC §2
  Round-3): `402fd482c38817c35ffa8ffb8c7d93143b749e7d315df7a81732a1ff43608497`.
  pyrekordbox 0.4.4 + `sqlcipher3-wheels` installs and wires up correctly, so a
  **valid** master.db opens and exposes `DjmdContent` `[see re #10]`.
- Exposes `DjmdContent` (one row per track) with: **`ID`** (the rekordbox
  internal Content ID), `FolderPath`, `FileNameL`, **`AnalysisDataPath`** (the
  `/PIONEER/USBANLZ/...` analysis path), **`ImagePath`** (artwork), and FK ids
  to Artist/Album/Genre/Label/Key/Composer; plus `djmdCue`, `djmdPlaylist`/
  `djmdSongPlaylist`, `djmdHistory`, `djmdMixerParam`, `djmdDevice`, etc.

> **Reality check from re #10 — currently BLOCKED on data, not on method:**
> - The captured `maple-py/captures/p17328_master.db` is **corrupt/truncated**
>   (zeroed SQLCipher salt + page 1) — no key can open it. The genuine
>   `exportLibrary.db` shipped beside export.pdb uses a *different* key scheme
>   that the global master.db key does not open, and pyrekordbox has no
>   exportLibrary.db support. So the IDs have **not yet been read** on this RE
>   machine — the path works only against a **live**
>   `%APPDATA%/Pioneer/rekordbox6/master.db`.
> - `opaque_track_id_lo == DjmdContent.ID & 0xFFFFFFFF` is a **strong inference,
>   still UNTESTED** (the lo range fits the 9-digit Content-ID band), not a
>   confirmed equality.
> - The per-install **seed** (`0x21b4a7ab`) is **NOT** a DjmdContent column and
>   its source is **still unidentified** (likely a library/USB UUID or
>   agentRegistry/local_usn-style marker; needs a readable master.db to find).
> - pyrekordbox `add_content` lets *rekordbox* assign the Content ID — there is
>   **no API to force a specific ID** — and it does **not** drive a USB export
>   (`get_device` is read-only). So pyrekordbox is a **read source of real IDs**,
>   not a writer/exporter and not a way to dictate IDs.

### Why this matters for maple (ties to FORMAT_SPEC §2 / KNOWN_DIVERGENCE #9)
maple's RE proved the export.pdb track blob at 0x14/0x18 is the rekordbox
**Content ID** (per-track, 28-bit band) plus a **per-install seed**, both stored
in master.db and **not derivable** from the audio/metadata maple has. maple
currently emits a non-zero *stand-in* (`crc32(id|file_path)` lo +
`TRACK_OPAQUE_SEED = 0x21b4a7ab`). Therefore:

- **GAP (closable in principle, not yet proven): real Content IDs.** If maple
  runs on a machine with rekordbox installed, it can use **pyrekordbox + the
  global key to open the LIVE `%APPDATA%/Pioneer/rekordbox6/master.db` and read
  `DjmdContent.ID`** per track, then write those into the pdb blob instead of
  the stand-in. `re` validated the *mechanism* (global key, DjmdContent schema)
  but could NOT confirm `ID == opaque_track_id_lo` here because the only
  captured master.db is corrupt — so this is a **strong-but-untested** lead, not
  a closed item. Next step (re #10): point pyrekordbox at a live master.db and
  confirm the equality for the AceMo tracks. The per-install **seed**
  `0x21b4a7ab` source is still unidentified, so maple keeps the configurable
  `TRACK_OPAQUE_SEED` either way.
- **Caveats:** (a) works only *with a rekordbox install present and a readable
  master.db* (not the corrupt capture), and only for tracks rekordbox already
  knows; (b) pyrekordbox `add_content` cannot **force** a chosen ID (rekordbox
  assigns it) and does **not** drive a USB export, so it can supply IDs but
  cannot replace maple's writer. Pure from-scratch (no rekordbox install) stays
  a documented divergence (KNOWN_DIVERGENCE #9), with maple's non-zero stand-in.

### rekordcrate
No master.db support at all — it is a *device-export* (USB) library, not a
*library-database* library. So it cannot help with Content IDs.

---

## 5. mysettings (\*SETTING.DAT)  — GAP (small, easy) maple doesn't write

- **pyrekordbox:** reads **and writes/builds** `DEVSETTING.DAT`,
  `MYSETTING.DAT` (40B, CDJ play settings), `MYSETTING2.DAT` (40B), and
  `DJMMYSETTING.DAT` (52B, DJM mixer) — dict-style get/set + save. (`DEVSETTING`
  payload still partly ununderstood.)
- **rekordcrate:** `setting` module **parses** these (read-only).
- **maple:** does **not** write any of them.

These live in `PIONEER/` on the USB alongside `rekordbox/`. They are **optional
preference files** — absent, the player just uses its own defaults; tracks and
playlists are unaffected. **Recommendation:** low priority. If desired,
pyrekordbox can *build* default ones in a few lines (it's read/write), so maple
could shell out to / import pyrekordbox to drop standard `MYSETTING.DAT` +
`MYSETTING2.DAT` rather than re-implementing the 40-byte structs. Confidence
they're optional for playback: high.

---

## 6. Overall directory layout + path schemes (cross-check) `[see re #11 / FORMAT_SPEC §7]`

Both references and Deep Symmetry agree on the USB tree maple targets:
`/PIONEER/rekordbox/export.pdb` (+ optional `exportExt.pdb`),
`/PIONEER/USBANLZ/<...>/ANLZ0000.{DAT,EXT,2EX}`,
`/PIONEER/Artwork/<...>/a<id>.jpg`, plus optional `/PIONEER/*SETTING.DAT`.
maple writes `PIONEER/rekordbox/export.pdb` (`cli.py:156`) and
`PIONEER/USBANLZ/<bucket>/<id>/ANLZ0000.DAT` (`cli.py:92`).

`re` resolved both path schemes (#11, FORMAT_SPEC §7):

- **Artwork — EXACTLY reproducible, and maple already matches.** Real rekordbox
  uses `/PIONEER/Artwork/{id//20+1:05d}/a{id}.jpg` (+ a `a{id}_m.jpg` 240px
  thumb), 20 ids per bucket folder (bucket 1 = ids 1-19). `re` verified
  176/176 in data/MAPLE, and maple's `artwork._bucket()` already implements this
  rule. **Reproducible, confirmed — no gap.**

- **USBANLZ — NOT reproducible; maple's own scheme is correct-by-construction.**
  Real rekordbox uses `/PIONEER/USBANLZ/P{bucket}/{8hex}/ANLZ0000.DAT` where the
  8-hex is an **opaque per-track analysis id** (NOT the track id, NOT the row
  0x14 lo, NOT any crc of the path — `re` ruled these out) and the P-bucket
  (0-127) is **not a derivable function** of it. Like the 0x14 blob, it is
  DB-sourced and non-reproducible. maple synthesises its own
  `P{(crc32(path)>>20)&0xFFF:03X}/{crc32(path):08X}` and writes the `.DAT`
  there, so the pdb's `analyze_path` and the on-disk file always agree. **This
  is fine:** the player only requires every pdb path to resolve to a real
  on-disk file, not to match rekordbox's id scheme.

> **Gotcha for the gaps below (from `re`):** `tools/build_acemo.py` MIRRORS a
> reference export's paths verbatim and writes **no** ANLZ/artwork files (it's a
> byte-diff fixture builder). The **production** `cli.py build()` is the one that
> actually writes each ANLZ/artwork file at the recorded path. Don't conflate
> the two when reasoning about what maple emits on disk.

---

## 7. Bottom line — prioritised GAP list for a full from-scratch export

1. **(high, lead — untested) Real opaque Content IDs via pyrekordbox/master.db**
   — mechanism validated by `re` (global SQLCipher key `402fd482…` +
   `Rekordbox6Database` reads `DjmdContent`), but `DjmdContent.ID ==
   opaque_track_id_lo` is a **strong INFERENCE, NOT validated** (the only
   captured master.db is corrupt — zeroed salt — so `re` never confirmed the
   equality on real data). Would close KNOWN_DIVERGENCE #9 *if* confirmed against
   a readable **live** `%APPDATA%/Pioneer/rekordbox6/master.db`.
   `[see re #10 / FORMAT_SPEC §2 Round-3]`
2. **(OPEN — Ghidra trace pending) Per-install `opaque_seed` source.** The seed
   (`0x21b4a7ab` / `0xd6d3a567`) is **NOT** a `DjmdContent` column and `re` has
   **not located it** — likely a per-library UUID/agent/usn marker or a computed
   value. It needs a readable master.db to inspect candidate rows, OR a Ghidra
   trace of rekordbox.exe (which `re` can now attempt — the Ghidra MCP just came
   online), so it is **not permanently blocked.** Until resolved, maple keeps the
   configurable `TRACK_OPAQUE_SEED` stand-in. This is the one genuinely open
   question touching the opaque blob. `[see re #10 — OPEN; FORMAT_SPEC §2 Round-3]`
3. **(high) ANLZ beat grid (PQTZ) + an `.EXT` colour waveform (PWV5)** — needs
   DSP maple must build; references validate the byte format only. Biggest
   functional gap on modern players.
4. **(structural — in progress) Per-table `empty_candidate` / last-page
   `next_page`** — the FREEZE fix (#13). maple must point each table's last page
   at a distinct free page recorded in the dir's `empty_candidate`, not a shared
   sentinel with `empty_candidate=0`. See the FREEZE section in §1.
5. **(low) mysettings** — optional; pyrekordbox can build them.
6. **(low/skip) exportExt.pdb** — optional My-Tag data; only if maple adds tags.
7. **(reference, not a gap) Use rekordcrate as a read-back oracle** — parse
   maple's `export.pdb` with rekordcrate's CLI to catch container regressions;
   use pyrekordbox's anlz parser as the oracle for any new ANLZ sections.
   **Caveat (proven this session):** rekordcrate's parser tolerates things the
   firmware rejects (the `next_page` self-loop) and is itself wrong on some
   fields (>255-row count, per-type alignment) — so it is a *regression* oracle,
   NOT a correctness authority. Validate load-critical fields against REAL
   export bytes (e.g. data/PIONEER), not against rekordcrate.

**Headline confirmation for the team:** rekordcrate's *writer* independently
validates maple's export.pdb track-row layout, DeviceSQL string encoding, and
page/row-group container down to the 0x5e/0x2a offsets — but it is *agnostic*
exactly where maple's RE added knowledge (`index_shift` formula, the opaque-id
semantics). So maple's FORMAT_SPEC is the authority for those fields, and the
remaining real work is **ANLZ generation** and **pulling the real Content IDs
from master.db** — neither of which any existing library does end-to-end.