ghidra

maple-py/README.md

@@ -0,0 +1,112 @@
+# maple-py — rekordbox `export.pdb` writer + reverse-engineering notes
+
+Python tooling to generate a rekordbox USB export (`PIONEER/rekordbox/export.pdb`
++ `exportExt.pdb` + `USBANLZ` analysis files) so a CDJ/XDJ can read a
+self-assembled music library. The writer lives in `src/maple/pdb/`.
+
+This README also documents what we learned reverse-engineering rekordbox's own
+exporter, because the on-disk format has real unknowns / magic numbers that the
+public kaitai schema (`src/kaitai/rekordbox_pdb.py`) only partially decodes.
+
+## The format, briefly
+
+`export.pdb` is a **DeviceSQL** database: a flat file of 4096-byte **pages**,
+grouped into 20 **tables** (tracks, artists, albums, playlists, artwork,
+colors, columns, history, and several still-unknown ones). Page 0 is the file
+header + table directory; each table is a linked list of pages. See
+`src/maple/pdb/{writer,pager,rows,strings}.py`.
+
+## Reverse-engineering findings (rekordbox 7.0.5, Windows)
+
+### Where the export code lives
+- The exporter is in **`rekordbox.exe`** (the 89 MB main binary), **not**
+  `rekordboxAgent.exe`. Confirmed by string scan: only `rekordbox.exe` contains
+  `USBANLZ`, `PIONEER`, `djmd…` table names, `.pdb`. It's a JUCE C++ app.
+- `export.pdb` is written by rekordbox's **bundled DeviceSQL "Page Manager"**
+  engine — functions prefixed `dsql*` / `edb_dyn_*` (error strings:
+  `dsqlOpenDeviceDB`, `edb_dyn_CriticalSectionOn ERROR`,
+  `Page Manager (Close/)Open returns ERROR`).
+- **Key consequence:** the page-level "magic numbers" maple guesses
+  (page flags `0x64`/`0x44`, `0x1fff` markers, `0x3ec`, free/used sizes) are
+  **DeviceSQL internals**, not rekordbox-specific. They are NOT immediate
+  constants in the code — they're produced by the page-manager logic, so they
+  must be read out of the `dsql*`/`edb_dyn_*` functions, not grep'd.
+
+### A rekordbox USB export writes three databases
+| file | format | writer |
+|------|--------|--------|
+| `export.pdb`        | DeviceSQL (CDJ reads this) | DeviceSQL Page Manager |
+| `exportExt.pdb`     | DeviceSQL extended         | same |
+| `exportLibrary.db`  | **SQLite** (rekordbox's own) | bundled `sqlite3_*` |
+| `USBANLZ/**/ANLZ*.DAT/.EXT/.2EX` | per-track waveform/beatgrid | main app |
+
+### `export.pdb` serializer call stack (live `WriteFile` backtrace → Ghidra)
+Ghidra static address = `0x140000000 + module_offset` (Ghidra image base is
+`0x140000000`; the ASLR runtime base varies per launch — never hardcode it,
+read `module+offset` from Frida).
+
+```
+0x1420e8804  calls WriteFile  ─┐
+0x1420d7358                     │ DeviceSQL write plumbing
+0x1420d6e74                     │ (transactions, page I/O)
+0x1420d62a4                     │
+0x1420eac94                     │
+0x1420e07d8                     │
+0x1420de140                    ─┘
+FUN_141d75c20  ← dsqlOpenDeviceDB / Page Manager (export.pdb-specific)
+0x141cad100
+0x141cac390
+0x141b8b4c0
+0x141aca850
+0x1423c9ab0   ← outer export driver
+```
+Other anchors found by string xref: `FUN_14119d2d0` (export orchestrator,
+builds `/PIONEER/rekordbox/export.pdb`, walks "Contents"), `FUN_140f37e50`
+(builds `PIONEER`/`USBANLZ` paths).
+
+### Gotchas
+- **Incremental writes:** rekordbox rewrites only the *changed* pages (seeked
+  `WriteFile`s), so a Frida capture is partial. The complete file is on the USB
+  (`F:\PIONEER\rekordbox\export.pdb`) — copy it from there for a full diff.
+- **CIG / injection:** `rekordboxAgent.exe` refuses live Frida injection
+  (Code Integrity Guard). The pdb writer (main `rekordbox.exe`) is hookable, but
+  only via **child-gating at spawn** — Windows has **no spawn-gating**.
+
+## Tooling
+
+### Static analysis (Ghidra, headless, in WSL)
+- JDK 21 (`~/jdk/jdk-21.0.11+10`), Ghidra 12.1.2. Ghidra rejects JDK 25 — use 21.
+- Analyzed project: `~/ghidra_proj` (project `rbx`, program `rekordbox.exe`).
+  Full analysis ≈ 1 h with `GHIDRA_HEADLESS_MAXMEM=10G -max-cpu 8`.
+- Re-run a script without re-analyzing:
+  ```bash
+  "$GHIDRA/support/analyzeHeadless" ~/ghidra_proj rbx -process rekordbox.exe \
+      -noanalysis -scriptPath ~/ghidra_scripts -postScript FindPdb.java
+  ```
+  Scripts: `~/ghidra_scripts/FindPdb.java` (string+constant xref search),
+  `DecPdb.java` (decompile an address list), `CallTree.java` (call-tree drill).
+
+### Dynamic capture (Frida, on Windows)
+- `tools/rbx_pdb_capture.js` — Frida agent. Hooks `NtCreateFile`/`NtWriteFile`/
+  `NtClose` + `NtCreateSection`/`NtMapViewOfSection` (memory-mapped writes).
+  Reconstructs files under `PIONEER`/`*.pdb` to `captures/`, prints each write's
+  backtrace as `rekordbox.exe+0xOFFSET` (→ Ghidra `0x140000000 + offset`).
+- `tools/capture.mjs` — Node host (`npm i frida`). Attaches to all rekordbox
+  processes, streams `[pid]`-tagged output live, stays alive, **child-gates** so
+  the spawn writer is hooked at birth. Clears `captures/` on start.
+  ```powershell
+  npm i frida                    # once
+  node tools\capture.mjs --spawn # launch rekordbox under the gate, then Export to USB
+  ```
+
+## Diffing maple's output vs rekordbox
+`compare_pdb.py` parses two pdbs with the kaitai schema and summarises tables/
+rows. Reference real exports: `data/PIONEER/rekordbox/export.pdb`,
+`data/acemo/PIONEER/rekordbox/export.pdb`.
+
+## Status / next steps
+- Writer fully located; capture + diff pipeline works.
+- To decode a specific page magic number: search the `dsql*`/`edb_dyn_*`
+  functions (image region `0x142000000+`) for the page-header builder.
+- To fix a rejected export: diff a complete USB `export.pdb` against maple's
+  output field-by-field; resolve any wrong value via the matching `dsql` function.