# maple-py — rekordbox `export.pdb` writer + reverse-engineering notes

Python tooling to generate a rekordbox USB export (`PIONEER/rekordbox/export.pdb`
+ `exportExt.pdb` + `USBANLZ` analysis files) so a CDJ/XDJ can read a
self-assembled music library. The writer lives in `src/maple/pdb/`.

This README also documents what we learned reverse-engineering rekordbox's own
exporter, because the on-disk format has real unknowns / magic numbers that the
public kaitai schema (`src/kaitai/rekordbox_pdb.py`) only partially decodes.

## The format, briefly

`export.pdb` is a **DeviceSQL** database: a flat file of 4096-byte **pages**,
grouped into 20 **tables** (tracks, artists, albums, playlists, artwork,
colors, columns, history, and several still-unknown ones). Page 0 is the file
header + table directory; each table is a linked list of pages. See
`src/maple/pdb/{writer,pager,rows,strings}.py`.

## Reverse-engineering findings (rekordbox 7.0.5, Windows)

### Where the export code lives
- The exporter is in **`rekordbox.exe`** (the 89 MB main binary), **not**
  `rekordboxAgent.exe`. Confirmed by string scan: only `rekordbox.exe` contains
  `USBANLZ`, `PIONEER`, `djmd…` table names, `.pdb`. It's a JUCE C++ app.
- `export.pdb` is written by rekordbox's **bundled DeviceSQL "Page Manager"**
  engine — functions prefixed `dsql*` / `edb_dyn_*` (error strings:
  `dsqlOpenDeviceDB`, `edb_dyn_CriticalSectionOn ERROR`,
  `Page Manager (Close/)Open returns ERROR`).
- **Key consequence:** the page-level "magic numbers" maple guesses
  (page flags `0x64`/`0x44`, `0x1fff` markers, `0x3ec`, free/used sizes) are
  **DeviceSQL internals**, not rekordbox-specific. They are NOT immediate
  constants in the code — they're produced by the page-manager logic, so they
  must be read out of the `dsql*`/`edb_dyn_*` functions, not grep'd.

### A rekordbox USB export writes three databases
| file | format | writer |
|------|--------|--------|
| `export.pdb`        | DeviceSQL (CDJ reads this) | DeviceSQL Page Manager |
| `exportExt.pdb`     | DeviceSQL extended         | same |
| `exportLibrary.db`  | **SQLite** (rekordbox's own) | bundled `sqlite3_*` |
| `USBANLZ/**/ANLZ*.DAT/.EXT/.2EX` | per-track waveform/beatgrid | main app |

### `export.pdb` serializer call stack (live `WriteFile` backtrace → Ghidra)
Ghidra static address = `0x140000000 + module_offset` (Ghidra image base is
`0x140000000`; the ASLR runtime base varies per launch — never hardcode it,
read `module+offset` from Frida).

```
0x1420e8804  calls WriteFile  ─┐
0x1420d7358                     │ DeviceSQL write plumbing
0x1420d6e74                     │ (transactions, page I/O)
0x1420d62a4                     │
0x1420eac94                     │
0x1420e07d8                     │
0x1420de140                    ─┘
FUN_141d75c20  ← dsqlOpenDeviceDB / Page Manager (export.pdb-specific)
0x141cad100
0x141cac390
0x141b8b4c0
0x141aca850
0x1423c9ab0   ← outer export driver
```
Other anchors found by string xref: `FUN_14119d2d0` (export orchestrator,
builds `/PIONEER/rekordbox/export.pdb`, walks "Contents"), `FUN_140f37e50`
(builds `PIONEER`/`USBANLZ` paths).

### Gotchas
- **Incremental writes:** rekordbox rewrites only the *changed* pages (seeked
  `WriteFile`s), so a Frida capture is partial. The complete file is on the USB
  (`F:\PIONEER\rekordbox\export.pdb`) — copy it from there for a full diff.
- **CIG / injection:** `rekordboxAgent.exe` refuses live Frida injection
  (Code Integrity Guard). The pdb writer (main `rekordbox.exe`) is hookable, but
  only via **child-gating at spawn** — Windows has **no spawn-gating**.

## Tooling

### Static analysis (Ghidra, headless, in WSL)
- JDK 21 (`~/jdk/jdk-21.0.11+10`), Ghidra 12.1.2. Ghidra rejects JDK 25 — use 21.
- Analyzed project: `~/ghidra_proj` (project `rbx`, program `rekordbox.exe`).
  Full analysis ≈ 1 h with `GHIDRA_HEADLESS_MAXMEM=10G -max-cpu 8`.
- Re-run a script without re-analyzing:
  ```bash
  "$GHIDRA/support/analyzeHeadless" ~/ghidra_proj rbx -process rekordbox.exe \
      -noanalysis -scriptPath ~/ghidra_scripts -postScript FindPdb.java
  ```
  Scripts: `~/ghidra_scripts/FindPdb.java` (string+constant xref search),
  `DecPdb.java` (decompile an address list), `CallTree.java` (call-tree drill).

### Dynamic capture (Frida, on Windows)
- `tools/rbx_pdb_capture.js` — Frida agent. Hooks `NtCreateFile`/`NtWriteFile`/
  `NtClose` + `NtCreateSection`/`NtMapViewOfSection` (memory-mapped writes).
  Reconstructs files under `PIONEER`/`*.pdb` to `captures/`, prints each write's
  backtrace as `rekordbox.exe+0xOFFSET` (→ Ghidra `0x140000000 + offset`).
- `tools/capture.mjs` — Node host (`npm i frida`). Attaches to all rekordbox
  processes, streams `[pid]`-tagged output live, stays alive, **child-gates** so
  the spawn writer is hooked at birth. Clears `captures/` on start.
  ```powershell
  npm i frida                    # once
  node tools\capture.mjs --spawn # launch rekordbox under the gate, then Export to USB
  ```

## Diffing maple's output vs rekordbox
`compare_pdb.py` parses two pdbs with the kaitai schema and summarises tables/
rows. Reference real exports: `data/PIONEER/rekordbox/export.pdb`,
`data/acemo/PIONEER/rekordbox/export.pdb`.

## Status / next steps
- Writer fully located; capture + diff pipeline works.
- To decode a specific page magic number: search the `dsql*`/`edb_dyn_*`
  functions (image region `0x142000000+`) for the page-header builder.
- To fix a rejected export: diff a complete USB `export.pdb` against maple's
  output field-by-field; resolve any wrong value via the matching `dsql` function.