113 lines
5.5 KiB
Markdown
113 lines
5.5 KiB
Markdown
# maple-py — rekordbox `export.pdb` writer + reverse-engineering notes
|
|
|
|
Python tooling to generate a rekordbox USB export (`PIONEER/rekordbox/export.pdb`
|
|
+ `exportExt.pdb` + `USBANLZ` analysis files) so a CDJ/XDJ can read a
|
|
self-assembled music library. The writer lives in `src/maple/pdb/`.
|
|
|
|
This README also documents what we learned reverse-engineering rekordbox's own
|
|
exporter, because the on-disk format has real unknowns / magic numbers that the
|
|
public kaitai schema (`src/kaitai/rekordbox_pdb.py`) only partially decodes.
|
|
|
|
## The format, briefly
|
|
|
|
`export.pdb` is a **DeviceSQL** database: a flat file of 4096-byte **pages**,
|
|
grouped into 20 **tables** (tracks, artists, albums, playlists, artwork,
|
|
colors, columns, history, and several still-unknown ones). Page 0 is the file
|
|
header + table directory; each table is a linked list of pages. See
|
|
`src/maple/pdb/{writer,pager,rows,strings}.py`.
|
|
|
|
## Reverse-engineering findings (rekordbox 7.0.5, Windows)
|
|
|
|
### Where the export code lives
|
|
- The exporter is in **`rekordbox.exe`** (the 89 MB main binary), **not**
|
|
`rekordboxAgent.exe`. Confirmed by string scan: only `rekordbox.exe` contains
|
|
`USBANLZ`, `PIONEER`, `djmd…` table names, `.pdb`. It's a JUCE C++ app.
|
|
- `export.pdb` is written by rekordbox's **bundled DeviceSQL "Page Manager"**
|
|
engine — functions prefixed `dsql*` / `edb_dyn_*` (error strings:
|
|
`dsqlOpenDeviceDB`, `edb_dyn_CriticalSectionOn ERROR`,
|
|
`Page Manager (Close/)Open returns ERROR`).
|
|
- **Key consequence:** the page-level "magic numbers" maple guesses
|
|
(page flags `0x64`/`0x44`, `0x1fff` markers, `0x3ec`, free/used sizes) are
|
|
**DeviceSQL internals**, not rekordbox-specific. They are NOT immediate
|
|
constants in the code — they're produced by the page-manager logic, so they
|
|
must be read out of the `dsql*`/`edb_dyn_*` functions, not grep'd.
|
|
|
|
### A rekordbox USB export writes three databases
|
|
| file | format | writer |
|
|
|------|--------|--------|
|
|
| `export.pdb` | DeviceSQL (CDJ reads this) | DeviceSQL Page Manager |
|
|
| `exportExt.pdb` | DeviceSQL extended | same |
|
|
| `exportLibrary.db` | **SQLite** (rekordbox's own) | bundled `sqlite3_*` |
|
|
| `USBANLZ/**/ANLZ*.DAT/.EXT/.2EX` | per-track waveform/beatgrid | main app |
|
|
|
|
### `export.pdb` serializer call stack (live `WriteFile` backtrace → Ghidra)
|
|
Ghidra static address = `0x140000000 + module_offset` (Ghidra image base is
|
|
`0x140000000`; the ASLR runtime base varies per launch — never hardcode it,
|
|
read `module+offset` from Frida).
|
|
|
|
```
|
|
0x1420e8804 calls WriteFile ─┐
|
|
0x1420d7358 │ DeviceSQL write plumbing
|
|
0x1420d6e74 │ (transactions, page I/O)
|
|
0x1420d62a4 │
|
|
0x1420eac94 │
|
|
0x1420e07d8 │
|
|
0x1420de140 ─┘
|
|
FUN_141d75c20 ← dsqlOpenDeviceDB / Page Manager (export.pdb-specific)
|
|
0x141cad100
|
|
0x141cac390
|
|
0x141b8b4c0
|
|
0x141aca850
|
|
0x1423c9ab0 ← outer export driver
|
|
```
|
|
Other anchors found by string xref: `FUN_14119d2d0` (export orchestrator,
|
|
builds `/PIONEER/rekordbox/export.pdb`, walks "Contents"), `FUN_140f37e50`
|
|
(builds `PIONEER`/`USBANLZ` paths).
|
|
|
|
### Gotchas
|
|
- **Incremental writes:** rekordbox rewrites only the *changed* pages (seeked
|
|
`WriteFile`s), so a Frida capture is partial. The complete file is on the USB
|
|
(`F:\PIONEER\rekordbox\export.pdb`) — copy it from there for a full diff.
|
|
- **CIG / injection:** `rekordboxAgent.exe` refuses live Frida injection
|
|
(Code Integrity Guard). The pdb writer (main `rekordbox.exe`) is hookable, but
|
|
only via **child-gating at spawn** — Windows has **no spawn-gating**.
|
|
|
|
## Tooling
|
|
|
|
### Static analysis (Ghidra, headless, in WSL)
|
|
- JDK 21 (`~/jdk/jdk-21.0.11+10`), Ghidra 12.1.2. Ghidra rejects JDK 25 — use 21.
|
|
- Analyzed project: `~/ghidra_proj` (project `rbx`, program `rekordbox.exe`).
|
|
Full analysis ≈ 1 h with `GHIDRA_HEADLESS_MAXMEM=10G -max-cpu 8`.
|
|
- Re-run a script without re-analyzing:
|
|
```bash
|
|
"$GHIDRA/support/analyzeHeadless" ~/ghidra_proj rbx -process rekordbox.exe \
|
|
-noanalysis -scriptPath ~/ghidra_scripts -postScript FindPdb.java
|
|
```
|
|
Scripts: `~/ghidra_scripts/FindPdb.java` (string+constant xref search),
|
|
`DecPdb.java` (decompile an address list), `CallTree.java` (call-tree drill).
|
|
|
|
### Dynamic capture (Frida, on Windows)
|
|
- `tools/rbx_pdb_capture.js` — Frida agent. Hooks `NtCreateFile`/`NtWriteFile`/
|
|
`NtClose` + `NtCreateSection`/`NtMapViewOfSection` (memory-mapped writes).
|
|
Reconstructs files under `PIONEER`/`*.pdb` to `captures/`, prints each write's
|
|
backtrace as `rekordbox.exe+0xOFFSET` (→ Ghidra `0x140000000 + offset`).
|
|
- `tools/capture.mjs` — Node host (`npm i frida`). Attaches to all rekordbox
|
|
processes, streams `[pid]`-tagged output live, stays alive, **child-gates** so
|
|
the spawn writer is hooked at birth. Clears `captures/` on start.
|
|
```powershell
|
|
npm i frida # once
|
|
node tools\capture.mjs --spawn # launch rekordbox under the gate, then Export to USB
|
|
```
|
|
|
|
## Diffing maple's output vs rekordbox
|
|
`compare_pdb.py` parses two pdbs with the kaitai schema and summarises tables/
|
|
rows. Reference real exports: `data/PIONEER/rekordbox/export.pdb`,
|
|
`data/acemo/PIONEER/rekordbox/export.pdb`.
|
|
|
|
## Status / next steps
|
|
- Writer fully located; capture + diff pipeline works.
|
|
- To decode a specific page magic number: search the `dsql*`/`edb_dyn_*`
|
|
functions (image region `0x142000000+`) for the page-header builder.
|
|
- To fix a rejected export: diff a complete USB `export.pdb` against maple's
|
|
output field-by-field; resolve any wrong value via the matching `dsql` function.
|