Add DRM downloads, scrapers, gallery index, and UI improvements

- DRM video download pipeline with pywidevine subprocess for Widevine key acquisition - Scraper system: forum threads, Coomer/Kemono API, and MediaLink (Fapello) scrapers - SQLite-backed media index for instant gallery loads with startup scan - Duplicate detection and gallery filtering/sorting - HLS video component, log viewer, and scrape management UI - Dockerfile updated for Python/pywidevine, docker-compose volume for CDM Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-16 11:29:11 -06:00
parent c60de19348
commit 1e5f54f60b
28 changed files with 4736 additions and 203 deletions
--- a/CLAUDE.md
+++ b/CLAUDE.md
@@ -43,10 +43,17 @@ Port mapping: HTTP 3002->3001, HTTPS 3003->3443. Data volumes at `/mnt/user/down
 | `proxy.js` | OF API proxy with auth headers + media/DRM proxy endpoints |
 | `signing.js` | Dynamic request signing (fetches rules from GitHub, auto-refreshes hourly) |
 | `download.js` | Background media download orchestration with resume support |
-| `db.js` | SQLite (better-sqlite3, WAL mode) — auth, download history, cursors, settings |
-| `gallery.js` | Serves downloaded media as browsable gallery |
+| `db.js` | SQLite (better-sqlite3, WAL mode) — auth, download history, cursors, settings, media index |
+| `gallery.js` | Gallery API (SQL-backed media index), video thumbnails, duplicate scanning, filesystem scanner |
 | `hls.js` | On-demand HLS transcoding via FFmpeg |
 | `settings.js` | Key-value settings API |
+| `scrape.js` | Job-based scraping orchestrator — forum + Coomer/Kemono + MediaLink scrapers with progress/logs |
+| `scrapers/forum.js` | Cheerio-based forum image scraper (XenForo, generic forums) |
+| `scrapers/coomer.js` | Coomer/Kemono API scraper with concurrent downloads |
+| `scrapers/medialink.js` | Fapello/gallery-site JSON API scraper — paginates media API, downloads full-size images + videos |
+| `widevine.js` | Widevine CDM — WVD parsing, used for browser DRM playback only (not downloads) |
+| `drm-download.js` | DRM video download pipeline — MPD parsing, pywidevine key acquisition, decrypt, mux |
+| `pywidevine_helper.py` | Python helper using pywidevine lib for Widevine license challenges (called as subprocess) |

 ### Auth & Signing Flow

@@ -74,6 +81,86 @@ CDN media from `*.onlyfans.com` is proxied through `/api/media-proxy` to avoid C

 `download.js` runs background downloads in-memory (not persisted across restarts). It paginates through `/api2/v2/users/:id/posts/medias`, downloads each file to `MEDIA_PATH/:username/`, records in SQLite, and supports cursor-based resume for interrupted downloads.

+**DRM Video Downloads**: When a media item has `files.drm.manifest.dash`, the downloader uses `drm-download.js` to:
+
+1. Fetch & parse the DASH MPD manifest (supports both segmented and on-demand profiles)
+2. Extract Widevine PSSH (filters by system ID `edef8ba9-...`, not PlayReady)
+3. Call `pywidevine_helper.py` as an async subprocess to get content keys — the helper routes license requests through the local `/api/drm-license` proxy (which handles OF auth/signing)
+4. Download encrypted tracks (single file for on-demand, init+segments for segmented)
+5. Decrypt with ffmpeg (`-decryption_key`) → mux audio+video
+
+Requires a `.wvd` (Widevine Device) file at `WVD_PATH` and Python 3 + pywidevine installed. Without the `.wvd`, DRM videos are skipped with a log message. The Dockerfile installs `python3 py3-pip` and `pywidevine`.
+
+### Gallery Media Index
+
+The gallery uses a SQLite `media_files` table instead of scanning the filesystem on every request. This makes gallery page loads instant even with tens of thousands of files.
+
+- **Startup scan**: On server boot, `scanMediaFiles()` walks `MEDIA_PATH`, stats every media file, and upserts into `media_files`. Also prunes rows for deleted files/folders. Takes ~6s for ~38k files.
+- **Incremental updates**: `download.js`, `scrapers/forum.js`, `scrapers/coomer.js`, and `scrapers/medialink.js` insert into `media_files` after each file is written.
+- **Manual rescan**: `POST /api/gallery/rescan` triggers a full re-index (accessible via Settings page "Rescan Media Library" button). Use this after manually adding/removing files from the media folder.
+- **Schema**: `media_files(folder, filename, type, size, modified, posted_at)` with indexes on folder, type, modified, posted_at. Unique on `(folder, filename)`.
+
+### Scraper System
+
+`scrape.js` provides a job-based scraping system with three scraper types:
+
+- **Forum scraper** (`scrapers/forum.js`): Scrapes image-hosting forum threads (XenForo-style). Uses Cheerio to parse HTML, finds images in post content areas, derives full-size URLs from thumbnails, filters out avatars/emojis/icons. Supports page range, delay between pages, auto-detection of max page count.
+- **Coomer/Kemono scraper** (`scrapers/coomer.js`): Uses the Coomer/Kemono API (`/api/v1/{service}/user/{userId}/posts`) to fetch posts and download attached files. Supports configurable concurrency (1-20 workers) and skips already-downloaded files.
+- **MediaLink scraper** (`scrapers/medialink.js`): Scrapes gallery sites like Fapello. Accepts a URL like `https://fapello.to/model/12345`, calls the site's JSON API (`GET /api/media/{userId}/{page}/{order}` with `X-Requested-With: XMLHttpRequest` + `Referer` headers to avoid 403), and downloads `newUrl` (full-size) for images or videos (`type "2"`). Paginates until the API returns empty. Supports configurable concurrency (1-10 workers), delay, and max pages.
+
+Jobs run in-memory with progress tracking, cancellation support, and per-job log history. The client polls for updates every 2s.
+
+## Setup Guides
+
+### Creating a WVD File (Widevine Device)
+
+DRM downloads require a `.wvd` file containing a Widevine L3 CDM. To create one:
+
+1. **Get CDM files** from a rooted Android device or dumped Chrome CDM:
+   - `client_id.bin` — the client identification blob
+   - `private_key.pem` — the RSA private key (PKCS#1 or PKCS#8)
+
+2. **Install pywidevine** (if not already):
+   ```bash
+   pip3 install pywidevine
+   ```
+
+3. **Create the .wvd file**:
+   ```bash
+   pywidevine create-device -t ANDROID -l 3 -k private_key.pem -c client_id.bin -o device.wvd
+   ```
+
+4. **Place the file** on the server at `/mnt/user/downloads/OFApp/cdm/device.wvd` (mapped to `/data/cdm/device.wvd` inside Docker).
+
+Note: Google periodically revokes L3 CDMs. When revoked, you'll need new `client_id.bin` + `private_key.pem` from a different device/source and regenerate the `.wvd`.
+
+### Updating Auth Cookies
+
+Auth cookies expire periodically. When you see 401 errors in the logs, re-extract from the browser:
+
+1. **Open Firefox** (or Chrome) and log into onlyfans.com
+2. **Open DevTools** → Network tab
+3. **Click any request** to `onlyfans.com/api2/...`
+4. **From the request headers, grab**:
+   - **Cookie** — the full cookie string (contains `sess`, `auth_id`, `st`, `csrf`, etc.)
+   - **user-id** — your numeric user ID (e.g. `101476031`)
+   - **x-bc** — browser checksum hash (e.g. `8dbf86e101ff9265acbfbeb648d74e85092b6206`)
+   - **user-agent** — your browser's UA string
+
+5. **Update via the app's settings page**, or directly in the DB:
+   ```bash
+   sqlite3 /mnt/user/downloads/OFApp/db/ofapp.db
+   DELETE FROM auth_config;
+   INSERT INTO auth_config (cookie, user_id, x_bc, user_agent) VALUES (
+     'sess=...; auth_id=...; st=...; ...',
+     '101476031',
+     '8dbf86e101ff9265acbfbeb648d74e85092b6206',
+     'Mozilla/5.0 ...'
+   );
+   ```
+
+   In Firefox specifically: DevTools → Storage tab → Cookies → `onlyfans.com` to see individual cookie values, or Network tab → right-click a request → Copy → Copy Request Headers.
+
 ## Environment Variables

 | Variable | Default | Notes |
@@ -84,6 +171,7 @@ CDN media from `*.onlyfans.com` is proxied through `/api/media-proxy` to avoid C
 | `MEDIA_PATH` | /data/media | Downloaded files |
 | `DOWNLOAD_DELAY` | 1000 | ms between downloads |
 | `HLS_ENABLED` | false | FFmpeg HLS transcoding |
+| `WVD_PATH` | /data/cdm/device.wvd | Widevine device file for DRM downloads |

 ## Key Gotchas

@@ -92,4 +180,12 @@ CDN media from `*.onlyfans.com` is proxied through `/api/media-proxy` to avoid C
 - Request signing rules normalize differently across GitHub sources — `signing.js` handles both old format (static_param) and new format (static-param with dashes).
 - CloudFront cookies for DRM content are IP-locked to the server's public IP (47.185.183.191). They won't work from a different IP.
 - The `/api/drm-hls` proxy skips `skd://` URIs in HLS manifests (FairPlay key identifiers, Safari only).
- Server log prefixes: `[signing]`, `[drm-license]`, `[drm-hls]`, `[download]`, `[media-proxy]`, `[hls]`.
+- Server log prefixes: `[signing]`, `[drm-license]`, `[drm-hls]`, `[download]`, `[drm-download]`, `[widevine]`, `[media-proxy]`, `[hls]`, `[gallery]`.
+- DRM downloads require a Widevine L3 device file (`.wvd`). Place it at `/data/cdm/device.wvd` (or set `WVD_PATH`). Without it, DRM videos are silently skipped during bulk downloads.
+- The pywidevine subprocess MUST be called with `execAsync` (not `execSync`) because it calls back to the local `/api/drm-license` proxy — `execSync` would deadlock the Node.js event loop.
+- MPD PSSH extraction must filter for the Widevine system ID (`edef8ba9-79d6-4ace-a3c8-27dcd51d21ed`), not PlayReady (`9a04f079`). The MPD may contain both.
+- OF uses DASH on-demand profile (`isoff-on-demand:2011`) with `<BaseURL>` + `<SegmentBase>` — the entire track is a single encrypted file, not segmented.
+- Widevine DRM requires a two-step license flow: first POST `[0x08, 0x04]` for the service certificate, then send the actual challenge. PallyCon's license server is strict about challenge format — our custom protobuf CDM was rejected, so we use pywidevine (proven library) instead.
+- Auth cookies expire — user needs to re-extract from browser when they get 401 errors.
+- Gallery endpoints read from the `media_files` SQLite index, not the filesystem. If files are added/removed manually outside the app, use the "Rescan Media Library" button in Settings (or `POST /api/gallery/rescan`) to re-index.
+- The forum scraper requires the `cheerio` npm package (in server dependencies). It's used for HTML parsing only on the server side.