OFApp/CLAUDE.md

# CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

## Build & Run

```bash
npm run install:all          # Install server + client deps
npm run dev                  # Concurrent: client (5173) + server (3001)
npm run dev:server           # Server only with --watch
npm run dev:client           # Vite dev server only
npm run build                # Build client to client/dist/
npm start                    # Production server (serves client/dist/)
```

## Deployment

Remote server: `root@10.3.3.11`, password `Intel22`, path `/mnt/user/appdata/OFApp/`

```bash
# Deploy
sshpass -p 'Intel22' rsync -avz --exclude='node_modules' --exclude='.git' --exclude='dist' --exclude='data' /Users/treyt/Desktop/code/OFApp/ root@10.3.3.11:/mnt/user/appdata/OFApp/

# Rebuild container
sshpass -p 'Intel22' ssh root@10.3.3.11 'cd /mnt/user/appdata/OFApp && docker-compose down && docker-compose up -d --build'

# Check logs
sshpass -p 'Intel22' ssh root@10.3.3.11 'docker logs ofapp 2>&1 | tail -20'
```

Port mapping: HTTP 3002->3001, HTTPS 3003->3443. Data volumes at `/mnt/user/downloads/OFApp/{db,media}`.

## Architecture

**Client**: React 18 + Vite + Tailwind. Pages in `client/src/pages/`, API wrappers in `client/src/api.js`. Vite proxies `/api` to `localhost:3001` in dev.

**Server**: Express (ESM modules). All routes are in separate router files mounted in `server/index.js`.

### Server Modules

| File | Purpose |
|------|---------|
| `proxy.js` | OF API proxy with auth headers + media/DRM proxy endpoints |
| `signing.js` | Dynamic request signing (fetches rules from GitHub, auto-refreshes hourly) |
| `download.js` | Background media download orchestration with resume support |
| `db.js` | SQLite (better-sqlite3, WAL mode) — auth, download history, cursors, settings, media index |
| `gallery.js` | Gallery API (SQL-backed media index), video thumbnails, duplicate scanning, filesystem scanner |
| `hls.js` | On-demand HLS transcoding via FFmpeg |
| `settings.js` | Key-value settings API |
| `scrape.js` | Job-based scraping orchestrator — forum + Coomer/Kemono + MediaLink scrapers with progress/logs |
| `scrapers/forum.js` | Cheerio-based forum image scraper (XenForo, generic forums) |
| `scrapers/coomer.js` | Coomer/Kemono API scraper with concurrent downloads |
| `scrapers/medialink.js` | Fapello/gallery-site JSON API scraper — paginates media API, downloads full-size images + videos |
| `widevine.js` | Widevine CDM — WVD parsing, used for browser DRM playback only (not downloads) |
| `drm-download.js` | DRM video download pipeline — MPD parsing, pywidevine key acquisition, decrypt, mux |
| `pywidevine_helper.py` | Python helper using pywidevine lib for Widevine license challenges (called as subprocess) |

### Auth & Signing Flow

All OF API requests go through `proxy.js` which adds auth headers (Cookie, user-id, x-bc, x-of-rev, app-token) plus a dynamic `sign` header generated by `signing.js`. The signing uses SHA-1 with parameters from community-maintained rules fetched from GitHub repos (rafa-9, datawhores, DATAHOARDERS — tried in order).

### DRM Video Playback

OF videos use PallyCon DRM (Widevine for Chrome/Firefox, FairPlay for Safari). The flow:

1. `MediaGrid.jsx` detects `files.drm` on video items, extracts DASH manifest URL + CloudFront cookies (and keeps the parent entity context: `entityType` + `entityId`, e.g. `post/{postId}`)
2. `DrmVideo.jsx` uses Shaka Player to load the DASH manifest via `/api/drm-hls` proxy
3. Shaka's request filter rewrites CDN segment URLs through `/api/drm-hls` (attaches CloudFront cookies server-side)
4. Widevine license challenges go to `/api/drm-license` which forwards to OF's DRM resolver:
   - Own media: `POST /api2/v2/users/media/{mediaId}/drm/?type=widevine`
   - Posts/messages/etc: `POST /api2/v2/users/media/{mediaId}/drm/{entityType}/{entityId}?type=widevine` (e.g. `post/{postId}`)
5. The `/api/drm-hls` proxy handles DASH manifests (injects `<BaseURL>`), HLS playlists (rewrites URLs), and binary segments (pipes through)

**EME requires HTTPS** — the server auto-generates a self-signed cert at `/data/certs/` on first boot. Access DRM content via `https://10.3.3.11:3003`.

### Media Proxy

CDN media from `*.onlyfans.com` is proxied through `/api/media-proxy` to avoid CORS issues. It passes Range headers and caches responses. URLs from `public.` and `thumbs.` subdomains are NOT proxied (client-side check in `MediaGrid.jsx`).

### Download System

`download.js` runs background downloads in-memory (not persisted across restarts). It paginates through `/api2/v2/users/:id/posts/medias`, downloads each file to `MEDIA_PATH/:username/`, records in SQLite, and supports cursor-based resume for interrupted downloads.

**DRM Video Downloads**: When a media item has `files.drm.manifest.dash`, the downloader uses `drm-download.js` to:

1. Fetch & parse the DASH MPD manifest (supports both segmented and on-demand profiles)
2. Extract Widevine PSSH (filters by system ID `edef8ba9-...`, not PlayReady)
3. Call `pywidevine_helper.py` as an async subprocess to get content keys — the helper routes license requests through the local `/api/drm-license` proxy (which handles OF auth/signing)
4. Download encrypted tracks (single file for on-demand, init+segments for segmented)
5. Decrypt with ffmpeg (`-decryption_key`) → mux audio+video

Requires a `.wvd` (Widevine Device) file at `WVD_PATH` and Python 3 + pywidevine installed. Without the `.wvd`, DRM videos are skipped with a log message. The Dockerfile installs `python3 py3-pip` and `pywidevine`.

### Gallery Media Index

The gallery uses a SQLite `media_files` table instead of scanning the filesystem on every request. This makes gallery page loads instant even with tens of thousands of files.

- **Startup scan**: On server boot, `scanMediaFiles()` walks `MEDIA_PATH`, stats every media file, and upserts into `media_files`. Also prunes rows for deleted files/folders. Takes ~6s for ~38k files.
- **Incremental updates**: `download.js`, `scrapers/forum.js`, `scrapers/coomer.js`, and `scrapers/medialink.js` insert into `media_files` after each file is written.
- **Manual rescan**: `POST /api/gallery/rescan` triggers a full re-index (accessible via Settings page "Rescan Media Library" button). Use this after manually adding/removing files from the media folder.
- **Schema**: `media_files(folder, filename, type, size, modified, posted_at)` with indexes on folder, type, modified, posted_at. Unique on `(folder, filename)`.

### Scraper System

`scrape.js` provides a job-based scraping system with three scraper types:

- **Forum scraper** (`scrapers/forum.js`): Scrapes image-hosting forum threads (XenForo-style). Uses Cheerio to parse HTML, finds images in post content areas, derives full-size URLs from thumbnails, filters out avatars/emojis/icons. Supports page range, delay between pages, auto-detection of max page count.
- **Coomer/Kemono scraper** (`scrapers/coomer.js`): Uses the Coomer/Kemono API (`/api/v1/{service}/user/{userId}/posts`) to fetch posts and download attached files. Supports configurable concurrency (1-20 workers) and skips already-downloaded files.
- **MediaLink scraper** (`scrapers/medialink.js`): Scrapes gallery sites like Fapello. Accepts a URL like `https://fapello.to/model/12345`, calls the site's JSON API (`GET /api/media/{userId}/{page}/{order}` with `X-Requested-With: XMLHttpRequest` + `Referer` headers to avoid 403), and downloads `newUrl` (full-size) for images or videos (`type "2"`). Paginates until the API returns empty. Supports configurable concurrency (1-10 workers), delay, and max pages.

Jobs run in-memory with progress tracking, cancellation support, and per-job log history. The client polls for updates every 2s.

## Setup Guides

### Creating a WVD File (Widevine Device)

DRM downloads require a `.wvd` file containing a Widevine L3 CDM. To create one:

1. **Get CDM files** from a rooted Android device or dumped Chrome CDM:
   - `client_id.bin` — the client identification blob
   - `private_key.pem` — the RSA private key (PKCS#1 or PKCS#8)

2. **Install pywidevine** (if not already):
   ```bash
   pip3 install pywidevine
   ```

3. **Create the .wvd file**:
   ```bash
   pywidevine create-device -t ANDROID -l 3 -k private_key.pem -c client_id.bin -o device.wvd
   ```

4. **Place the file** on the server at `/mnt/user/downloads/OFApp/cdm/device.wvd` (mapped to `/data/cdm/device.wvd` inside Docker).

Note: Google periodically revokes L3 CDMs. When revoked, you'll need new `client_id.bin` + `private_key.pem` from a different device/source and regenerate the `.wvd`.

### Updating Auth Cookies

Auth cookies expire periodically. When you see 401 errors in the logs, re-extract from the browser:

1. **Open Firefox** (or Chrome) and log into onlyfans.com
2. **Open DevTools** → Network tab
3. **Click any request** to `onlyfans.com/api2/...`
4. **From the request headers, grab**:
   - **Cookie** — the full cookie string (contains `sess`, `auth_id`, `st`, `csrf`, etc.)
   - **user-id** — your numeric user ID (e.g. `101476031`)
   - **x-bc** — browser checksum hash (e.g. `8dbf86e101ff9265acbfbeb648d74e85092b6206`)
   - **user-agent** — your browser's UA string

5. **Update via the app's settings page**, or directly in the DB:
   ```bash
   sqlite3 /mnt/user/downloads/OFApp/db/ofapp.db
   DELETE FROM auth_config;
   INSERT INTO auth_config (cookie, user_id, x_bc, user_agent) VALUES (
     'sess=...; auth_id=...; st=...; ...',
     '101476031',
     '8dbf86e101ff9265acbfbeb648d74e85092b6206',
     'Mozilla/5.0 ...'
   );
   ```

   In Firefox specifically: DevTools → Storage tab → Cookies → `onlyfans.com` to see individual cookie values, or Network tab → right-click a request → Copy → Copy Request Headers.

## Environment Variables

| Variable | Default | Notes |
|----------|---------|-------|
| `PORT` | 3001 | HTTP |
| `HTTPS_PORT` | 3443 | HTTPS for DRM/EME |
| `DB_PATH` | /data/db/ofapp.db | SQLite |
| `MEDIA_PATH` | /data/media | Downloaded files |
| `DOWNLOAD_DELAY` | 1000 | ms between downloads |
| `HLS_ENABLED` | false | FFmpeg HLS transcoding |
| `WVD_PATH` | /data/cdm/device.wvd | Widevine device file for DRM downloads |

## Key Gotchas

- Widevine license challenges must be treated as raw binary. `server/index.js` mounts `express.raw()` for `/api/drm-license` before the global `express.json()`, and `server/proxy.js` prefers `req.body` (Buffer) with a fallback stream read.
- For subscribed content, `/api/drm-license` must include `entityType` + `entityId` (e.g. `entityType=post&entityId=<postId>`). Missing entity context typically yields OF `403 User have no permissions`, which Shaka surfaces as `6007` (`LICENSE_REQUEST_FAILED`).
- Request signing rules normalize differently across GitHub sources — `signing.js` handles both old format (static_param) and new format (static-param with dashes).
- CloudFront cookies for DRM content are IP-locked to the server's public IP (47.185.183.191). They won't work from a different IP.
- The `/api/drm-hls` proxy skips `skd://` URIs in HLS manifests (FairPlay key identifiers, Safari only).
- Server log prefixes: `[signing]`, `[drm-license]`, `[drm-hls]`, `[download]`, `[drm-download]`, `[widevine]`, `[media-proxy]`, `[hls]`, `[gallery]`.
- DRM downloads require a Widevine L3 device file (`.wvd`). Place it at `/data/cdm/device.wvd` (or set `WVD_PATH`). Without it, DRM videos are silently skipped during bulk downloads.
- The pywidevine subprocess MUST be called with `execAsync` (not `execSync`) because it calls back to the local `/api/drm-license` proxy — `execSync` would deadlock the Node.js event loop.
- MPD PSSH extraction must filter for the Widevine system ID (`edef8ba9-79d6-4ace-a3c8-27dcd51d21ed`), not PlayReady (`9a04f079`). The MPD may contain both.
- OF uses DASH on-demand profile (`isoff-on-demand:2011`) with `<BaseURL>` + `<SegmentBase>` — the entire track is a single encrypted file, not segmented.
- Widevine DRM requires a two-step license flow: first POST `[0x08, 0x04]` for the service certificate, then send the actual challenge. PallyCon's license server is strict about challenge format — our custom protobuf CDM was rejected, so we use pywidevine (proven library) instead.
- Auth cookies expire — user needs to re-extract from browser when they get 401 errors.
- Gallery endpoints read from the `media_files` SQLite index, not the filesystem. If files are added/removed manually outside the app, use the "Rescan Media Library" button in Settings (or `POST /api/gallery/rescan`) to re-index.
- The forum scraper requires the `cheerio` npm package (in server dependencies). It's used for HTML parsing only on the server side.