admin/AppStore
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Phase 4-5: build pipeline + device enrollment

Browse Source 
Builder service (Mac mini):
- Build worker: xcodebuild archive + export + fastlane signing + upload to unraid
- /api/build/upload (source archive) and /api/build/git (clone) ingest paths
- SSE-streamed build logs, builds list UI, live status updates
- /api/devices/from-enrollment bridge endpoint (shared-secret auth)

Storefront (unraid):
- /enroll/ public flow: landing page, mobileconfig generator, callback parser
- Forwards extracted UDIDs to the Mac mini builder for ASC registration
- docker-compose.yml now passes BUILDER_URL and BUILDER_SHARED_SECRET

Updated CLAUDE.md with full architecture, deploy flow, and gotchas.
This commit is contained in:
trey
2026-04-11 14:04:32 -05:00
parent e9b6936904
commit 8dbe87da2e
14 changed files with 1203 additions and 44 deletions
Download Patch File Download Diff File Expand all files Collapse all files

196
CLAUDE.md
View File

@@ -1,79 +1,191 @@
# iOS App Store
Self-hosted iOS OTA distribution server. Node.js/Express + SQLite + Docker, deployed on unraid behind Nginx Proxy Manager.
Hybrid iOS distribution system. Two independent services sharing one git repo:
- **`/` (root)** — the public **storefront** on unraid. Hosts IPAs, serves OTA installs, handles device enrollment. Reached by iPhones at `https://appstore.treytartt.com`.
- **`builder/`** — the private **build console** on the Mac mini. Takes source code (archive or git URL), runs xcodebuild + fastlane, pushes finished IPAs to the storefront. Reached by the developer on the LAN at `http://Treys-Mac-mini.local:3090` (`10.3.3.192:3090`).
The split exists because `xcodebuild` needs macOS and the Mac mini is the only mac we have, but unraid is a much better place to store + serve IPAs long-term.
## Live deployment
- **URL**: https://appstore.treytartt.com
- **Container**: `ios-appstore` on unraid (port `3080` internally, proxied via NPM at host `10.3.3.11:3080`)
- **App code**: `/mnt/user/appdata/ios-appstore/` on unraid (Dockerfile, source, compose, `.env`)
- **Data volume**: `/mnt/user/downloads/ios-appstore/` mounted as `/data` (SQLite DB, IPAs, icons)
### Storefront (unraid)
This split is intentional — app code in appdata, persistent data in downloads. Don't put data volumes in appdata or app source in downloads.
- **URL**: https://appstore.treytartt.com (public, Let's Encrypt via NPM)
- **Container**: `ios-appstore` on unraid, port `3080` internally
- **App code**: `/mnt/user/appdata/ios-appstore/` (Dockerfile, source, compose, `.env`)
- **Data volume**: `/mnt/user/downloads/ios-appstore/` → mounted as `/data` (SQLite DB, IPAs, icons)
- **NPM proxy host #16**: `appstore.treytartt.com` → `10.3.3.11:3080`, SSL forced
- **Env vars** (in `.env` on the server):
- `ADMIN_PASSWORD`, `API_TOKEN`, `SESSION_SECRET`, `BASE_URL`
- `BUILDER_URL=http://10.3.3.192:3090` — LAN address of the Mac mini builder
- `BUILDER_SHARED_SECRET` — must match `builder/.env` on the Mac mini
The `.env` on the server holds `ADMIN_PASSWORD`, `API_TOKEN`, and `SESSION_SECRET`. Read it from `/mnt/user/appdata/ios-appstore/.env` when you need them.
### Builder (Mac mini)
- **URL**: http://Treys-Mac-mini.local:3090 (LAN-only, no SSL, no public DNS)
- **Native Node** (not Docker — `xcodebuild` can't run in a container on macOS)
- **App code**: `/Users/m4mini/AppStoreBuilder/app/` (copied from `builder/` subtree via `builder/bin/deploy.sh`)
- **Data**: `/Users/m4mini/AppStoreBuilder/data/` (SQLite + ASC keys + source archives + build artifacts + logs)
- **Process supervision**: launchd — `~/Library/LaunchAgents/com.88oak.appstorebuilder.plist` (KeepAlive, RunAtLoad)
- **Env vars** (in `builder/.env`, loaded non-destructively by `src/server.js`):
- `ADMIN_PASSWORD`, `SESSION_SECRET`, `DATA_DIR`, `PORT`, `BUILDER_SHARED_SECRET`
**Important**: The builder code must NOT live under `~/Desktop/` when running via launchd. macOS TCC blocks launchd-spawned processes from reading Desktop, which causes the Node process to hang on `__getcwd` during startup. That's why we copy to `/Users/m4mini/AppStoreBuilder/app/` via the deploy script instead of pointing launchd directly at the git checkout in `~/Desktop/code/ios-appstore/builder/`.
## Deploy flow
### Storefront (unraid)
```bash
# 1. Sync local changes to unraid (excludes node_modules, data, .env)
rsync -avz --exclude node_modules --exclude data --exclude .env \
# Sync changes (excluding builder/, data, .env, node_modules)
rsync -avz --exclude node_modules --exclude data --exclude .env --exclude builder \
/Users/m4mini/Desktop/code/ios-appstore/ \
unraid:/mnt/user/appdata/ios-appstore/
# 2. Rebuild and restart
# Rebuild + restart
ssh unraid "cd /mnt/user/appdata/ios-appstore && docker compose up -d --build"
# 3. Verify
ssh unraid "docker logs ios-appstore --tail 20 && curl -s http://localhost:3080/api/health"
# Verify
curl -s https://appstore.treytartt.com/api/health
```
`docker-compose.yml` builds the image from local source — no registry. The data volume persists across rebuilds.
### Builder (Mac mini)
## Architecture
```bash
# Single command: rsync to /Users/m4mini/AppStoreBuilder/app/, kickstart launchd, health check
/Users/m4mini/Desktop/code/ios-appstore/builder/bin/deploy.sh
```
- `src/server.js` — Express app, all routes, multer upload handling
## Architecture overview
```
iPhone ────► https://appstore.treytartt.com ◄──── Developer on LAN
│ │
│ │
▼ ▼
NPM + LE (unraid) http://Treys-Mac-mini.local:3090
│ │
▼ ▼
ios-appstore container AppStoreBuilder (Node + launchd)
(Docker, Node 20 Alpine) ─────────────────────────────────
───────────────────── • Source upload + git clone
• /api/apps browse • xcodebuild archive + export
• /api/upload (IPAs in) • fastlane sigh (ad-hoc profiles)
• /api/manifest OTA • ASC API (devices, profiles)
• /api/download IPA • Build queue + log streaming
• /enroll/ public flow ─────►• /api/devices/from-enrollment
▲ │
│ │
└──────────────────────────────┘
finished IPAs POSTed
to /api/upload via
the existing API_TOKEN
```
## Storefront (root `/src/`)
- `src/server.js` — Express app, all routes, multer upload handling, `/enroll/*` bridge
- `src/db.js` — SQLite schema (apps, builds, devices)
- `src/ipa-parser.js` — Unzips IPA, extracts `Info.plist` and app icon
- `src/manifest.js` — Generates the OTA manifest plist iOS fetches
- `src/mobileconfig.js` — Generates the `.mobileconfig` Profile Service payload and parses callback plists
- `src/auth.js` — Session middleware (web UI) + token middleware (API)
- `views/` — Login, app listing, upload pages (vanilla HTML)
- `views/` — Login, app listing, upload, enroll pages
- `public/` — CSS + client-side JS
Apps are keyed by bundle ID. Uploading the same bundle ID adds a new build to the existing app instead of duplicating it.
### Storefront auth
## Auth model
- **Session cookies** for browser users (`ADMIN_PASSWORD`)
- **`X-Api-Token` header** for CLI/automation (`API_TOKEN`) — this is what the Mac mini uses when POSTing IPAs
- **Public (no auth)**: `/api/manifest/:id`, `/api/download/:id`, `/enroll/*` — iOS fetches these unauthenticated
Two parallel auth schemes:
- **Session cookies** for browser users (login at `/login` with `ADMIN_PASSWORD`)
- **`X-Api-Token` header** for CLI/automation (`API_TOKEN`)
## Builder (`builder/`)
`requireAuth` accepts either. The manifest and IPA download endpoints (`/api/manifest/:id`, `/api/download/:id`) are intentionally **public** — iOS fetches them unauthenticated during the OTA install.
- `builder/src/server.js` — Express app, session auth, mounts all routes, starts the build worker
- `builder/src/db.js` — SQLite schema (settings, devices, apps, profiles, build_jobs)
- `builder/src/auth.js` — Session (web UI) + shared-secret (enrollment bridge from unraid)
- `builder/src/asc-api.js` — App Store Connect REST client (ES256 JWT, `/v1/devices`, `/v1/profiles`, `/v1/bundleIds`)
- `builder/src/profile-manager.js` — Wraps `fastlane sigh`, caches `.mobileprovision` files, auto-installs into `~/Library/MobileDevice/Provisioning Profiles/`
- `builder/src/build-worker.js` — In-process build queue: preparing → signing → archiving → exporting → uploading → succeeded
- `builder/src/build-routes.js` — `/api/build/upload`, `/api/build/git`, `/api/builds`, `/api/builds/:id/logs` (SSE)
- `builder/fastlane/Fastfile` — Single `generate_adhoc` lane using the ASC API key
- `builder/bin/deploy.sh` — Copies source to `/Users/m4mini/AppStoreBuilder/app/` and kickstarts launchd
- `builder/views/` — Login, builds, build, devices, settings pages
- `builder/public/` — CSS (copied from the storefront for visual continuity) + client JS
## OTA gotchas
### How a build works
- **Development-signed IPAs cannot be installed OTA.** iOS only allows OTA installs of `ad-hoc` or `enterprise`-signed builds. If a user reports "integrity could not be verified", first check the export method in their `ExportOptions.plist`.
- Ad-hoc requires a **distribution certificate** (not a development cert) and an **ad-hoc provisioning profile** with the target device UDIDs registered.
- HTTPS is mandatory and must use a trusted CA. Self-signed certs don't work on iOS 12+. NPM's Let's Encrypt cert handles this.
- The manifest's `bundle-identifier` must exactly match the IPA's bundle ID.
1. User posts source (.zip/.tar.gz or git URL) to `/api/build/upload` or `/api/build/git`. Server creates a `build_jobs` row with `status=pending`, extracts/clones the source into `data/source/<job-id>/`, and kicks the worker.
2. **preparing**: worker finds `.xcodeproj`/`.xcworkspace`, picks a scheme, runs `xcodebuild -showBuildSettings -json` to extract every target's `PRODUCT_BUNDLE_IDENTIFIER` and the `DEVELOPMENT_TEAM`.
3. **signing**: for each bundle ID, `profile-manager.getProfile()` ensures a fresh ad-hoc profile exists — serving from cache if possible, running `fastlane sigh` if stale. Each profile is installed into `~/Library/MobileDevice/Provisioning Profiles/` so `xcodebuild` finds it.
4. **archiving**: `xcodebuild archive` with `CODE_SIGN_STYLE=Manual`, the detected team ID, `-allowProvisioningUpdates`. xcodebuild matches bundle IDs to the pre-installed profiles automatically.
5. **exporting**: generate `ExportOptions.plist` with `method=ad-hoc` and the full `provisioningProfiles` map, then `xcodebuild -exportArchive`.
6. **uploading**: the produced `.ipa` is POSTed to `https://appstore.treytartt.com/api/upload` using the existing `API_TOKEN`. The storefront's existing parser + DB insert + manifest generation run unchanged.
7. **succeeded**: clean up source + archive (keep log + IPA + ExportOptions.plist in `data/build/<job-id>/`).
## Testing changes
Profile cache invalidation: whenever a device is added/deleted (manually via UI or via the enrollment bridge), `invalidateProfilesForDeviceChange()` clears `updated_at` on every row in `profiles`, forcing the next build to regenerate via `sigh`.
For backend changes, after deploying:
```bash
# Health
curl -s https://appstore.treytartt.com/api/health
## Enrollment flow
# List apps (needs token)
curl -s -H "X-Api-Token: $TOKEN" https://appstore.treytartt.com/api/apps
1. Tester opens `https://appstore.treytartt.com/enroll` on their iPhone.
2. Tap "Install Profile" → downloads `/enroll/profile.mobileconfig` (Profile Service payload pointing back at `/enroll/callback`). iOS shows "Not Signed" — acceptable for an internal store, just tap Install.
3. iOS installs the profile, collects device attributes (UDID, PRODUCT, VERSION, DEVICE_NAME, SERIAL), wraps them in a CMS-signed plist, and POSTs to `/enroll/callback`.
4. The unraid storefront doesn't validate the CMS signature — it just scans the raw body for the inner `<?xml … </plist>` block and extracts UDID/name/model.
5. The unraid server forwards `{udid, name, model}` to the Mac mini at `http://10.3.3.192:3090/api/devices/from-enrollment` with `Authorization: Bearer <BUILDER_SHARED_SECRET>`.
6. The builder's `/api/devices/from-enrollment` endpoint:
- Upserts into the local `devices` table
- Calls `asc.registerDevice()` to register with Apple via the App Store Connect API
- Clears the profile cache so the next build picks up the new device
7. iOS is redirected (303) to `/enroll/success`.
# Upload an IPA
curl -X POST https://appstore.treytartt.com/api/upload \
-H "X-Api-Token: $TOKEN" \
-F "ipa=@/path/to/App.ipa" \
-F "notes=test"
```
The shared secret lives in both `.env` files and must match. Rotate by updating both sides.
Get `$TOKEN` from `/mnt/user/appdata/ios-appstore/.env` on unraid.
## Endpoints quick reference
For frontend changes, just rsync + restart and refresh the browser.
### Storefront (public URL)
| Method | Path | Auth | Description |
|--------|------|------|-------------|
| GET | `/` | session | App listing UI |
| GET | `/upload` | session | Upload page |
| POST | `/api/upload` | token | Upload IPA (used by Mac mini builder) |
| GET | `/api/apps` | token | List apps |
| GET | `/api/manifest/:id` | public | iOS install manifest |
| GET | `/api/download/:id` | public | Download IPA |
| GET | `/enroll` | public | Enrollment landing page |
| GET | `/enroll/profile.mobileconfig` | public | Profile Service payload |
| POST | `/enroll/callback` | public (raw body) | iOS UDID posts here |
| GET | `/enroll/success` | public | Post-enrollment page |
### Builder (LAN URL)
| Method | Path | Auth | Description |
|--------|------|------|-------------|
| GET | `/` | session | Build list UI |
| GET | `/build` | session | New build form |
| GET | `/devices` | session | Device management UI |
| GET | `/settings` | session | ASC + unraid settings |
| POST | `/api/build/upload` | session | Upload source archive |
| POST | `/api/build/git` | session | Clone git repo |
| GET | `/api/builds` | session | List jobs |
| GET | `/api/builds/:id/logs` | session | SSE log stream |
| GET | `/api/devices` | session | List devices |
| POST | `/api/devices` | session | Register (local + ASC) |
| POST | `/api/devices/from-enrollment` | shared secret | Called by unraid enroll bridge |
| GET | `/api/profile/:bundleId` | session | Fetch/generate ad-hoc profile |
| POST | `/api/settings/test-asc` | session | Verify ASC key works |
| POST | `/api/settings/test-unraid` | session | Verify unraid API token works |
## Gotchas
- **Dev-signed IPAs cannot be installed OTA.** The builder always produces ad-hoc builds via fastlane sigh. Manual uploads via `/api/upload` still work, but the uploader is responsible for the signing method.
- **`builder/` code must not live under `~/Desktop/`** when launchd runs it — TCC blocks the process. That's why `deploy.sh` copies to `/Users/m4mini/AppStoreBuilder/app/`.
- **Docker compose env vars are substituted at compose-time** from the host `.env`, not read from `.env` at runtime inside the container. When adding a new env var to the storefront, update BOTH `.env` on the server AND `docker-compose.yml`.
- **The builder's Node version is 25.x**, the storefront's container is Node 20. They're independent; mixing package-lock.json files between the two will not work.
- **Device cache invalidation is eager**: adding or removing any device marks every profile stale. This is fine for a personal setup but would be worth scoping per-app in a larger deployment.
## Local credentials (look these up, don't hardcode)
- Storefront admin password, API token, session secret: `/mnt/user/appdata/ios-appstore/.env` on unraid
- Builder admin password, session secret, shared secret: `/Users/m4mini/Desktop/code/ios-appstore/builder/.env` (dev) and `/Users/m4mini/AppStoreBuilder/app/.env` (deployed — same file, synced via deploy.sh excluding .env so they can diverge)