admin/honeyDueAPI
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Harden API security: input validation, safe auth extraction, new tests, and deploy config

Browse Source 
Comprehensive security hardening from audit findings:
- Add validation tags to all DTO request structs (max lengths, ranges, enums)
- Replace unsafe type assertions with MustGetAuthUser helper across all handlers
- Remove query-param token auth from admin middleware (prevents URL token leakage)
- Add request validation calls in handlers that were missing c.Validate()
- Remove goroutines in handlers (timezone update now synchronous)
- Add sanitize middleware and path traversal protection (path_utils)
- Stop resetting admin passwords on migration restart
- Warn on well-known default SECRET_KEY
- Add ~30 new test files covering security regressions, auth safety, repos, and services
- Add deploy/ config, audit digests, and AUDIT_FINDINGS documentation

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
Trey t
2026-03-02 09:48:01 -06:00
parent 56d6fa4514
commit 7690f07a2b
123 changed files with 8321 additions and 750 deletions
Download Patch File Download Diff File Expand all files Collapse all files

18
deploy/.gitignore vendored Normal file
View File

@@ -0,0 +1,18 @@
# Local deploy inputs (copy from *.example files)
cluster.env
registry.env
prod.env
# Local secret material
secrets/*.txt
secrets/*.p8
# Keep templates and docs tracked
!*.example
!README.md
!shit_deploy_cant_do.md
!swarm-stack.prod.yml
!scripts/
!scripts/**
!secrets/*.example
!secrets/README.md

134
deploy/README.md Normal file
View File

@@ -0,0 +1,134 @@
# Deploy Folder
This folder is the full production deploy toolkit for `myCribAPI-go`.
Run deploy with:
```bash
./.deploy_prod
```
The script will refuse to run until all required values are set.
## First-Time Prerequisite: Create The Swarm Cluster
You must do this once before `./.deploy_prod` can work.
1. SSH to manager #1 and initialize Swarm:
```bash
docker swarm init --advertise-addr <manager1-private-ip>
```
2. On manager #1, get join commands:
```bash
docker swarm join-token manager
docker swarm join-token worker
```
3. SSH to each additional node and run the appropriate `docker swarm join ...` command.
4. Verify from manager #1:
```bash
docker node ls
```
## Security Requirements Before Public Launch
Use this as a mandatory checklist before you route production traffic.
### 1) Firewall Rules (Node-Level)
Apply firewall rules to all Swarm nodes:
- SSH port (for example `2222/tcp`): your IP only
- `80/tcp`, `443/tcp`: Hetzner LB only (or Cloudflare IP ranges only if no LB)
- `2377/tcp`: Swarm nodes only
- `7946/tcp,udp`: Swarm nodes only
- `4789/udp`: Swarm nodes only
- Everything else: blocked
### 2) SSH Hardening
On each node, harden `/etc/ssh/sshd_config`:
```text
Port 2222
PermitRootLogin no
PasswordAuthentication no
PubkeyAuthentication yes
AllowUsers deploy
```
### 3) Cloudflare Origin Lockdown
- Keep public DNS records proxied (orange cloud on).
- Point Cloudflare to LB, not node IPs.
- Do not publish Swarm node IPs in DNS.
- Enforce firewall source restrictions so public traffic cannot bypass Cloudflare/LB.
### 4) Secrets Policy
- Keep runtime secrets in Docker Swarm secrets only.
- Do not put production secrets in git or plain `.env` files.
- `./.deploy_prod` already creates versioned Swarm secrets from files in `deploy/secrets/`.
- Rotate secrets after incidents or credential exposure.
### 5) Data Path Security
- Neon/Postgres: `DB_SSLMODE=require`, strong DB password, Neon IP allowlist limited to node IPs.
- Backblaze B2: HTTPS only, scoped app keys (not master key), least-privilege bucket access.
- Swarm overlay: encrypted network enabled in stack (`driver_opts.encrypted: "true"`).
### 6) Dozzle Hardening
- Keep Dozzle private (no public DNS/ingress).
- Put auth/SSO in front (Cloudflare Access or equivalent).
- Prefer a Docker socket proxy with restricted read-only scope.
### 7) Backup + Restore Readiness
- Postgres PITR path tested in staging.
- Redis persistence enabled and restore path tested.
- Written runbook for restore and secret rotation.
- Named owner for incident response.
## Files You Fill In
Paste your values into these files:
- `deploy/cluster.env`
- `deploy/registry.env`
- `deploy/prod.env`
- `deploy/secrets/postgres_password.txt`
- `deploy/secrets/secret_key.txt`
- `deploy/secrets/email_host_password.txt`
- `deploy/secrets/fcm_server_key.txt`
- `deploy/secrets/apns_auth_key.p8`
If one is missing, the deploy script auto-copies it from its `.example` template and exits so you can fill it.
## What `./.deploy_prod` Does
1. Validates all required config files and credentials.
2. Builds and pushes `api`, `worker`, and `admin` images.
3. Uploads deploy bundle to your Swarm manager over SSH.
4. Creates versioned Docker secrets on the manager.
5. Deploys the stack with `docker stack deploy --with-registry-auth`.
6. Waits until service replicas converge.
7. Runs an HTTP health check (if `DEPLOY_HEALTHCHECK_URL` is set).
## Useful Flags
Environment flags:
- `SKIP_BUILD=1 ./.deploy_prod` to deploy already-pushed images.
- `SKIP_HEALTHCHECK=1 ./.deploy_prod` to skip final URL check.
- `DEPLOY_TAG=<tag> ./.deploy_prod` to deploy a specific image tag.
## Important
- `deploy/shit_deploy_cant_do.md` lists the manual tasks this script cannot automate.
- Keep real credentials and secret files out of git.

22
deploy/cluster.env.example Normal file
View File

@@ -0,0 +1,22 @@
# Swarm manager connection
DEPLOY_MANAGER_HOST=CHANGEME_MANAGER_IP_OR_HOSTNAME
DEPLOY_MANAGER_USER=deploy
DEPLOY_MANAGER_SSH_PORT=22
DEPLOY_SSH_KEY_PATH=~/.ssh/id_ed25519
# Stack settings
DEPLOY_STACK_NAME=casera
DEPLOY_REMOTE_DIR=/opt/casera/deploy
DEPLOY_WAIT_SECONDS=420
DEPLOY_HEALTHCHECK_URL=https://api.casera.app/api/health/
# Replicas and published ports
API_REPLICAS=3
WORKER_REPLICAS=2
ADMIN_REPLICAS=1
API_PORT=8000
ADMIN_PORT=3000
DOZZLE_PORT=9999
# Build behavior
PUSH_LATEST_TAG=true

73
deploy/prod.env.example Normal file
View File

@@ -0,0 +1,73 @@
# API service settings
DEBUG=false
ALLOWED_HOSTS=api.casera.app,casera.app
CORS_ALLOWED_ORIGINS=https://casera.app,https://admin.casera.app
TIMEZONE=UTC
BASE_URL=https://casera.app
PORT=8000
# Admin service settings
NEXT_PUBLIC_API_URL=https://api.casera.app
ADMIN_PANEL_URL=https://admin.casera.app
# Database (Neon recommended)
DB_HOST=CHANGEME_NEON_HOST
DB_PORT=5432
POSTGRES_USER=CHANGEME_DB_USER
POSTGRES_DB=casera
DB_SSLMODE=require
DB_MAX_OPEN_CONNS=25
DB_MAX_IDLE_CONNS=10
DB_MAX_LIFETIME=600s
# Redis (in stack defaults to redis://redis:6379/0)
REDIS_URL=redis://redis:6379/0
REDIS_DB=0
# Email (password goes in deploy/secrets/email_host_password.txt)
EMAIL_HOST=smtp.gmail.com
EMAIL_PORT=587
EMAIL_USE_TLS=true
EMAIL_HOST_USER=CHANGEME_EMAIL_USER
DEFAULT_FROM_EMAIL=Casera <noreply@casera.app>
# Push notifications
# APNS private key goes in deploy/secrets/apns_auth_key.p8
APNS_AUTH_KEY_ID=CHANGEME_APNS_KEY_ID
APNS_TEAM_ID=CHANGEME_APNS_TEAM_ID
APNS_TOPIC=com.tt.casera
APNS_USE_SANDBOX=false
APNS_PRODUCTION=true
# Worker schedules (UTC)
TASK_REMINDER_HOUR=14
OVERDUE_REMINDER_HOUR=15
DAILY_DIGEST_HOUR=3
# Storage
STORAGE_UPLOAD_DIR=/app/uploads
STORAGE_BASE_URL=/uploads
STORAGE_MAX_FILE_SIZE=10485760
STORAGE_ALLOWED_TYPES=image/jpeg,image/png,image/gif,image/webp,application/pdf
# Feature flags
FEATURE_PUSH_ENABLED=true
FEATURE_EMAIL_ENABLED=true
FEATURE_WEBHOOKS_ENABLED=true
FEATURE_ONBOARDING_EMAILS_ENABLED=true
FEATURE_PDF_REPORTS_ENABLED=true
FEATURE_WORKER_ENABLED=true
# Optional auth/iap values
APPLE_CLIENT_ID=
APPLE_TEAM_ID=
GOOGLE_CLIENT_ID=
GOOGLE_ANDROID_CLIENT_ID=
GOOGLE_IOS_CLIENT_ID=
APPLE_IAP_KEY_ID=
APPLE_IAP_ISSUER_ID=
APPLE_IAP_BUNDLE_ID=
APPLE_IAP_KEY_PATH=
APPLE_IAP_SANDBOX=false
GOOGLE_IAP_PACKAGE_NAME=
GOOGLE_IAP_SERVICE_ACCOUNT_PATH=

11
deploy/registry.env.example Normal file
View File

@@ -0,0 +1,11 @@
# Container registry used for deploy images.
# For GHCR:
# REGISTRY=ghcr.io
# REGISTRY_NAMESPACE=<github-username-or-org>
# REGISTRY_USERNAME=<github-username>
# REGISTRY_TOKEN=<github-pat-with-read:packages,write:packages>
REGISTRY=ghcr.io
REGISTRY_NAMESPACE=CHANGEME_NAMESPACE
REGISTRY_USERNAME=CHANGEME_USERNAME
REGISTRY_TOKEN=CHANGEME_TOKEN

397
deploy/scripts/deploy_prod.sh Executable file
View File

@@ -0,0 +1,397 @@
#!/usr/bin/env bash
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
DEPLOY_DIR="$(cd "${SCRIPT_DIR}/.." && pwd)"
REPO_DIR="$(cd "${DEPLOY_DIR}/.." && pwd)"
STACK_TEMPLATE="${DEPLOY_DIR}/swarm-stack.prod.yml"
CLUSTER_ENV="${DEPLOY_DIR}/cluster.env"
REGISTRY_ENV="${DEPLOY_DIR}/registry.env"
PROD_ENV="${DEPLOY_DIR}/prod.env"
SECRET_POSTGRES="${DEPLOY_DIR}/secrets/postgres_password.txt"
SECRET_APP_KEY="${DEPLOY_DIR}/secrets/secret_key.txt"
SECRET_EMAIL_PASS="${DEPLOY_DIR}/secrets/email_host_password.txt"
SECRET_FCM_KEY="${DEPLOY_DIR}/secrets/fcm_server_key.txt"
SECRET_APNS_KEY="${DEPLOY_DIR}/secrets/apns_auth_key.p8"
SKIP_BUILD="${SKIP_BUILD:-0}"
SKIP_HEALTHCHECK="${SKIP_HEALTHCHECK:-0}"
log() {
printf '[deploy] %s\n' "$*"
}
warn() {
printf '[deploy][warn] %s\n' "$*" >&2
}
die() {
printf '[deploy][error] %s\n' "$*" >&2
exit 1
}
require_cmd() {
command -v "$1" >/dev/null 2>&1 || die "Missing required command: $1"
}
contains_placeholder() {
local value="$1"
[[ -z "${value}" ]] && return 0
local lowered
lowered="$(printf '%s' "${value}" | tr '[:upper:]' '[:lower:]')"
case "${lowered}" in
*changeme*|*replace_me*|*example.com*|*your-*|*todo*|*fill_me*|*paste_here*)
return 0
;;
*)
return 1
;;
esac
}
ensure_file_from_example() {
local path="$1"
local example="${path}.example"
if [[ -f "${path}" ]]; then
return
fi
if [[ -f "${example}" ]]; then
cp "${example}" "${path}"
die "Created ${path} from template. Fill it in and rerun."
fi
die "Missing required file: ${path}"
}
require_var() {
local name="$1"
local value="${!name:-}"
[[ -n "${value}" ]] || die "Missing required value: ${name}"
if contains_placeholder "${value}"; then
die "Value still uses placeholder text: ${name}=${value}"
fi
}
require_secret_file() {
local path="$1"
local label="$2"
ensure_file_from_example "${path}"
local contents
contents="$(tr -d '\r' < "${path}" | sed 's/[[:space:]]*$//')"
[[ -n "${contents}" ]] || die "Secret file is empty: ${path}"
if contains_placeholder "${contents}"; then
die "Secret file still has placeholder text (${label}): ${path}"
fi
}
print_usage() {
cat <<'EOF'
Usage:
./.deploy_prod
Optional environment flags:
SKIP_BUILD=1 Deploy existing image tags without rebuilding/pushing.
SKIP_HEALTHCHECK=1 Skip final HTTP health check.
DEPLOY_TAG=<tag> Override image tag (default: git short sha).
EOF
}
while (($# > 0)); do
case "$1" in
-h|--help)
print_usage
exit 0
;;
*)
die "Unknown argument: $1"
;;
esac
done
require_cmd docker
require_cmd ssh
require_cmd scp
require_cmd git
require_cmd awk
require_cmd sed
require_cmd grep
require_cmd mktemp
require_cmd date
require_cmd curl
ensure_file_from_example "${CLUSTER_ENV}"
ensure_file_from_example "${REGISTRY_ENV}"
ensure_file_from_example "${PROD_ENV}"
require_secret_file "${SECRET_POSTGRES}" "Postgres password"
require_secret_file "${SECRET_APP_KEY}" "SECRET_KEY"
require_secret_file "${SECRET_EMAIL_PASS}" "SMTP password"
require_secret_file "${SECRET_FCM_KEY}" "FCM server key"
require_secret_file "${SECRET_APNS_KEY}" "APNS private key"
set -a
# shellcheck disable=SC1090
source "${CLUSTER_ENV}"
# shellcheck disable=SC1090
source "${REGISTRY_ENV}"
# shellcheck disable=SC1090
source "${PROD_ENV}"
set +a
DEPLOY_MANAGER_SSH_PORT="${DEPLOY_MANAGER_SSH_PORT:-22}"
DEPLOY_STACK_NAME="${DEPLOY_STACK_NAME:-casera}"
DEPLOY_REMOTE_DIR="${DEPLOY_REMOTE_DIR:-/opt/casera/deploy}"
DEPLOY_WAIT_SECONDS="${DEPLOY_WAIT_SECONDS:-420}"
DEPLOY_TAG="${DEPLOY_TAG:-$(git -C "${REPO_DIR}" rev-parse --short HEAD)}"
PUSH_LATEST_TAG="${PUSH_LATEST_TAG:-true}"
require_var DEPLOY_MANAGER_HOST
require_var DEPLOY_MANAGER_USER
require_var DEPLOY_STACK_NAME
require_var DEPLOY_REMOTE_DIR
require_var REGISTRY
require_var REGISTRY_NAMESPACE
require_var REGISTRY_USERNAME
require_var REGISTRY_TOKEN
require_var ALLOWED_HOSTS
require_var CORS_ALLOWED_ORIGINS
require_var BASE_URL
require_var NEXT_PUBLIC_API_URL
require_var DB_HOST
require_var DB_PORT
require_var POSTGRES_USER
require_var POSTGRES_DB
require_var DB_SSLMODE
require_var REDIS_URL
require_var EMAIL_HOST
require_var EMAIL_PORT
require_var EMAIL_HOST_USER
require_var DEFAULT_FROM_EMAIL
require_var APNS_AUTH_KEY_ID
require_var APNS_TEAM_ID
require_var APNS_TOPIC
if [[ ! "$(tr -d '\r\n' < "${SECRET_APNS_KEY}")" =~ BEGIN[[:space:]]+PRIVATE[[:space:]]+KEY ]]; then
die "APNS key file does not look like a private key: ${SECRET_APNS_KEY}"
fi
app_secret_len="$(tr -d '\r\n' < "${SECRET_APP_KEY}" | wc -c | tr -d ' ')"
if (( app_secret_len < 32 )); then
die "deploy/secrets/secret_key.txt must be at least 32 characters."
fi
REGISTRY_PREFIX="${REGISTRY%/}/${REGISTRY_NAMESPACE#/}"
API_IMAGE="${REGISTRY_PREFIX}/casera-api:${DEPLOY_TAG}"
WORKER_IMAGE="${REGISTRY_PREFIX}/casera-worker:${DEPLOY_TAG}"
ADMIN_IMAGE="${REGISTRY_PREFIX}/casera-admin:${DEPLOY_TAG}"
SSH_KEY_PATH="${DEPLOY_SSH_KEY_PATH:-}"
if [[ -n "${SSH_KEY_PATH}" ]]; then
SSH_KEY_PATH="${SSH_KEY_PATH/#\~/${HOME}}"
fi
SSH_TARGET="${DEPLOY_MANAGER_USER}@${DEPLOY_MANAGER_HOST}"
SSH_OPTS=(-p "${DEPLOY_MANAGER_SSH_PORT}")
SCP_OPTS=(-P "${DEPLOY_MANAGER_SSH_PORT}")
if [[ -n "${SSH_KEY_PATH}" ]]; then
SSH_OPTS+=(-i "${SSH_KEY_PATH}")
SCP_OPTS+=(-i "${SSH_KEY_PATH}")
fi
log "Validating SSH access to ${SSH_TARGET}"
if ! ssh "${SSH_OPTS[@]}" "${SSH_TARGET}" "echo ok" >/dev/null 2>&1; then
die "SSH connection failed to ${SSH_TARGET}"
fi
remote_swarm_state="$(ssh "${SSH_OPTS[@]}" "${SSH_TARGET}" "docker info --format '{{.Swarm.LocalNodeState}} {{.Swarm.ControlAvailable}}'" 2>/dev/null || true)"
if [[ -z "${remote_swarm_state}" ]]; then
die "Could not read Docker Swarm state on manager. Is Docker installed/running?"
fi
if [[ "${remote_swarm_state}" != "active true" ]]; then
die "Remote node must be a Swarm manager. Got: ${remote_swarm_state}"
fi
if [[ "${SKIP_BUILD}" != "1" ]]; then
log "Logging in to ${REGISTRY}"
printf '%s' "${REGISTRY_TOKEN}" | docker login "${REGISTRY}" -u "${REGISTRY_USERNAME}" --password-stdin >/dev/null
log "Building API image ${API_IMAGE}"
docker build --target api -t "${API_IMAGE}" "${REPO_DIR}"
log "Building Worker image ${WORKER_IMAGE}"
docker build --target worker -t "${WORKER_IMAGE}" "${REPO_DIR}"
log "Building Admin image ${ADMIN_IMAGE}"
docker build --target admin -t "${ADMIN_IMAGE}" "${REPO_DIR}"
log "Pushing deploy images"
docker push "${API_IMAGE}"
docker push "${WORKER_IMAGE}"
docker push "${ADMIN_IMAGE}"
if [[ "${PUSH_LATEST_TAG}" == "true" ]]; then
log "Updating :latest tags"
docker tag "${API_IMAGE}" "${REGISTRY_PREFIX}/casera-api:latest"
docker tag "${WORKER_IMAGE}" "${REGISTRY_PREFIX}/casera-worker:latest"
docker tag "${ADMIN_IMAGE}" "${REGISTRY_PREFIX}/casera-admin:latest"
docker push "${REGISTRY_PREFIX}/casera-api:latest"
docker push "${REGISTRY_PREFIX}/casera-worker:latest"
docker push "${REGISTRY_PREFIX}/casera-admin:latest"
fi
else
warn "SKIP_BUILD=1 set. Using prebuilt images for tag: ${DEPLOY_TAG}"
fi
DEPLOY_ID_RAW="${DEPLOY_TAG}-$(date +%Y%m%d%H%M%S)"
DEPLOY_ID="$(printf '%s' "${DEPLOY_ID_RAW}" | tr -c 'a-zA-Z0-9_-' '_')"
POSTGRES_PASSWORD_SECRET="${DEPLOY_STACK_NAME}_postgres_password_${DEPLOY_ID}"
SECRET_KEY_SECRET="${DEPLOY_STACK_NAME}_secret_key_${DEPLOY_ID}"
EMAIL_HOST_PASSWORD_SECRET="${DEPLOY_STACK_NAME}_email_host_password_${DEPLOY_ID}"
FCM_SERVER_KEY_SECRET="${DEPLOY_STACK_NAME}_fcm_server_key_${DEPLOY_ID}"
APNS_AUTH_KEY_SECRET="${DEPLOY_STACK_NAME}_apns_auth_key_${DEPLOY_ID}"
TMP_DIR="$(mktemp -d)"
cleanup() {
rm -rf "${TMP_DIR}"
}
trap cleanup EXIT
cp "${STACK_TEMPLATE}" "${TMP_DIR}/swarm-stack.prod.yml"
cp "${PROD_ENV}" "${TMP_DIR}/prod.env"
cp "${REGISTRY_ENV}" "${TMP_DIR}/registry.env"
mkdir -p "${TMP_DIR}/secrets"
cp "${SECRET_POSTGRES}" "${TMP_DIR}/secrets/postgres_password.txt"
cp "${SECRET_APP_KEY}" "${TMP_DIR}/secrets/secret_key.txt"
cp "${SECRET_EMAIL_PASS}" "${TMP_DIR}/secrets/email_host_password.txt"
cp "${SECRET_FCM_KEY}" "${TMP_DIR}/secrets/fcm_server_key.txt"
cp "${SECRET_APNS_KEY}" "${TMP_DIR}/secrets/apns_auth_key.p8"
cat > "${TMP_DIR}/runtime.env" <<EOF
API_IMAGE=${API_IMAGE}
WORKER_IMAGE=${WORKER_IMAGE}
ADMIN_IMAGE=${ADMIN_IMAGE}
API_REPLICAS=${API_REPLICAS:-3}
WORKER_REPLICAS=${WORKER_REPLICAS:-2}
ADMIN_REPLICAS=${ADMIN_REPLICAS:-1}
API_PORT=${API_PORT:-8000}
ADMIN_PORT=${ADMIN_PORT:-3000}
DOZZLE_PORT=${DOZZLE_PORT:-9999}
POSTGRES_PASSWORD_SECRET=${POSTGRES_PASSWORD_SECRET}
SECRET_KEY_SECRET=${SECRET_KEY_SECRET}
EMAIL_HOST_PASSWORD_SECRET=${EMAIL_HOST_PASSWORD_SECRET}
FCM_SERVER_KEY_SECRET=${FCM_SERVER_KEY_SECRET}
APNS_AUTH_KEY_SECRET=${APNS_AUTH_KEY_SECRET}
EOF
log "Uploading deploy bundle to ${SSH_TARGET}:${DEPLOY_REMOTE_DIR}"
ssh "${SSH_OPTS[@]}" "${SSH_TARGET}" "mkdir -p '${DEPLOY_REMOTE_DIR}/secrets'"
scp "${SCP_OPTS[@]}" "${TMP_DIR}/swarm-stack.prod.yml" "${SSH_TARGET}:${DEPLOY_REMOTE_DIR}/swarm-stack.prod.yml"
scp "${SCP_OPTS[@]}" "${TMP_DIR}/prod.env" "${SSH_TARGET}:${DEPLOY_REMOTE_DIR}/prod.env"
scp "${SCP_OPTS[@]}" "${TMP_DIR}/registry.env" "${SSH_TARGET}:${DEPLOY_REMOTE_DIR}/registry.env"
scp "${SCP_OPTS[@]}" "${TMP_DIR}/runtime.env" "${SSH_TARGET}:${DEPLOY_REMOTE_DIR}/runtime.env"
scp "${SCP_OPTS[@]}" "${TMP_DIR}/secrets/postgres_password.txt" "${SSH_TARGET}:${DEPLOY_REMOTE_DIR}/secrets/postgres_password.txt"
scp "${SCP_OPTS[@]}" "${TMP_DIR}/secrets/secret_key.txt" "${SSH_TARGET}:${DEPLOY_REMOTE_DIR}/secrets/secret_key.txt"
scp "${SCP_OPTS[@]}" "${TMP_DIR}/secrets/email_host_password.txt" "${SSH_TARGET}:${DEPLOY_REMOTE_DIR}/secrets/email_host_password.txt"
scp "${SCP_OPTS[@]}" "${TMP_DIR}/secrets/fcm_server_key.txt" "${SSH_TARGET}:${DEPLOY_REMOTE_DIR}/secrets/fcm_server_key.txt"
scp "${SCP_OPTS[@]}" "${TMP_DIR}/secrets/apns_auth_key.p8" "${SSH_TARGET}:${DEPLOY_REMOTE_DIR}/secrets/apns_auth_key.p8"
log "Creating Docker secrets and deploying stack on manager"
ssh "${SSH_OPTS[@]}" "${SSH_TARGET}" "bash -s -- '${DEPLOY_REMOTE_DIR}' '${DEPLOY_STACK_NAME}'" <<'EOF'
set -euo pipefail
REMOTE_DIR="$1"
STACK_NAME="$2"
set -a
# shellcheck disable=SC1090
source "${REMOTE_DIR}/registry.env"
# shellcheck disable=SC1090
source "${REMOTE_DIR}/prod.env"
# shellcheck disable=SC1090
source "${REMOTE_DIR}/runtime.env"
set +a
create_secret() {
local name="$1"
local src="$2"
if docker secret inspect "${name}" >/dev/null 2>&1; then
echo "[remote] secret exists: ${name}"
else
docker secret create "${name}" "${src}" >/dev/null
echo "[remote] created secret: ${name}"
fi
}
printf '%s' "${REGISTRY_TOKEN}" | docker login "${REGISTRY}" -u "${REGISTRY_USERNAME}" --password-stdin >/dev/null
rm -f "${REMOTE_DIR}/registry.env"
create_secret "${POSTGRES_PASSWORD_SECRET}" "${REMOTE_DIR}/secrets/postgres_password.txt"
create_secret "${SECRET_KEY_SECRET}" "${REMOTE_DIR}/secrets/secret_key.txt"
create_secret "${EMAIL_HOST_PASSWORD_SECRET}" "${REMOTE_DIR}/secrets/email_host_password.txt"
create_secret "${FCM_SERVER_KEY_SECRET}" "${REMOTE_DIR}/secrets/fcm_server_key.txt"
create_secret "${APNS_AUTH_KEY_SECRET}" "${REMOTE_DIR}/secrets/apns_auth_key.p8"
rm -f "${REMOTE_DIR}/secrets/postgres_password.txt"
rm -f "${REMOTE_DIR}/secrets/secret_key.txt"
rm -f "${REMOTE_DIR}/secrets/email_host_password.txt"
rm -f "${REMOTE_DIR}/secrets/fcm_server_key.txt"
rm -f "${REMOTE_DIR}/secrets/apns_auth_key.p8"
set -a
# shellcheck disable=SC1090
source "${REMOTE_DIR}/prod.env"
# shellcheck disable=SC1090
source "${REMOTE_DIR}/runtime.env"
set +a
docker stack deploy --with-registry-auth -c "${REMOTE_DIR}/swarm-stack.prod.yml" "${STACK_NAME}"
EOF
log "Waiting for stack convergence (${DEPLOY_WAIT_SECONDS}s max)"
start_epoch="$(date +%s)"
while true; do
services="$(ssh "${SSH_OPTS[@]}" "${SSH_TARGET}" "docker stack services '${DEPLOY_STACK_NAME}' --format '{{.Name}} {{.Replicas}}'" 2>/dev/null || true)"
if [[ -n "${services}" ]]; then
all_ready=1
while IFS=' ' read -r svc replicas; do
[[ -z "${svc}" ]] && continue
current="${replicas%%/*}"
desired="${replicas##*/}"
if [[ "${desired}" == "0" ]]; then
continue
fi
if [[ "${current}" != "${desired}" ]]; then
all_ready=0
fi
done <<< "${services}"
if [[ "${all_ready}" -eq 1 ]]; then
break
fi
fi
now_epoch="$(date +%s)"
elapsed=$((now_epoch - start_epoch))
if (( elapsed >= DEPLOY_WAIT_SECONDS )); then
die "Timed out waiting for stack to converge. Check: ssh ${SSH_TARGET} docker stack services ${DEPLOY_STACK_NAME}"
fi
sleep 10
done
if [[ "${SKIP_HEALTHCHECK}" != "1" && -n "${DEPLOY_HEALTHCHECK_URL:-}" ]]; then
log "Running health check: ${DEPLOY_HEALTHCHECK_URL}"
curl -fsS --max-time 20 "${DEPLOY_HEALTHCHECK_URL}" >/dev/null
fi
log "Deploy completed successfully."
log "Stack: ${DEPLOY_STACK_NAME}"
log "Images:"
log " ${API_IMAGE}"
log " ${WORKER_IMAGE}"
log " ${ADMIN_IMAGE}"

11
deploy/secrets/README.md Normal file
View File

@@ -0,0 +1,11 @@
# Secrets Directory
Create these files (copy from `.example` files):
- `deploy/secrets/postgres_password.txt`
- `deploy/secrets/secret_key.txt`
- `deploy/secrets/email_host_password.txt`
- `deploy/secrets/fcm_server_key.txt`
- `deploy/secrets/apns_auth_key.p8`
These are consumed by `./.deploy_prod` and converted into Docker Swarm secrets.

3
deploy/secrets/apns_auth_key.p8.example Normal file
View File

@@ -0,0 +1,3 @@
-----BEGIN PRIVATE KEY-----
CHANGEME_APNS_PRIVATE_KEY
-----END PRIVATE KEY-----

1
deploy/secrets/email_host_password.txt.example Normal file
View File

@@ -0,0 +1 @@
CHANGEME_SMTP_PASSWORD

1
deploy/secrets/fcm_server_key.txt.example Normal file
View File

@@ -0,0 +1 @@
CHANGEME_FCM_SERVER_KEY

1
deploy/secrets/postgres_password.txt.example Normal file
View File

@@ -0,0 +1 @@
CHANGEME_DATABASE_PASSWORD

1
deploy/secrets/secret_key.txt.example Normal file
View File

@@ -0,0 +1 @@
CHANGEME_SECRET_KEY_MIN_32_CHARS

67
deploy/shit_deploy_cant_do.md Normal file
View File

@@ -0,0 +1,67 @@
# Shit Deploy Can't Do
This is everything `./.deploy_prod` cannot safely automate for you.
## 1. Create Infrastructure
Step:
Create Hetzner servers, networking, and load balancer.
Reason:
The script only deploys app workloads. It cannot create paid cloud resources without cloud API credentials and IaC wiring.
## 2. Join Nodes To Swarm
Step:
Run `docker swarm init` on the first manager and `docker swarm join` on other nodes.
Reason:
Joining nodes requires one-time bootstrap tokens and host-level control.
## 3. Configure Firewall And Origin Restrictions
Step:
Set firewall rules so only expected ingress paths can reach your nodes.
Reason:
Firewall policies live in provider networking controls, outside this repo.
## 4. Configure DNS / Cloudflare
Step:
Point DNS at LB, enable proxying, set SSL mode, and lock down origin access.
Reason:
DNS and CDN settings are account-level operations in Cloudflare, not deploy-time app actions.
## 5. Configure External Services
Step:
Create and configure Neon, B2, email provider, APNS, and FCM credentials.
Reason:
These credentials are issued in vendor dashboards and must be manually generated/rotated.
## 6. Seed SSH Trust
Step:
Ensure your local machine can SSH to the manager with the key in `deploy/cluster.env`.
Reason:
The script assumes SSH already works; it cannot grant itself SSH access.
## 7. First-Time Smoke Testing Beyond `/api/health/`
Step:
Manually test login, push, background jobs, and admin panel flows after first deploy.
Reason:
Automated health checks prove container readiness, not end-to-end business behavior.
## 8. Safe Secret Garbage Collection
Step:
Periodically remove old versioned Docker secrets that are no longer referenced.
Reason:
This deploy script creates versioned secrets for safe rollouts and does not auto-delete old ones to avoid breaking running services.

288
deploy/swarm-stack.prod.yml Normal file
View File

@@ -0,0 +1,288 @@
version: "3.8"
services:
redis:
image: redis:7-alpine
command: redis-server --appendonly yes --appendfsync everysec
volumes:
- redis_data:/data
healthcheck:
test: ["CMD", "redis-cli", "ping"]
interval: 10s
timeout: 5s
retries: 5
deploy:
replicas: 1
restart_policy:
condition: any
delay: 5s
placement:
max_replicas_per_node: 1
networks:
- casera-network
api:
image: ${API_IMAGE}
ports:
- target: 8000
published: ${API_PORT}
protocol: tcp
mode: ingress
environment:
PORT: "8000"
DEBUG: "${DEBUG}"
ALLOWED_HOSTS: "${ALLOWED_HOSTS}"
CORS_ALLOWED_ORIGINS: "${CORS_ALLOWED_ORIGINS}"
TIMEZONE: "${TIMEZONE}"
BASE_URL: "${BASE_URL}"
ADMIN_PANEL_URL: "${ADMIN_PANEL_URL}"
DB_HOST: "${DB_HOST}"
DB_PORT: "${DB_PORT}"
POSTGRES_USER: "${POSTGRES_USER}"
POSTGRES_DB: "${POSTGRES_DB}"
DB_SSLMODE: "${DB_SSLMODE}"
DB_MAX_OPEN_CONNS: "${DB_MAX_OPEN_CONNS}"
DB_MAX_IDLE_CONNS: "${DB_MAX_IDLE_CONNS}"
DB_MAX_LIFETIME: "${DB_MAX_LIFETIME}"
REDIS_URL: "${REDIS_URL}"
REDIS_DB: "${REDIS_DB}"
EMAIL_HOST: "${EMAIL_HOST}"
EMAIL_PORT: "${EMAIL_PORT}"
EMAIL_HOST_USER: "${EMAIL_HOST_USER}"
DEFAULT_FROM_EMAIL: "${DEFAULT_FROM_EMAIL}"
EMAIL_USE_TLS: "${EMAIL_USE_TLS}"
APNS_AUTH_KEY_PATH: "/run/secrets/apns_auth_key"
APNS_AUTH_KEY_ID: "${APNS_AUTH_KEY_ID}"
APNS_TEAM_ID: "${APNS_TEAM_ID}"
APNS_TOPIC: "${APNS_TOPIC}"
APNS_USE_SANDBOX: "${APNS_USE_SANDBOX}"
APNS_PRODUCTION: "${APNS_PRODUCTION}"
STORAGE_UPLOAD_DIR: "${STORAGE_UPLOAD_DIR}"
STORAGE_BASE_URL: "${STORAGE_BASE_URL}"
STORAGE_MAX_FILE_SIZE: "${STORAGE_MAX_FILE_SIZE}"
STORAGE_ALLOWED_TYPES: "${STORAGE_ALLOWED_TYPES}"
FEATURE_PUSH_ENABLED: "${FEATURE_PUSH_ENABLED}"
FEATURE_EMAIL_ENABLED: "${FEATURE_EMAIL_ENABLED}"
FEATURE_WEBHOOKS_ENABLED: "${FEATURE_WEBHOOKS_ENABLED}"
FEATURE_ONBOARDING_EMAILS_ENABLED: "${FEATURE_ONBOARDING_EMAILS_ENABLED}"
FEATURE_PDF_REPORTS_ENABLED: "${FEATURE_PDF_REPORTS_ENABLED}"
FEATURE_WORKER_ENABLED: "${FEATURE_WORKER_ENABLED}"
APPLE_CLIENT_ID: "${APPLE_CLIENT_ID}"
APPLE_TEAM_ID: "${APPLE_TEAM_ID}"
GOOGLE_CLIENT_ID: "${GOOGLE_CLIENT_ID}"
GOOGLE_ANDROID_CLIENT_ID: "${GOOGLE_ANDROID_CLIENT_ID}"
GOOGLE_IOS_CLIENT_ID: "${GOOGLE_IOS_CLIENT_ID}"
APPLE_IAP_KEY_PATH: "${APPLE_IAP_KEY_PATH}"
APPLE_IAP_KEY_ID: "${APPLE_IAP_KEY_ID}"
APPLE_IAP_ISSUER_ID: "${APPLE_IAP_ISSUER_ID}"
APPLE_IAP_BUNDLE_ID: "${APPLE_IAP_BUNDLE_ID}"
APPLE_IAP_SANDBOX: "${APPLE_IAP_SANDBOX}"
GOOGLE_IAP_SERVICE_ACCOUNT_PATH: "${GOOGLE_IAP_SERVICE_ACCOUNT_PATH}"
GOOGLE_IAP_PACKAGE_NAME: "${GOOGLE_IAP_PACKAGE_NAME}"
command:
- /bin/sh
- -lc
- |
set -eu
export POSTGRES_PASSWORD="$$(cat /run/secrets/postgres_password)"
export SECRET_KEY="$$(cat /run/secrets/secret_key)"
export EMAIL_HOST_PASSWORD="$$(cat /run/secrets/email_host_password)"
export FCM_SERVER_KEY="$$(cat /run/secrets/fcm_server_key)"
exec /app/api
secrets:
- source: ${POSTGRES_PASSWORD_SECRET}
target: postgres_password
- source: ${SECRET_KEY_SECRET}
target: secret_key
- source: ${EMAIL_HOST_PASSWORD_SECRET}
target: email_host_password
- source: ${FCM_SERVER_KEY_SECRET}
target: fcm_server_key
- source: ${APNS_AUTH_KEY_SECRET}
target: apns_auth_key
volumes:
- uploads:/app/uploads
healthcheck:
test: ["CMD", "curl", "-f", "http://127.0.0.1:8000/api/health/"]
interval: 30s
timeout: 10s
start_period: 15s
retries: 3
deploy:
replicas: ${API_REPLICAS}
restart_policy:
condition: any
delay: 5s
update_config:
parallelism: 1
delay: 10s
order: start-first
rollback_config:
parallelism: 1
delay: 5s
order: stop-first
networks:
- casera-network
admin:
image: ${ADMIN_IMAGE}
ports:
- target: 3000
published: ${ADMIN_PORT}
protocol: tcp
mode: ingress
environment:
PORT: "3000"
HOSTNAME: "0.0.0.0"
NEXT_PUBLIC_API_URL: "${NEXT_PUBLIC_API_URL}"
healthcheck:
test: ["CMD", "wget", "--no-verbose", "--tries=1", "--spider", "http://127.0.0.1:3000/admin/"]
interval: 30s
timeout: 10s
retries: 3
deploy:
replicas: ${ADMIN_REPLICAS}
restart_policy:
condition: any
delay: 5s
update_config:
parallelism: 1
delay: 10s
order: start-first
rollback_config:
parallelism: 1
delay: 5s
order: stop-first
networks:
- casera-network
worker:
image: ${WORKER_IMAGE}
environment:
DB_HOST: "${DB_HOST}"
DB_PORT: "${DB_PORT}"
POSTGRES_USER: "${POSTGRES_USER}"
POSTGRES_DB: "${POSTGRES_DB}"
DB_SSLMODE: "${DB_SSLMODE}"
DB_MAX_OPEN_CONNS: "${DB_MAX_OPEN_CONNS}"
DB_MAX_IDLE_CONNS: "${DB_MAX_IDLE_CONNS}"
DB_MAX_LIFETIME: "${DB_MAX_LIFETIME}"
REDIS_URL: "${REDIS_URL}"
REDIS_DB: "${REDIS_DB}"
EMAIL_HOST: "${EMAIL_HOST}"
EMAIL_PORT: "${EMAIL_PORT}"
EMAIL_HOST_USER: "${EMAIL_HOST_USER}"
DEFAULT_FROM_EMAIL: "${DEFAULT_FROM_EMAIL}"
EMAIL_USE_TLS: "${EMAIL_USE_TLS}"
APNS_AUTH_KEY_PATH: "/run/secrets/apns_auth_key"
APNS_AUTH_KEY_ID: "${APNS_AUTH_KEY_ID}"
APNS_TEAM_ID: "${APNS_TEAM_ID}"
APNS_TOPIC: "${APNS_TOPIC}"
APNS_USE_SANDBOX: "${APNS_USE_SANDBOX}"
APNS_PRODUCTION: "${APNS_PRODUCTION}"
TASK_REMINDER_HOUR: "${TASK_REMINDER_HOUR}"
OVERDUE_REMINDER_HOUR: "${OVERDUE_REMINDER_HOUR}"
DAILY_DIGEST_HOUR: "${DAILY_DIGEST_HOUR}"
FEATURE_PUSH_ENABLED: "${FEATURE_PUSH_ENABLED}"
FEATURE_EMAIL_ENABLED: "${FEATURE_EMAIL_ENABLED}"
FEATURE_WEBHOOKS_ENABLED: "${FEATURE_WEBHOOKS_ENABLED}"
FEATURE_ONBOARDING_EMAILS_ENABLED: "${FEATURE_ONBOARDING_EMAILS_ENABLED}"
FEATURE_PDF_REPORTS_ENABLED: "${FEATURE_PDF_REPORTS_ENABLED}"
FEATURE_WORKER_ENABLED: "${FEATURE_WORKER_ENABLED}"
command:
- /bin/sh
- -lc
- |
set -eu
export POSTGRES_PASSWORD="$$(cat /run/secrets/postgres_password)"
export SECRET_KEY="$$(cat /run/secrets/secret_key)"
export EMAIL_HOST_PASSWORD="$$(cat /run/secrets/email_host_password)"
export FCM_SERVER_KEY="$$(cat /run/secrets/fcm_server_key)"
exec /app/worker
secrets:
- source: ${POSTGRES_PASSWORD_SECRET}
target: postgres_password
- source: ${SECRET_KEY_SECRET}
target: secret_key
- source: ${EMAIL_HOST_PASSWORD_SECRET}
target: email_host_password
- source: ${FCM_SERVER_KEY_SECRET}
target: fcm_server_key
- source: ${APNS_AUTH_KEY_SECRET}
target: apns_auth_key
deploy:
replicas: ${WORKER_REPLICAS}
restart_policy:
condition: any
delay: 5s
update_config:
parallelism: 1
delay: 10s
order: start-first
rollback_config:
parallelism: 1
delay: 5s
order: stop-first
networks:
- casera-network
dozzle:
image: amir20/dozzle:latest
ports:
- target: 8080
published: ${DOZZLE_PORT}
protocol: tcp
mode: ingress
environment:
DOZZLE_NO_ANALYTICS: "true"
volumes:
- /var/run/docker.sock:/var/run/docker.sock:ro
deploy:
replicas: 1
restart_policy:
condition: any
delay: 5s
placement:
constraints:
- node.role == manager
networks:
- casera-network
volumes:
redis_data:
uploads:
networks:
casera-network:
driver: overlay
driver_opts:
encrypted: "true"
secrets:
postgres_password:
external: true
name: ${POSTGRES_PASSWORD_SECRET}
secret_key:
external: true
name: ${SECRET_KEY_SECRET}
email_host_password:
external: true
name: ${EMAIL_HOST_PASSWORD_SECRET}
fcm_server_key:
external: true
name: ${FCM_SERVER_KEY_SECRET}
apns_auth_key:
external: true
name: ${APNS_AUTH_KEY_SECRET}