admin/honeyDueAPI
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
0798ae8d74435b45a1c04060f1a83d0c7d38fbb8
honeyDueAPI/README.md
T
Trey t c9ac273dbd
Backend CI / Test (push) Has been cancelled
Details
Backend CI / Contract Tests (push) Has been cancelled
Details
Backend CI / Build (push) Has been cancelled
Details
Backend CI / Lint (push) Has been cancelled
Details
Backend CI / Secret Scanning (push) Has been cancelled
Details
docs: capture latency optimizations + new caching invariants 
Shipping commit 88fb175 changed the trace shape and added a new caching
layer with required invalidation rules. Updating the operator-facing
docs so they match the running system.

ch08 (database):
- DB_HOST is the -pooler Neon endpoint, not direct compute
- Connection pool: MaxIdleConns 20 (was 10), MaxLifetime 30m (was 10m),
  MaxIdleTime 0 (never close idle)
- New \"Pool warm-up at boot\" section documenting the 20-parallel-ping
  warm-up in database.Connect
- Replaced the \"Neon regions\" section: explicit RTT numbers, the
  optimization stack that minimizes round-trips, when this still matters

ch15 (observability):
- Replaced the 2,473ms/5-span sample trace with the new 229ms/2-span
  post-optimization trace; kept the old one underneath for diff context

ch16 (failure modes):
- Added: stale residence-IDs cache (data freshness bug + recovery)
- Added: Redis at maxmemory limit (verify allkeys-lru policy)
- Added: Neon pooler unreachable but direct endpoint up — emergency
  switchover procedure

ch17 (runbook):
- §23 Invalidate residence-IDs cache for a user (DEL key + grep for
  missing invalidation in new code)
- §24 Verify DB pool warm-up is working (log pattern + impact test)
- §25 Switch DB host between pooler and direct endpoints

observability-plan.md status flipped from \"plan only\" to shipped
with the latency-cut summary.

README links to the new ch08 latency section.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-25 17:36:36 -05:00

401 lines
17 KiB
Markdown
Raw Blame History

# honeyDue API
Go REST API for the honeyDue property management platform. Powers iOS and Android mobile apps built with Kotlin Multiplatform.
## Tech Stack
- **Language**: Go 1.25
- **HTTP Framework**: [Echo v4](https://github.com/labstack/echo)
- **ORM**: [GORM](https://gorm.io/) with PostgreSQL
- **Background Jobs**: [Asynq](https://github.com/hibiken/asynq) (Redis-backed)
- **Push Notifications**: APNs ([apns2](https://github.com/sideshow/apns2)) + FCM HTTP API
- **Caching**: Redis
- **Object Storage**: Backblaze B2 via [minio-go/v7](https://github.com/minio/minio-go) (S3-compatible)
- **Email**: SMTP via [wneessen/go-mail](https://github.com/wneessen/go-mail) (Fastmail in prod)
- **Logging**: [zerolog](https://github.com/rs/zerolog)
- **Configuration**: [Viper](https://github.com/spf13/viper)
- **Admin Panel**: Next.js 16 (separate build target; auto-seeded via `ADMIN_EMAIL`/`ADMIN_PASSWORD`)
- **Production Orchestrator**: K3s v1.34.6 HA on Hetzner Cloud, fronted by Cloudflare. See [`docs/deployment/`](./docs/deployment/README.md) for the full book.
## Prerequisites
- **Go 1.25+** — [install](https://go.dev/dl/)
- **PostgreSQL 16+** — via Docker (recommended) or [native install](https://www.postgresql.org/download/)
- **Redis 7+** — via Docker (recommended) or [native install](https://redis.io/docs/getting-started/)
- **Docker & Docker Compose** — [install](https://docs.docker.com/get-docker/) (recommended for local development)
- **Make** — pre-installed on macOS; `apt install make` on Linux
## Getting Started on a New Machine
### Option A: Docker (Recommended)
This starts PostgreSQL, Redis, the API server, background worker, and admin panel in containers using the self-contained dev compose file.
```bash
# 1. Clone the repo
git clone <repo-url>
cd honeyDueAPI-go
# 2. Create your environment file
cp .env.example .env
# Edit .env — at minimum set:
# - SECRET_KEY to a random 32+ char string
# - ADMIN_EMAIL + ADMIN_PASSWORD if you want the Next.js admin panel seeded
# 3. Build and start all services
make docker-dev
# 4. Verify the API is running
# Lookups, admin user, and task templates auto-seed on first boot
# via internal/database/migration_seed_initial_data.go. The data_migrations
# table tracks what's been applied so re-runs are safe.
curl http://localhost:8000/api/health/
# 5. (Optional) Seed dev test data — creates test users, residences, tasks
docker exec -i honeydue-db psql -U honeydue -d honeydue < seeds/002_test_data.sql
```
The API is now available at `http://localhost:8000`.
### Option B: Run Locally (No Docker for the API)
Use Docker for PostgreSQL and Redis, but run the Go server natively for faster iteration.
```bash
# 1. Clone and enter the repo
git clone <repo-url>
cd honeyDueAPI-go
# 2. Install Go dependencies
make deps
# 3. Start PostgreSQL and Redis via Docker
docker compose -f docker-compose.dev.yml up -d db redis
# 4. Create your environment file
cp .env.example .env
# Edit .env:
# - Set SECRET_KEY to a random 32+ char string
# - Set DB_HOST=localhost
# - Set DB_PORT=5433 (docker-compose maps 5433 externally)
# - Set REDIS_URL=redis://localhost:6379/0
# - Set ADMIN_EMAIL + ADMIN_PASSWORD if you want the Next.js admin panel seeded
# 5. Run the API server
# First boot auto-seeds lookups, admin user, and task templates
make run
# 6. (Optional, dev only) Seed dev test data in a separate terminal
psql -h localhost -p 5433 -U honeydue -d honeydue < seeds/002_test_data.sql
# 7. (Optional) Run the background worker in a separate terminal
make run-worker
```
### Option C: Fully Native (No Docker)
Install PostgreSQL and Redis natively on your machine.
```bash
# macOS with Homebrew
brew install postgresql@16 redis
# Start services
brew services start postgresql@16
brew services start redis
# Create the database
createdb honeydue
# Then follow Option B steps 2-7, using:
# DB_HOST=localhost, DB_PORT=5432, POSTGRES_USER=<your-user>, POSTGRES_PASSWORD=<your-password>
```
## Environment Variables
Copy `.env.example` to `.env` and configure. The table below covers what's
needed for local dev. For the complete production env var reference
(~60 vars), see
[`docs/deployment/10-secrets-config.md`](./docs/deployment/10-secrets-config.md).
### Core (required)
| Variable | Description | Default |
|----------|-------------|---------|
| `SECRET_KEY` | Signing secret (32+ chars) | — |
| `POSTGRES_DB` | Database name | `honeydue` |
| `POSTGRES_USER` | Database user | `postgres` |
| `POSTGRES_PASSWORD` | Database password | — |
| `DB_HOST` | Database host | `localhost` |
| `DB_PORT` | Database port | `5432` |
| `DB_SSLMODE` | TLS mode for Postgres (`disable` for local, `require` for Neon) | `disable` |
| `REDIS_URL` | Redis connection URL | `redis://localhost:6379/0` |
### Server (optional)
| Variable | Description | Default |
|----------|-------------|---------|
| `PORT` | Server port | `8000` |
| `DEBUG` | Enable debug logging | `true` |
| `BASE_URL` | Canonical base URL used in generated links | `http://localhost:8000` |
| `ALLOWED_HOSTS` | Comma-separated Host header allowlist | `*` |
| `CORS_ALLOWED_ORIGINS` | Comma-separated CORS origins | `*` |
| `ADMIN_PANEL_URL` | URL of the Next.js admin panel | `http://localhost:3000` |
| `NEXT_PUBLIC_API_URL` | API URL baked into the admin bundle | `http://localhost:8000` |
### Admin panel seed (new — see [commit 4ec4bbb](https://gitea.treytartt.com/admin/honeyDueAPI/commit/4ec4bbbfe88b6060e17c046e0fe2b321f7110bf1))
| Variable | Description |
|----------|-------------|
| `ADMIN_EMAIL` | Email for the initial `admin_users` row. Created on first boot. |
| `ADMIN_PASSWORD` | Password for that admin (bcrypt-hashed at seed time). Change in the admin UI after first login, then blank this out. |
### Email (for transactional mail)
| Variable | Description | Default |
|----------|-------------|---------|
| `EMAIL_HOST` | SMTP server | `smtp.gmail.com` |
| `EMAIL_PORT` | SMTP port | `587` |
| `EMAIL_HOST_USER` | SMTP username | — |
| `EMAIL_HOST_PASSWORD` | SMTP app password | — |
| `DEFAULT_FROM_EMAIL` | From address | — |
| `EMAIL_USE_TLS` | Opportunistic STARTTLS | `true` |
### Push notifications (optional; gated behind `FEATURE_PUSH_ENABLED`)
| Variable | Description |
|----------|-------------|
| `APNS_AUTH_KEY_PATH` | Path to APNs `.p8` key file |
| `APNS_AUTH_KEY_ID` | APNs key ID |
| `APNS_TEAM_ID` | Apple Team ID |
| `APNS_TOPIC` | Bundle ID |
| `FCM_SERVER_KEY` | Firebase server key |
### Object storage (Backblaze B2 in prod; local volume in dev)
| Variable | Description |
|----------|-------------|
| `B2_ENDPOINT` | S3-compatible endpoint (e.g. `s3.us-east-005.backblazeb2.com`) |
| `B2_KEY_ID` | B2 app key ID |
| `B2_APP_KEY` | B2 app key secret |
| `B2_BUCKET_NAME` | Bucket name |
| `B2_REGION` | Region code (e.g. `us-east-005`) |
| `B2_USE_SSL` | Use HTTPS | `true` |
Leave all four `B2_*` empty in dev to fall back to a local `/app/uploads` volume.
### Worker schedules (UTC hours)
| Variable | Description | Default |
|----------|-------------|---------|
| `TASK_REMINDER_HOUR` | When to send task reminders | `14` |
| `OVERDUE_REMINDER_HOUR` | When to send overdue reminders | `15` |
| `DAILY_DIGEST_HOUR` | When to send daily digest | `3` |
### Feature flags (default `true` in dev, configurable in prod)
| Variable | Description |
|----------|-------------|
| `FEATURE_PUSH_ENABLED` | APNs / FCM push notifications |
| `FEATURE_EMAIL_ENABLED` | Transactional email |
| `FEATURE_WEBHOOKS_ENABLED` | Outbound webhooks |
| `FEATURE_ONBOARDING_EMAILS_ENABLED` | Onboarding drip emails |
| `FEATURE_PDF_REPORTS_ENABLED` | PDF task report generation |
| `FEATURE_WORKER_ENABLED` | Run the Asynq worker |
### Apple / Google auth + IAP (optional)
`APPLE_CLIENT_ID`, `APPLE_TEAM_ID`, `GOOGLE_CLIENT_ID`, `GOOGLE_ANDROID_CLIENT_ID`,
`GOOGLE_IOS_CLIENT_ID`, plus `APPLE_IAP_*` and `GOOGLE_IAP_*` for receipt
validation. See Chapter 10 of the deployment book for details.
## Project Structure
```
honeyDueAPI-go/
├── cmd/
│ ├── api/main.go # API server entry point
│ └── worker/main.go # Background worker entry point
├── internal/
│ ├── config/ # Viper configuration
│ ├── database/ # PostgreSQL connection setup
│ ├── models/ # GORM models (map to PostgreSQL tables)
│ ├── repositories/ # Data access layer
│ ├── services/ # Business logic
│ ├── handlers/ # HTTP handlers (Echo)
│ ├── middleware/ # Auth, timezone, logging middleware
│ ├── router/ # Route registration
│ ├── dto/ # Request/Response DTOs
│ │ ├── requests/ # Incoming request structs
│ │ └── responses/ # Outgoing response structs
│ ├── task/ # Centralized task logic
│ │ ├── predicates/ # IsCompleted, IsOverdue, etc.
│ │ ├── scopes/ # GORM query scopes
│ │ └── categorization/ # Kanban column assignment
│ ├── apperrors/ # Structured error types
│ ├── push/ # APNs + FCM push notification clients
│ ├── worker/ # Asynq background jobs
│ ├── i18n/ # Internationalization (en, es, fr)
│ ├── validator/ # Input validation
│ ├── monitoring/ # Health checks
│ └── integration/ # Integration + contract tests
├── admin/ # Next.js admin panel
├── migrations/ # SQL migration files
├── seeds/ # Seed data (lookups, test users, admin, templates)
├── templates/emails/ # Email templates
├── docs/ # API docs, OpenAPI spec
│ └── deployment/ # Full production deployment book (26 chapters)
├── deploy-k3s/ # Production Kubernetes manifests + migration notes
├── deploy/ # Legacy Docker Swarm config (to be removed)
├── docker-compose.yml # Legacy (Swarm-era); prod now runs on K3s
├── docker-compose.dev.yml # Self-contained local dev config
├── Dockerfile # Multi-stage build (api, worker, admin)
├── Makefile # Build, test, Docker commands
└── .env.example # Environment variable template
```
## Development
### Common Commands
```bash
make run # Run API server
make run-worker # Run background worker
make deps # Install/tidy Go dependencies
make build # Build API binary
make build-all # Build API + worker binaries
make fmt # Format code
make vet # Vet code
make lint # Run golangci-lint
make clean # Remove build artifacts
```
### Testing
```bash
make test # Run all tests with race detection + coverage
make test-coverage # Run tests and generate HTML coverage report
make contract-test # Run route + KMP contract validation tests
# Run specific packages
go test ./internal/handlers -v
go test -run TestTaskHandler_CreateTask ./internal/handlers
```
### Docker (local dev)
- **`docker-compose.dev.yml`** — self-contained local dev (build from source, `container_name`, `depends_on`, dev defaults)
- **`docker-compose.yml`** — legacy (Swarm-era). Prod now runs on K3s (see [deployment book](./docs/deployment/README.md)). File retained temporarily.
```bash
# Dev
make docker-dev # Build and start all dev containers (foreground)
make docker-up # Start dev containers (detached)
make docker-down # Stop dev containers
make docker-logs # Tail dev container logs
make docker-restart # Restart dev containers
# Build production amd64 images (push to registry handled separately)
make docker-build-prod
```
For the full production build + deploy workflow, see
[`docs/deployment/14-deployment-process.md`](./docs/deployment/14-deployment-process.md).
### Database Migrations
```bash
make migrate-up # Apply pending migrations
make migrate-down # Roll back last migration
make migrate-create name=add_column # Create a new migration pair
```
## Seed Data
Seed files in `seeds/`:
| File | Purpose | Loaded by |
|------|---------|-----------|
| `001_lookups.sql` | Residence types, task categories, priorities, frequencies, contractor specialties | **Auto-seeded on first API boot** |
| `003_admin_user.sql` | Placeholder admin panel user (also seeded via `ADMIN_EMAIL` / `ADMIN_PASSWORD` env vars) | **Auto-seeded on first API boot** |
| `003_task_templates.sql` | Pre-built task templates for onboarding | **Auto-seeded on first API boot** |
| `002_test_data.sql` | Dev-only: test users, residences, tasks, contractors, documents, notifications | **Manual** (`psql` or `docker exec`) |
Auto-seeded files run once per database — tracked in the `data_migrations`
table by `internal/database/migration_seed_initial_data.go`. Each INSERT
uses `ON CONFLICT DO UPDATE`, so re-runs are safe.
### Test Users (from `002_test_data.sql`; dev only)
All test users have password: `password123`
| Username | Email | Tier | Notes |
|----------|-------|------|-------|
| admin | admin@example.com | Pro | Admin user |
| john | john@example.com | Pro | Owns 2 residences |
| jane | jane@example.com | Free | Owns 1 residence, shared access |
| bob | bob@example.com | Free | Owns 1 residence |
## API Documentation
- **OpenAPI spec**: `docs/openapi.yaml` (81 paths, 104 operations, 81 schemas)
- **Health check**: `GET /api/health/`
- **Auth**: `POST /api/auth/login/`, `POST /api/auth/register/`
- **Static data**: `GET /api/static_data/` (ETag-cached lookups)
All protected endpoints require an `Authorization: Token <token>` header.
## Production Deployment
Production runs on a **3-node K3s HA cluster** on Hetzner Cloud, fronted
by Cloudflare, with Neon Postgres, Backblaze B2, and a self-hosted Gitea
container registry. Live observability (VictoriaMetrics + Jaeger +
Grafana) runs on a separate Linode VPS at
[`grafana.88oakapps.com`](https://grafana.88oakapps.com) and is fed by a
`vmagent` sidecar in-cluster. See the full deployment book for every
detail:
**→ [docs/deployment/](./docs/deployment/README.md) — The Deployment Book**
26 chapters and ~42,000 words covering:
- **Part I — The System**: overview, Hetzner infrastructure, why K3s
(and not Swarm, full Kubernetes, or Nomad)
- **Part II — Networking**: Flannel VXLAN, CoreDNS, kube-proxy, every
UFW rule on every node, Cloudflare DNS setup
- **Part III — Security**: RBAC, Pod Security, secrets, TLS chain
- **Part IV — Workloads**: api, admin, worker, redis per-service deep
dives; Neon Postgres config; Backblaze B2 storage; Gitea registry
- **Part V — Operation**: end-to-end data flow, deploy process,
observability, failure modes, operator runbook
- **Part VI — Context**: cost breakdown, postmortem of the bugs from
the Swarm→K3s migration, roadmap
Quick links:
- **Runbook** — [docs/deployment/17-runbook.md](./docs/deployment/17-runbook.md) — 22 common ops procedures
- **kubectl cheat sheet** — [docs/deployment/appendices/b-commands.md](./docs/deployment/appendices/b-commands.md)
- **Deploy process** — [docs/deployment/14-deployment-process.md](./docs/deployment/14-deployment-process.md) — `bash deploy-k3s/scripts/03-deploy.sh` builds → pushes → rolls out
- **Observability** — [docs/deployment/15-observability.md](./docs/deployment/15-observability.md) — VictoriaMetrics + Jaeger + Grafana on `obs.88oakapps.com`
- **Observability plan** — [docs/observability-plan.md](./docs/observability-plan.md) — design doc and rollout phases
- **Database / pool tuning** — [docs/deployment/08-database.md](./docs/deployment/08-database.md) — Neon pooler endpoint, GORM pool, warm-up, RTT budget
- **Failure modes** — [docs/deployment/16-failure-modes.md](./docs/deployment/16-failure-modes.md) — what happens when X dies
- **Swarm postmortem** — [docs/deployment/19-postmortem-swarm.md](./docs/deployment/19-postmortem-swarm.md) — why we migrated
Operational state lives under:
- `deploy-k3s/manifests/` — Kubernetes manifests (apply with `kubectl`)
- `deploy-k3s/MIGRATION_NOTES.md` — notes from the Swarm → K3s migration
- `deploy/` — legacy Swarm config (retained temporarily; to be removed)
## Related Projects
- **Deployment Book**: [`docs/deployment/`](./docs/deployment/README.md) — full production operations reference
- **Mobile App (KMM)**: `../HoneyDueKMM` — Kotlin Multiplatform iOS/Android client
- **Task Logic Docs**: `docs/TASK_LOGIC_ARCHITECTURE.md` — required reading before task-related work
- **Push Notification Docs**: `docs/PUSH_NOTIFICATIONS.md`
## License
Proprietary — honeyDue
Reference in New Issue View Git Blame Copy Permalink