# 11 — Container Registry (Gitea)

## Summary

We host our own container registry on Gitea at `gitea.treytartt.com`.
Every image push and pull goes there, not Docker Hub or GHCR. The Gitea
instance runs outside this k3s cluster (on its own VPS) and is available
at `https://gitea.treytartt.com` with public HTTPS. Image pulls are
authenticated via a Personal Access Token stored as a Kubernetes
`dockerconfigjson` Secret.

## Why Gitea

### Decision matrix

| Option | Cost | Auth model | Pros | Cons |
|---|---|---|---|---|
| **Gitea built-in registry** | $0 (already running Gitea) | Gitea PAT | Self-hosted, integrated with code | Another service to maintain |
| GHCR (GitHub Container Registry) | Free for public, $0 for private with paid plan | GitHub PAT | Popular, reliable | Uses GitHub; vendor dependency |
| Docker Hub | Free tier limited; paid $5-7/mo | Docker Hub account | Ubiquitous | Rate limits on anonymous pulls |
| AWS ECR | ~$1/mo for small use | IAM | Integrates with AWS workloads | AWS account required |
| Harbor (self-hosted) | $0 | Many options | Best enterprise features | Heavy to operate |

Gitea won primarily because **the operator was already running Gitea for
code hosting**. Container registry is built into Gitea 1.17+ as a free
feature. One fewer service to set up.

Side benefits:
- Code and images live together (one backup policy, one access model)
- PATs are scoped and rotatable via the same UI
- No external vendor to worry about for this critical piece of the
  deploy pipeline

Rejected alternatives:
- **Docker Hub** — rate limits on unauthenticated pulls would bite us if
  nodes pull the same image repeatedly during rolling updates
- **GHCR** — fine but adds GitHub dependency we don't otherwise have
- **Harbor** — massive overkill; we're not a 100-team enterprise

## Layout

Images live under the authenticated user's namespace:

```
gitea.treytartt.com/admin/honeydue-api:237c6b8
gitea.treytartt.com/admin/honeydue-worker:237c6b8
gitea.treytartt.com/admin/honeydue-admin:237c6b8
```

`admin` is the Gitea user that owns the images. Images are **private**
by default.

### Image tagging strategy

Tags are git short SHAs (e.g., `237c6b8`). Not `:latest`. Not semantic
version.

Rationale:
- `:latest` is ambiguous — which build? Rolling updates should roll a
  *specific* tag so rollbacks are deterministic.
- `:v1.2.3` works for released libraries but our app rolls forward
  continuously; versioning per deploy is unnecessary overhead.
- Git SHAs are unique, immutable, and tie each image to the exact
  commit that built it.

`PUSH_LATEST_TAG=false` is set in `deploy/cluster.env`. When we rebuild
and push, only the SHA tag gets pushed. The `latest` tag is never
created by our deploy pipeline.

## Authentication

### Creating the PAT

At <https://gitea.treytartt.com/-/user/settings/applications>, we created
a token with scopes:

- `read:package`
- `write:package`

No other scopes. This token can only interact with package registry; it
can't read repo contents, create issues, or touch account settings.

### PAT on the operator workstation

Stored in `deploy/registry.env`:

```
REGISTRY=gitea.treytartt.com
REGISTRY_NAMESPACE=admin
REGISTRY_USERNAME=admin
REGISTRY_TOKEN=<pat>
```

This file is `.gitignore`d in `deploy/.gitignore`. If it ever gets
committed accidentally, rotate the PAT immediately.

### PAT in the cluster

Stored as the `gitea-credentials` Secret (type `dockerconfigjson`) in
the `honeydue` namespace. See Chapter 10.

Kubelet reads this Secret when a pod needs to pull from the Gitea
registry.

## The build pipeline

### Dockerfile multi-stage

`honeyDueAPI-go/Dockerfile` has three target stages:

- `api` — compiled Go binary + static assets for the HTTP API
- `worker` — compiled Go binary for the background worker
- `admin` — Next.js standalone build of the admin panel

A single Dockerfile keeps build-cache sharing efficient (the Go builder
stage produces binaries for both api and worker; admin reuses its own
Node builder stage).

### Multi-arch cross-compilation

The operator workstation is **arm64** (Apple Silicon). The Hetzner nodes
are **x86_64**. A naive `docker build` on arm64 produces arm64 images
that won't run on the nodes (`exec format error`).

The deploy pipeline uses `docker buildx`:

```bash
docker buildx build \
  --platform linux/amd64 \
  --target api \
  -t gitea.treytartt.com/admin/honeydue-api:$SHA \
  --push \
  /Users/treyt/Desktop/code/honeyDue/honeyDueAPI-go
```

- **`--platform linux/amd64`** — cross-compile to x86_64
- **`--target api`** — which Dockerfile stage to build
- **`--push`** — push directly to the registry (skip local image cache)

The Go stages use the `TARGETARCH` build arg to produce the right
architecture binary. Node stages use QEMU emulation (which is slower but
acceptable for our ~1 min admin build).

### Buildx builder

We use a named buildx builder to keep state out of Docker's default
environment:

```bash
docker buildx create --name honeydue-builder --use
docker buildx inspect --bootstrap
```

The `honeydue-builder` is a docker-container driver — spawns a
BuildKit container when building, tears it down when idle. Supports
multi-platform and caches layers across builds.

## From local file to cluster — the full path

```mermaid
flowchart LR
    subgraph dev[Operator workstation]
        Code[Source code]
        Dockerfile
        Buildx[docker buildx]
    end
    subgraph Gitea[gitea.treytartt.com]
        Reg[Package registry]
    end
    subgraph K8s[k3s cluster]
        Kubelet
        Containerd
        Pod
    end

    Code --> Dockerfile
    Dockerfile --> Buildx
    Buildx -- push --> Reg
    Reg -- pull --> Kubelet
    Kubelet --> Containerd
    Containerd --> Pod
```

### End-to-end

1. **Operator pushes code**: commits to `main` locally
2. **Operator builds + pushes image**: `docker buildx build --push ...`
   from the repo root. Build takes 1–3 minutes first time, seconds on
   warm cache.
3. **Image lands in Gitea**: visible at
   `https://gitea.treytartt.com/admin/-/packages/container/honeydue-api`
4. **Operator updates Deployment**: `kubectl set image deployment/api
   api=gitea.treytartt.com/admin/honeydue-api:$NEW_SHA -n honeydue`
5. **K8s begins rolling update**: creates new ReplicaSet with new image
6. **Kubelet on target node** sees a pod with an image it doesn't have
7. **Kubelet calls containerd**: "pull this image using these creds"
8. **Containerd authenticates** to Gitea registry using the PAT from
   `gitea-credentials` Secret, downloads the image
9. **Containerd starts the container** with the new image
10. **Readiness probe passes**: new pod joins the Service endpoints
11. **Kubelet tears down** an old pod

## Pushing manually

If you need to push a one-off image (e.g., testing a fix):

```bash
# Login (once per session)
set -a; source deploy/registry.env; set +a
printf '%s' "$REGISTRY_TOKEN" | docker login "$REGISTRY" -u "$REGISTRY_USERNAME" --password-stdin

# Build + push
cd honeyDueAPI-go
SHA=$(git rev-parse --short HEAD)
docker buildx build \
  --platform linux/amd64 \
  --target api \
  -t "gitea.treytartt.com/admin/honeydue-api:${SHA}" \
  --push .

# Logout (don't leave creds in ~/.docker/config.json)
docker logout gitea.treytartt.com
```

## Image sizes

Current images:

| Image | Size | Layers |
|---|---|---|
| `honeydue-api` | ~53 MB | Alpine base + Go binary |
| `honeydue-worker` | ~50 MB | Alpine base + Go binary |
| `honeydue-admin` | ~150 MB | Node 20 alpine + Next.js standalone |

The Go binaries are statically compiled, CGO_ENABLED=0. Alpine is the
base for smallest footprint.

## Image retention

Gitea does **not auto-prune** images. Every `:<sha>` tag accumulates
forever. The package page at
`https://gitea.treytartt.com/admin/-/packages/container/honeydue-api`
lists them all.

At current pace (deploys ~few/week, images ~50-150 MB each), this grows
~10 GB/year. Not critical; 80 GB node disk can take years.

**TODO**: Add a monthly cleanup: delete all but last 30 tags per image.
Can be a cron job or a manual quarterly cleanup.

## Image verification — not yet

We do not sign images or verify signatures. An attacker who compromised
Gitea could push a malicious image under an existing tag (though Gitea
should prevent tag reuse if immutable tags are configured).

**TODO** (Chapter 20): Add [cosign](https://github.com/sigstore/cosign)
for signing at build time + `Kyverno` or `Connaisseur` policy to verify
at pull time.

## Gitea registry itself

The Gitea instance runs outside this k3s cluster on its own VPS
(operator's existing infrastructure). It's **not** part of the honeyDue
deployment — it's adjacent infrastructure.

If the Gitea host goes down:
- Currently-running pods keep working (they already pulled their images)
- New deployments/scale-ups fail at the image-pull step
- No impact on existing user traffic

This is an acceptable external dependency. Gitea host has its own
uptime story.

## Cost

**$0/mo.** Gitea registry is included in the Gitea install we already
pay the VPS for (not accounted to honeyDue's cost).

If we ever switched to GHCR, cost would still be $0 for public images
or bundled with our (nonexistent) GitHub Team subscription.

## What we don't have

- **Image scanning** (Trivy, Snyk) — scan images for known CVEs on push
- **Image signing** (cosign)
- **Multi-region replication** — only hosted in one place
- **High availability** — Gitea is single-instance

For our scale, none of these are needed. TODO (Chapter 20) if the
operator appetite increases.

## Operator cheat sheet

```bash
# List packages via API
curl -sS "https://gitea.treytartt.com/api/v1/packages/admin?type=container" \
  -H "Accept: application/json" | jq .

# Browse in UI
#   https://gitea.treytartt.com/admin/-/packages

# Delete a specific tag via API
curl -X DELETE \
  -H "Authorization: token $GITEA_PAT" \
  "https://gitea.treytartt.com/api/v1/packages/admin/container/honeydue-api/237c6b8"

# Login from kubectl side (refresh the Secret)
kubectl create secret docker-registry gitea-credentials -n honeydue \
  --docker-server=gitea.treytartt.com \
  --docker-username=admin \
  --docker-password=<new PAT> \
  --dry-run=client -o yaml | kubectl apply -f -

# After rotating PAT, restart pods that use it for pulls
kubectl rollout restart -n honeydue deploy/api deploy/admin deploy/worker
```

## References

- [Gitea Container Registry][gitea-cr]
- [Docker buildx multi-platform][buildx]
- [Kubernetes image pull secrets][pull-secrets]
- [cosign][cosign]

[gitea-cr]: https://docs.gitea.com/usage/packages/container
[buildx]: https://docs.docker.com/build/buildx/
[pull-secrets]: https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/
[cosign]: https://github.com/sigstore/cosign