Trey t
29c9014a33
Replaces the multipart-via-API path for image uploads with a three-step
direct-to-storage flow:
1. Client POSTs /api/uploads/presign with content_length + content_type;
server validates size (10 MB cap), mime allow-list per category, rate
limit (50/hour/user via Redis sliding window), and concurrent unclaimed
cap (10 in-flight per user). On success it persists a pending_uploads
row, signs an S3 POST policy with content-length-range bound to the
claimed length ±256 bytes, and returns the URL+fields.
2. Client POSTs the bytes directly to B2 using the signed policy. B2
enforces size, content-type, and key match before accepting.
3. Client passes upload_ids[] to /api/task-completions/ or /api/documents/.
Service HEADs each B2 object, verifies size matches expected_bytes
within slack, marks pending_uploads claimed_at, and creates the
associated TaskCompletionImage / DocumentImage rows.
Bytes never traverse our API server. The 1 MB Echo BodyLimit middleware
that was rejecting all task-completion image uploads becomes irrelevant
for this path. Existing multipart endpoints stay functional alongside,
soak-testing the new path before legacy removal.
Cleanup:
- cmd/worker registers a new hourly cron (TypeUploadCleanup, "30 * * * *")
that reaps pending_uploads where claimed_at IS NULL AND expires_at < NOW().
Reaps both the B2 object and the row.
- B2 bucket lifecycle rule on `uploads/` prefix (7 days hide → 1 day delete)
documented in deploy-k3s/manifests/b2-lifecycle.md as a backstop.
Schema:
- migrations/000002_pending_uploads.sql adds the table + partial index for
cleanup + nullable pending_upload_id FKs on task_taskcompletionimage and
task_documentimage.
Policy (single tier, no free/pro split):
- 10 MB cap per upload
- 50 presigns/hour/user
- 10 concurrent unclaimed uploads/user
- allow-list: jpeg/png/heic/heif/webp for image categories;
+ pdf for document_file
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
40 lines
1.7 KiB
SQL
40 lines
1.7 KiB
SQL
-- +goose Up
|
|
-- pending_uploads tracks short-lived presigned-URL upload sessions for direct
|
|
-- client-to-B2 uploads. A row is created when the client requests a presigned
|
|
-- POST policy, and either claimed (linked to a task_completion_image or
|
|
-- document_image) or reaped by the cleanup worker after expiry.
|
|
CREATE TABLE pending_uploads (
|
|
id BIGSERIAL PRIMARY KEY,
|
|
user_id BIGINT NOT NULL REFERENCES auth_user(id) ON DELETE CASCADE,
|
|
category VARCHAR(32) NOT NULL,
|
|
b2_key VARCHAR(255) NOT NULL UNIQUE,
|
|
content_type VARCHAR(127) NOT NULL,
|
|
expected_bytes BIGINT NOT NULL,
|
|
actual_bytes BIGINT,
|
|
claimed_at TIMESTAMPTZ,
|
|
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
|
expires_at TIMESTAMPTZ NOT NULL
|
|
);
|
|
|
|
-- Quota lookups: SUM/COUNT by user, ordered by recency.
|
|
CREATE INDEX idx_pending_uploads_user_created
|
|
ON pending_uploads (user_id, created_at DESC);
|
|
|
|
-- Cleanup worker scan: only unclaimed expired rows. Partial index keeps it tiny.
|
|
CREATE INDEX idx_pending_uploads_cleanup
|
|
ON pending_uploads (expires_at) WHERE claimed_at IS NULL;
|
|
|
|
-- task_completion_image and document_image gain an optional FK to the
|
|
-- pending_uploads row that produced them. Nullable so legacy rows (uploaded
|
|
-- through the multipart path) keep working.
|
|
ALTER TABLE task_taskcompletionimage
|
|
ADD COLUMN pending_upload_id BIGINT REFERENCES pending_uploads(id) ON DELETE SET NULL;
|
|
|
|
ALTER TABLE task_documentimage
|
|
ADD COLUMN pending_upload_id BIGINT REFERENCES pending_uploads(id) ON DELETE SET NULL;
|
|
|
|
-- +goose Down
|
|
ALTER TABLE task_documentimage DROP COLUMN IF EXISTS pending_upload_id;
|
|
ALTER TABLE task_taskcompletionimage DROP COLUMN IF EXISTS pending_upload_id;
|
|
DROP TABLE IF EXISTS pending_uploads;
|