feat(uploads): direct-to-B2 presigned uploads with content-length-range policy
Backend CI / Test (push) Has been cancelled
Backend CI / Contract Tests (push) Has been cancelled
Backend CI / Build (push) Has been cancelled
Backend CI / Lint (push) Has been cancelled
Backend CI / Secret Scanning (push) Has been cancelled

Replaces the multipart-via-API path for image uploads with a three-step
direct-to-storage flow:

  1. Client POSTs /api/uploads/presign with content_length + content_type;
     server validates size (10 MB cap), mime allow-list per category, rate
     limit (50/hour/user via Redis sliding window), and concurrent unclaimed
     cap (10 in-flight per user). On success it persists a pending_uploads
     row, signs an S3 POST policy with content-length-range bound to the
     claimed length ±256 bytes, and returns the URL+fields.
  2. Client POSTs the bytes directly to B2 using the signed policy. B2
     enforces size, content-type, and key match before accepting.
  3. Client passes upload_ids[] to /api/task-completions/ or /api/documents/.
     Service HEADs each B2 object, verifies size matches expected_bytes
     within slack, marks pending_uploads claimed_at, and creates the
     associated TaskCompletionImage / DocumentImage rows.

Bytes never traverse our API server. The 1 MB Echo BodyLimit middleware
that was rejecting all task-completion image uploads becomes irrelevant
for this path. Existing multipart endpoints stay functional alongside,
soak-testing the new path before legacy removal.

Cleanup:
  - cmd/worker registers a new hourly cron (TypeUploadCleanup, "30 * * * *")
    that reaps pending_uploads where claimed_at IS NULL AND expires_at < NOW().
    Reaps both the B2 object and the row.
  - B2 bucket lifecycle rule on `uploads/` prefix (7 days hide → 1 day delete)
    documented in deploy-k3s/manifests/b2-lifecycle.md as a backstop.

Schema:
  - migrations/000002_pending_uploads.sql adds the table + partial index for
    cleanup + nullable pending_upload_id FKs on task_taskcompletionimage and
    task_documentimage.

Policy (single tier, no free/pro split):
  - 10 MB cap per upload
  - 50 presigns/hour/user
  - 10 concurrent unclaimed uploads/user
  - allow-list: jpeg/png/heic/heif/webp for image categories;
    + pdf for document_file

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Trey t
2026-05-01 14:36:42 -07:00
parent 9bee436e86
commit 29c9014a33
20 changed files with 1032 additions and 9 deletions
+48 -2
View File
@@ -7,9 +7,11 @@ import (
"github.com/rs/zerolog/log"
"github.com/treytartt/honeydue-api/internal/apperrors"
"github.com/treytartt/honeydue-api/internal/dto/requests"
"github.com/treytartt/honeydue-api/internal/dto/responses"
"github.com/treytartt/honeydue-api/internal/i18n"
"github.com/treytartt/honeydue-api/internal/middleware"
"github.com/treytartt/honeydue-api/internal/models"
"github.com/treytartt/honeydue-api/internal/services"
)
@@ -22,18 +24,26 @@ type FileOwnershipChecker interface {
// UploadHandler handles file upload endpoints
type UploadHandler struct {
storageService *services.StorageService
storageService *services.StorageService
uploadService *services.UploadService // optional — only set when S3 storage is configured
fileOwnershipChecker FileOwnershipChecker
}
// NewUploadHandler creates a new upload handler
func NewUploadHandler(storageService *services.StorageService, fileOwnershipChecker FileOwnershipChecker) *UploadHandler {
return &UploadHandler{
storageService: storageService,
storageService: storageService,
fileOwnershipChecker: fileOwnershipChecker,
}
}
// SetUploadService wires the presigned-URL upload service. Called from the
// router only when S3 storage is configured; with local-disk storage the
// presign endpoint is unsupported and returns 503.
func (h *UploadHandler) SetUploadService(s *services.UploadService) {
h.uploadService = s
}
// UploadImage handles POST /api/uploads/image
// Accepts multipart/form-data with "file" field
func (h *UploadHandler) UploadImage(c echo.Context) error {
@@ -138,3 +148,39 @@ func (h *UploadHandler) DeleteFile(c echo.Context) error {
return c.JSON(http.StatusOK, responses.MessageResponse{Message: i18n.LocalizedMessage(c, "message.file_deleted")})
}
// PresignUpload handles POST /api/uploads/presign.
//
// Returns a short-lived signed POST policy that the client uses to upload an
// image or document directly to B2, bypassing the API entirely for the byte
// transfer. The returned `id` is later passed in `upload_ids[]` on the
// task-completion or document creation endpoints to attach the object.
func (h *UploadHandler) PresignUpload(c echo.Context) error {
if h.uploadService == nil {
return apperrors.Internal(nil)
}
user, err := middleware.MustGetAuthUser(c)
if err != nil {
return err
}
var req requests.PresignUploadRequest
if err := c.Bind(&req); err != nil {
return apperrors.BadRequest("error.invalid_request")
}
if err := c.Validate(&req); err != nil {
return err
}
resp, err := h.uploadService.Presign(
c.Request().Context(),
user.ID,
models.UploadCategory(req.Category),
req.ContentType,
req.ContentLength,
)
if err != nil {
return err
}
return c.JSON(http.StatusCreated, resp)
}