feat(uploads): direct-to-B2 presigned uploads with content-length-range policy
Replaces the multipart-via-API path for image uploads with a three-step
direct-to-storage flow:
1. Client POSTs /api/uploads/presign with content_length + content_type;
server validates size (10 MB cap), mime allow-list per category, rate
limit (50/hour/user via Redis sliding window), and concurrent unclaimed
cap (10 in-flight per user). On success it persists a pending_uploads
row, signs an S3 POST policy with content-length-range bound to the
claimed length ±256 bytes, and returns the URL+fields.
2. Client POSTs the bytes directly to B2 using the signed policy. B2
enforces size, content-type, and key match before accepting.
3. Client passes upload_ids[] to /api/task-completions/ or /api/documents/.
Service HEADs each B2 object, verifies size matches expected_bytes
within slack, marks pending_uploads claimed_at, and creates the
associated TaskCompletionImage / DocumentImage rows.
Bytes never traverse our API server. The 1 MB Echo BodyLimit middleware
that was rejecting all task-completion image uploads becomes irrelevant
for this path. Existing multipart endpoints stay functional alongside,
soak-testing the new path before legacy removal.
Cleanup:
- cmd/worker registers a new hourly cron (TypeUploadCleanup, "30 * * * *")
that reaps pending_uploads where claimed_at IS NULL AND expires_at < NOW().
Reaps both the B2 object and the row.
- B2 bucket lifecycle rule on `uploads/` prefix (7 days hide → 1 day delete)
documented in deploy-k3s/manifests/b2-lifecycle.md as a backstop.
Schema:
- migrations/000002_pending_uploads.sql adds the table + partial index for
cleanup + nullable pending_upload_id FKs on task_taskcompletionimage and
task_documentimage.
Policy (single tier, no free/pro split):
- 10 MB cap per upload
- 50 presigns/hour/user
- 10 concurrent unclaimed uploads/user
- allow-list: jpeg/png/heic/heif/webp for image categories;
+ pdf for document_file
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -26,6 +26,7 @@ const (
|
||||
TypeSendPush = "push:send"
|
||||
TypeOnboardingEmails = "email:onboarding"
|
||||
TypeReminderLogCleanup = "maintenance:reminder_log_cleanup"
|
||||
TypeUploadCleanup = "maintenance:upload_cleanup" // Reaps expired pending_uploads
|
||||
)
|
||||
|
||||
// Handler handles background job processing
|
||||
@@ -39,9 +40,17 @@ type Handler struct {
|
||||
emailService EmailSender
|
||||
notificationService NotificationSender
|
||||
onboardingService OnboardingEmailSender
|
||||
uploadService *services.UploadService
|
||||
config *config.Config
|
||||
}
|
||||
|
||||
// SetUploadService wires the upload service so HandleUploadCleanup can reap
|
||||
// expired pending_uploads rows. Optional; nil-safe — the cleanup handler
|
||||
// no-ops when not configured (e.g. local-disk dev environments).
|
||||
func (h *Handler) SetUploadService(us *services.UploadService) {
|
||||
h.uploadService = us
|
||||
}
|
||||
|
||||
// NewHandler creates a new job handler
|
||||
func NewHandler(db *gorm.DB, pushClient *push.Client, emailService *services.EmailService, notificationService *services.NotificationService, cfg *config.Config) *Handler {
|
||||
h := &Handler{
|
||||
@@ -647,3 +656,24 @@ func (h *Handler) HandleReminderLogCleanup(ctx context.Context, task *asynq.Task
|
||||
log.Info().Int64("deleted", deleted).Msg("Reminder log cleanup completed")
|
||||
return nil
|
||||
}
|
||||
|
||||
// HandleUploadCleanup reaps expired pending_uploads rows and their B2 objects.
|
||||
//
|
||||
// Runs hourly. Each tick processes up to 500 expired sessions; if the queue
|
||||
// is deeper than that, the next hourly run picks up the rest. The B2 bucket
|
||||
// also has a 7-day lifecycle rule on the uploads/ prefix as a backstop in
|
||||
// case this worker is offline for long stretches.
|
||||
func (h *Handler) HandleUploadCleanup(ctx context.Context, task *asynq.Task) error {
|
||||
if h.uploadService == nil {
|
||||
log.Debug().Msg("Upload cleanup skipped: upload service not configured (local-disk storage)")
|
||||
return nil
|
||||
}
|
||||
log.Info().Msg("Processing pending_uploads cleanup...")
|
||||
reaped, err := h.uploadService.CleanupExpired(ctx, 500)
|
||||
if err != nil {
|
||||
log.Error().Err(err).Msg("Pending uploads cleanup failed")
|
||||
return err
|
||||
}
|
||||
log.Info().Int("reaped", reaped).Msg("Pending uploads cleanup completed")
|
||||
return nil
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user