001 bullmq vs pg queues

ADR-001 — BullMQ over PostgreSQL-backed queues

Date: 2025-01-15 Status: Accepted Deciders: Nicolas (founder)

---

Context

The platform needs an async job queue to:

• Trigger PowerShell audit pipelines (long-running, up to 20 min)
• Import audit results JSON into PostgreSQL after completion
• Schedule recurring audits (cron)
• Retry failed jobs with backoff

Two main options were considered: BullMQ (Redis-backed) and pg-boss (PostgreSQL-backed).

Decision

BullMQ with Redis was chosen.

Rationale

Criterion	BullMQ (Redis)	pg-boss (PG)
Job throughput	Very high (Redis in-memory)	Limited (DB polling)
Real-time monitoring	Bull Board UI out of the box	Manual dashboard needed
Retry / backoff	Built-in exponential backoff	Manual implementation
Cron scheduling	Built-in	Built-in
Infrastructure	Redis already required (sessions/rate-limiting)	No extra infra
Worker isolation	Docker container, separate process	Same DB transaction scope

Redis is already part of the stack (NextAuth sessions, rate limiting). Adding pg-boss would not eliminate the Redis dependency.

Consequences

• Redis is a required infrastructure component (documented in docker-compose)
• Bull Board UI served at /api/admin/queues (MSSP_ADMIN only)
• Worker runs as a separate Docker service (audit-worker) — isolated from the Next.js process
• Job failure notifications via BullMQ event hooks → Alert model

Alternatives rejected

• pg-boss: Would share the PostgreSQL connection pool; polling-based latency unacceptable for audit triggers
• Direct GitLab CI pipeline trigger: Acceptable for initial MVP but couples the platform to GitLab availability; BullMQ adds resilience (retry, queue inspection)