Getting Started with BullMQ v5 in Next.js 15
BullMQ v5 builds on Redis Streams to provide at-least-once delivery semantics, job priorities, delays, and repeatable schedules. When paired with Next.js 15's App Router, it allows you to offload heavy work from server actions and route handlers while keeping your frontend responsive.
Start by creating a new project with the following command:
npx create-next-app@latest bullmq-demo --typescript --tailwind --eslint --app --yes
Then install the core dependencies. Use these exact versions that were stable at the time of writing:
npm install bullmq@5.8.2 ioredis@5.4.1 @bull-board/api@5.14.0 @bull-board/ui@5.14.0
For the worker process we will also need tsx for development watching:
npm install -D tsx
Create a lib/redis.ts file to centralize your connection. Using Upstash Redis is the fastest path for Vercel deployments because it offers HTTP-based access that works within serverless constraints, though for high-throughput workloads a direct TCP connection to a self-hosted Redis 7.2+ instance yields lower latency.
import { Redis } from 'ioredis';
export const redisConnection = new Redis({
host: process.env.REDIS_HOST || 'localhost',
port: parseInt(process.env.REDIS_PORT || '6379'),
password: process.env.REDIS_PASSWORD,
maxRetriesPerRequest: null, // required by BullMQ
enableReadyCheck: false,
});
export const defaultQueueOptions = {
connection: redisConnection,
defaultJobOptions: {
removeOnComplete: { count: 1000 },
removeOnFail: { count: 500 },
attempts: 3,
backoff: {
type: 'exponential',
delay: 1000,
},
},
};
The maxRetriesPerRequest: null setting is mandatory for BullMQ v5 compatibility with ioredis. The removeOnComplete and removeOnFail options prevent unbounded memory growth in Redis.
Creating Your First Queue and Adding Jobs
Define a queue for document processing. Create lib/queues/documentQueue.ts:
import { Queue } from 'bullmq';
import { defaultQueueOptions } from '../redis';
export const documentQueue = new Queue('document-processing', defaultQueueOptions);
export type DocumentJobData = {
userId: string;
documentUrl: string;
task: 'summarize' | 'embed' | 'classify';
metadata?: Record;
};
From a Next.js Route Handler or Server Action you can add jobs without blocking the response. Here is a practical server action example using Zod for validation:
'use server';
import { z } from 'zod';
import { documentQueue, type DocumentJobData } from '@/lib/queues/documentQueue';
const schema = z.object({
documentUrl: z.string().url(),
task: z.enum(['summarize', 'embed', 'classify']),
});
export async function queueDocumentProcessing(formData: FormData) {
const parsed = schema.parse({
documentUrl: formData.get('documentUrl'),
task: formData.get('task'),
});
const job = await documentQueue.add(
'process-document',
{
userId: 'user_123',
documentUrl: parsed.documentUrl,
task: parsed.task,
} satisfies DocumentJobData,
{
priority: parsed.task === 'embed' ? 1 : 10,
delay: parsed.task === 'classify' ? 5000 : 0,
}
);
return { jobId: job.id };
}
Note the priority field: lower numbers mean higher priority. Embedding jobs are scheduled first because they are usually prerequisites for search features. The delay option is useful for debouncing repeated user actions.
Implementing the Worker Process
Workers run independently of your Next.js request handlers. Create workers/documentWorker.ts:
import { Worker, type Job } from 'bullmq';
import { redisConnection } from '../lib/redis';
import { documentQueue } from '../lib/queues/documentQueue';
async function processDocument(job: Job) {
const data = job.data as DocumentJobData;
console.log(`Processing ${data.task} for document ${data.documentUrl}`);
// Simulate different processing times and failure modes
if (data.task === 'summarize') {
// In production this would call Ollama, Groq, or Anthropic
await new Promise(resolve => setTimeout(resolve, 4500));
return { summary: 'This document discusses quarterly revenue growth...', tokensUsed: 1240 };
}
if (data.task === 'embed') {
// This is where you would call Transformers.js, Voyage AI, or OpenAI embeddings
await new Promise(resolve => setTimeout(resolve, 2200));
return { embeddingDimension: 1024, vectorId: 'vec_987' };
}
throw new Error('Unknown task type');
}
const worker = new Worker(documentQueue.name, processDocument, {
connection: redisConnection,
concurrency: 5,
limiter: {
max: 20,
duration: 60000, // max 20 jobs per minute
},
});
worker.on('completed', (job) => {
console.log(`Job ${job.id} completed successfully`);
});
worker.on('failed', (job, err) => {
console.error(`Job ${job?.id} failed with ${err.message}`);
});
console.log('Document worker started with concurrency 5');
Run the worker during development with:
npx tsx watch workers/documentWorker.ts
The limiter option prevents hammering external AI APIs. Concurrency of 5 is a safe starting point for CPU-bound embedding work on a 2-vCPU machine; adjust upward for I/O bound tasks.
Complex Workflows with BullMQ Flows
Many AI pipelines require sequential or parallel steps. BullMQ Flows let you define parent-child relationships with automatic propagation of failures.
Extend the queue setup with a flow producer:
import { FlowProducer } from 'bullmq';
const flowProducer = new FlowProducer({ connection: redisConnection });
export async function createDocumentProcessingFlow(documentUrl: string, userId: string) {
const flow = await flowProducer.add({
name: 'document-pipeline',
queueName: 'document-processing',
data: { userId, documentUrl, task: 'embed' as const },
children: [
{
name: 'generate-summary',
queueName: 'document-processing',
data: { userId, documentUrl, task: 'summarize' as const },
},
{
name: 'classify-content',
queueName: 'document-processing',
data: { userId, documentUrl, task: 'classify' as const },
},
],
opts: { priority: 1 },
});
return flow;
}
When the embedding job completes, both the summary and classification jobs receive the embedding vector as additional data via parent.data inside the child processor. This pattern avoids race conditions common in naive setTimeout chaining.
Error Handling Strategies and Retries
BullMQ v5 provides sophisticated retry mechanisms. The defaultJobOptions shown earlier use exponential backoff. You can customize per job type:
await documentQueue.add('process-document', jobData, {
attempts: 5,
backoff: {
type: 'exponential',
delay: 2000,
},
failOnError: false,
});
Implement a repeatable job for nightly report generation:
await documentQueue.add(
'nightly-reports',
{ type: 'portfolio-summary' },
{
repeat: {
pattern: '0 3 * * *', // every day at 3 AM
},
}
);
Monitor stalled jobs by configuring stallInterval on the worker. For jobs that exceed 30 seconds, BullMQ will emit a stalled event allowing you to alert via Slack or email.
Real-Time Monitoring with Bull Board
Expose a monitoring UI securely. Create an API route at app/api/admin/queues/route.ts:
import { createBullBoard } from '@bull-board/api';
import { BullMQAdapter } from '@bull-board/api/bullMQAdapter';
import { ExpressAdapter } from '@bull-board/express';
import { documentQueue } from '@/lib/queues/documentQueue';
const serverAdapter = new ExpressAdapter();
serverAdapter.setBasePath('/api/admin/queues');
createBullBoard({
queues: [new BullMQAdapter(documentQueue)],
serverAdapter,
});
export async function GET(req: Request) {
// Add auth middleware here in production (e.g. NextAuth or Clerk)
return serverAdapter.getHandler()(req);
}
Access the dashboard at /api/admin/queues. It shows real-time job throughput, failure rates, and allows manual retry of failed jobs. In production, protect this route with IP allow-listing or strong authentication because it exposes Redis contents.
Production Deployment Patterns
Running workers on Vercel is not recommended because serverless functions have maximum execution timeouts. Instead deploy workers to a dedicated VM or use Coolify on a VPS with PM2 ecosystem file:
{
"apps": [{
"name": "document-worker",
"script": "tsx",
"args": "workers/documentWorker.ts",
"instances": 3,
"exec_mode": "cluster",
"env": {
"NODE_ENV": "production"
}
}]
}
Redis should be configured with persistence (AOF enabled) and at least 2 replicas for high availability. For a 4-core machine, set worker concurrency to roughly 3× the number of CPU cores for I/O heavy jobs and 1× for CPU heavy embedding tasks.
Track job metrics by listening to BullMQ events and forwarding them to Prometheus. A simple counter for completed jobs per minute helps set autoscaling thresholds.
Case Study: AI Document Intelligence Pipeline
Consider a SaaS that lets users upload financial reports. The pipeline must:
- Generate vector embeddings for semantic search
- Extract a concise executive summary using an LLM
- Classify the document type (10-K, earnings call, etc.)
- Notify the user when complete
The flow defined earlier orchestrates these steps. The embedding job writes its vector ID to Supabase using Drizzle ORM. Child jobs read that ID to enrich their context before calling the LLM. This reduces token usage because the LLM only receives relevant extracted text instead of the full document.
Here is the production worker updated with real Supabase and OpenAI integration points (using placeholders for API keys):
import { createClient } from '@supabase/supabase-js';
import OpenAI from 'openai';
const supabase = createClient(
process.env.SUPABASE_URL!,
process.env.SUPABASE_SERVICE_KEY!
);
const openai = new OpenAI({ apiKey: process.env.OPENAI_API_KEY });
async function processDocument(job: Job) {
const { documentUrl, task, userId } = job.data;
// Fetch document text (in reality use pdf-parse or similar)
const text = 'Extracted document text here...';
if (task === 'embed') {
const embedding = await openai.embeddings.create({
model: 'text-embedding-3-small',
input: text.slice(0, 8000),
});
const vector = embedding.data[0].embedding;
await supabase.from('document_vectors').insert({
user_id: userId,
document_url: documentUrl,
embedding: vector,
});
return { vectorId: 'vec_' + Date.now() };
}
if (task === 'summarize') {
const completion = await openai.chat.completions.create({
model: 'gpt-4o-mini',
messages: [{
role: 'system',
content: 'You are an expert financial analyst. Provide a 3-sentence summary.'
}, {
role: 'user',
content: text
}],
max_tokens: 300,
});
await supabase.from('document_summaries').insert({
user_id: userId,
document_url: documentUrl,
summary: completion.choices[0].message.content,
});
return { success: true };
}
}
This pattern keeps your API routes under 100 ms response time while the heavy LLM work happens asynchronously. The Supabase inserts can trigger Edge Functions or database webhooks for user notifications via PartyKit or Resend email.
Scaling Considerations and Trade-offs
BullMQ scales horizontally by adding more worker processes. Each additional worker consumes one Redis connection. Monitor the connected_clients metric on Redis and increase the maxclients limit if you run more than 30 workers.
Payload size matters. Redis strings are limited to 512 MB but practical job data should stay under 100 KB. For larger payloads, store the bulk data in S3 or Supabase Storage and pass only a reference in the job.
Comparison with alternatives:
- Inngest offers simpler developer experience and built-in retry UIs but charges per execution and requires sending data to their infrastructure.
- Temporal provides durable execution guarantees and is superior for long-running sagas spanning days, yet has steeper operational complexity.
- Native BullMQ on self-hosted Redis gives you complete data sovereignty and zero per-execution cost after infrastructure is paid for.
For most startups processing fewer than 50k jobs per day, self-hosted Redis 7.2 on a Hetzner CPX31 (4 vCPU, 8 GB) running BullMQ workers costs under $30/month while delivering sub-50 ms job scheduling latency.
Security Best Practices
Never put secrets or PII directly in job data. Use a reference pattern: upload sensitive files to a private bucket and pass a short-lived signed URL that expires in 15 minutes. Encrypt any sensitive metadata with libsodium before storing in Redis.
Implement queue access controls by using different Redis databases or ACLs for different environments. Rate-limit job creation per user to prevent denial-of-service via queue flooding. Log job IDs but not full payloads for audit trails.
Enable Redis authentication and TLS in production. For Bull Board, wrap the endpoint with Auth.js middleware and role-based checks so only admin users can view or retry jobs.
Observability and Alerting
Expose Prometheus metrics by adding the bullmq-prom package or manually incrementing counters on completed and failed events. Key metrics to track:
- Jobs completed per minute
- Failure rate (target < 0.5%)
- Average job duration per task type
- Queue depth (alert if > 200 pending jobs)
Integrate with Grafana for dashboards showing queue health. Set up an alert that triggers if the oldest unprocessed job is older than 5 minutes for high-priority queues. This gives you early warning before users notice degraded performance.
Maintenance and Job Lifecycle Management
Schedule a recurring job that calls queue.clean(0, 1000, 'completed') to prune old jobs. Monitor Redis memory usage with the INFO memory command. Set maxmemory-policy allkeys-lru so Redis automatically evicts old keys under pressure.
Use repeatable jobs sparingly; each repeat creates a new job ID. For cron-like behavior consider a separate lightweight scheduler such as Temporal or a simple Node.js process with node-cron that adds jobs to BullMQ.
When updating job processor logic, version your job names (e.g. process-document-v2) so old jobs in the queue do not break after a deploy. Gradually migrate by keeping both processors temporarily.
Performance Tuning Techniques
Enable Redis pipelining by setting enableAutoPipelining: true on the ioredis client. This batches commands and can improve throughput by 30-40% for high-volume queue additions.
Use separate queues for different latency classes. A realtime-notifications queue with dedicated high-priority workers ensures user-facing jobs are not starved by heavy batch embedding jobs in the default queue.
Profile worker CPU and memory with clinic.js or 0x. If you see high garbage collection pressure, reduce concurrency or increase available RAM. For GPU-accelerated embedding workloads, consider moving those specific jobs to a dedicated Python worker that communicates via Redis streams directly.
These patterns have been validated across production systems handling thousands of daily AI jobs without data loss or duplicate processing when the idempotency keys pattern is followed: always check Supabase for existing results before performing expensive LLM calls inside the worker.
The combination of BullMQ v5, Redis, and Next.js 15 gives you a robust, observable, and cost-effective foundation for any asynchronous workload from simple email delivery to sophisticated multi-step AI document intelligence systems.
