A user clicks Place Order. Your app charges the card, generates a PDF receipt, sends a confirmation email, notifies the warehouse API, updates three database tables, and fires an analytics event. The user stares at a spinner for nine seconds. Some of those seconds are your code. Most of them are someone else's server thinking about it.
That nine-second request is not a performance bug you can fix with a faster database query. It is an architectural problem. You have packed synchronous work and asynchronous work into the same moment, and your user is paying the tax on all of it.
The fix is a background job queue — a foundational pattern that separates what the user needs to see right now from work the system can finish after the response is sent. It sounds simple. The decisions inside it are not.
The Real Cost of Doing Everything in the Request Cycle
Every HTTP request holds a thread, a connection, and a user's attention. The longer that request takes, the more resources your application server pins in place. Under low traffic, nobody notices. Under real load — a product launch, a marketing push, a seasonal spike — those pinned resources stack up fast.
Here is what actually breaks:
- Thread exhaustion. Your web server has a finite thread pool. If each request takes eight seconds because it is waiting on a third-party email API, you can only serve a handful of concurrent users before new requests queue or time out. The app is not slow because it lacks CPU — it is slow because every thread is blocked waiting on something it does not control.
- Cascading timeouts. A slow upstream service (a payment gateway, a PDF renderer, a webhook endpoint) does not just slow one request. It backs up your entire server. Users retry. Retries compound. You get a pile-up that looks like a traffic spike but is actually a downstream latency problem amplified by your own architecture.
- Unreliable user experience. If sending a welcome email fails mid-request, what do you show the user? An error page for a successful signup? Swallowing the error silently means the email never goes out and nobody knows. Neither outcome is acceptable.
These are not hypothetical edge cases. They are the normal failure modes of a growing application that has not yet separated its synchronous and asynchronous concerns.
When to Add a Message Queue to Your App
Not every application needs a job queue on day one. But there are clear signals that the architecture is ready for one — or overdue.
You are calling external APIs inside the request
Any outbound call to a service you do not own — email delivery, SMS, payment processing callbacks, third-party analytics, webhook notifications — introduces latency and failure modes you cannot control. Move it out of the request. The user does not need to wait for Mailgun to accept the message before they see their confirmation page.
You are doing CPU- or I/O-heavy work synchronously
Image resizing, PDF generation, video transcoding, large CSV exports, report aggregation — these operations take seconds or minutes, not milliseconds. They belong in a worker process that can take its time without blocking the web tier.
You need reliable delivery, not just fast delivery
If a task must happen — an invoice must be generated, a compliance record must be written, a downstream system must be notified — then you need a mechanism that persists the intent, retries on failure, and surfaces errors to an operator. A fire-and-forget thread in your web process is not that mechanism. A durable queue with retry logic and dead-letter handling is.
Your deploy or scaling model needs the work decoupled
When web servers and background workers are the same process, you cannot scale them independently. You end up over-provisioning web capacity to handle background load, or starving background work to keep the web tier responsive. Splitting them lets you right-size each tier — and that matters for cost optimization as your cloud bill grows.
What Belongs in the Queue — and What Does Not
The decision is not can this be async? but does the user need the result of this work to continue their action?
Good candidates for background processing
- Notification delivery — email, SMS, push notifications, Slack messages. The user's action is complete whether or not the notification has landed yet.
- Document and media generation — PDFs, thumbnails, transcoded video. Show the user a placeholder or a processing status and deliver the artifact when it is ready.
- Third-party syncs — pushing data to a CRM, an analytics warehouse, an ERP system. These are system-to-system concerns, not user-facing ones.
- Scheduled maintenance — cache warming, stale-data cleanup, usage aggregation, nightly report runs.
- Webhook fanout — delivering event payloads to multiple subscriber endpoints. Each delivery can fail independently and should be retried independently.
What should stay in the request
- The primary write. If the user is creating an order, the order record needs to exist before you return a response. The side effects of that order (notifications, syncs, analytics) can be queued.
- Authorization checks. Never defer access control. The decision about whether the user can do something must happen synchronously.
- Anything the response depends on. If the next page the user sees requires data produced by this operation, it cannot be async. A common mistake is queueing a task and then immediately redirecting to a page that reads the result — and showing stale or missing data because the worker has not finished yet.
Architectural Decisions That Matter Early
Choosing to add a queue is the easy part. The design decisions around it are where teams stumble.
Pick the right backing store for your stage
You do not need Apache Kafka on day one. For most applications starting out, a managed queue service — Amazon SQS, Google Cloud Pub/Sub, or even a Redis-backed queue like Sidekiq or BullMQ — is the right starting point. These are operationally simpler, cost less at low volume, and still scale respectably.
Kafka, NATS, or RabbitMQ become relevant when you need ordered event streams, fan-out to multiple consumers, replay capability, or throughput beyond what a simple queue handles. Adopting them too early means you are operating distributed infrastructure that does not yet earn its complexity.
Design jobs to be idempotent
Workers will crash. Messages will be delivered more than once. Network partitions will happen. If processing the same job twice produces a duplicate invoice or sends the same email again, your queue is a liability, not an asset.
Every job should be safe to retry. That means using unique identifiers to check whether work has already been done, designing database writes to be upserts where appropriate, and treating external API calls as operations that might need deduplication on the other side.
Make jobs small and observable
A job that does seven things is hard to retry, hard to debug, and hard to monitor. Break complex workflows into a chain of smaller jobs, each with a clear responsibility. When a PDF generation job fails, you want to see that job in your dead-letter queue — not a monolithic process-order job where you have to guess which of seven steps broke.
This is where an observability stack earns its keep. Instrument your workers with the same rigor you apply to your web tier: job execution time, failure rate, queue depth, retry count. A queue that silently backs up is worse than no queue at all, because the failure is invisible until it is catastrophic.
Plan your retry and dead-letter strategy before you ship
Retries without backoff will hammer a failing upstream service. Unlimited retries will loop forever. No dead-letter queue means failed jobs vanish silently.
A reasonable starting point: exponential backoff with jitter, a maximum retry count (three to five for most jobs), and a dead-letter queue that alerts your team when it has entries. From there, you tune based on what you observe in production.
Version your job payloads
Your job schema will change. You will add fields, rename parameters, change how data is structured. If you deploy a new worker that expects a new payload format while old-format messages are still in the queue, those messages will fail. Include a version identifier in every job payload, and write workers that handle at least the current and previous version gracefully. This is especially important during rolling deploys in container orchestration environments, where old and new worker versions coexist for a window of time.
Infrastructure as Code Makes This Repeatable
A background job system is not just application code. It is infrastructure: queue resources, worker scaling policies, dead-letter queues, IAM permissions, alerting rules. If any of that is configured by hand in a cloud console, it will drift, it will break during a rebuild, and it will be undocumented when the person who set it up leaves.
Defining your queue infrastructure declaratively — in Terraform, Pulumi, or CloudFormation — means every queue, every permission policy, every alarm threshold is version-controlled, reviewable in a pull request, and reproducible across environments. When you need a staging environment that mirrors production, you apply the same module. No guesswork, no snowflake configuration.
This is the soil work that makes the rest sustainable. The queue itself is straightforward. The discipline around it — automated deploys, policy-as-code checks, environment parity — is what determines whether it stays reliable as the system grows.
A Pattern, Not a Product
Background job processing is not a feature you bolt on. It is an architectural pattern that reshapes how your application handles work, how it fails, and how it scales. Done well, it makes your web tier fast, your side effects reliable, and your infrastructure costs proportional to actual load instead of worst-case capacity.
Done poorly — or deferred too long — it leaves you with a web application that slows down under growth, fails unpredictably when third-party services hiccup, and costs more to run than it should because every server is doing everything.
The right time to architect this is before the pain compounds. If your deployment process is already manual, your scaling strategy is already reactive, and your request times are already climbing, the queue is one piece of a larger foundation that needs attention.
Related service: Cloud infrastructure and DevOps services in Tampa Bay — what we build, how we work, and what it costs.
Build the Foundation Before the Load Arrives
At FigTree Development, we design cloud architecture and DevOps foundations that are scalable from day one — infrastructure as code, CI/CD pipelines, container orchestration, and observability stacks that surface problems before your users do. Whether you are untangling a monolith that has outgrown its original design or building the right patterns into a new system from the start, we bring principal-level engineering judgment without the overhead of a full-time senior hire.
If your application is doing too much in the request cycle and you are not sure where to draw the line, that is exactly the kind of architectural decision we help teams think through. Book a free 20-minute discovery call and let us look at where the bottlenecks are — and what the right foundation looks like for where you are headed.