Your 'Zero-Downtime' Deploy Still Drops Requests: Graceful Shutdown, Connection Draining, and the SIGTERM Handling Most Apps Skip

You configured rolling deployments. You set maxUnavailable: 0. You watched the deploy go green. And somewhere in your logs, a handful of 502s appeared anyway — brief, quiet, easy to dismiss. Until they hit a payment callback, a webhook from a third-party integration, or an API call from a mobile client mid-checkout.

The deploy was not zero-downtime. It was low-downtime, which is a fundamentally different promise.

This is one of the most common gaps I see in Kubernetes environments, and it is almost never a cluster problem. It is an application problem — specifically, how your application responds (or fails to respond) when it receives a SIGTERM signal during a rolling deploy. The fix is not complicated, but it requires understanding three things that have to work together: SIGTERM handling in your web app, connection draining at the load balancer, and the sequencing of pod shutdown in Kubernetes itself.

Why Rolling Deploys Still Drop Requests

A rolling deploy in Kubernetes replaces pods one at a time (or in batches). The intent is straightforward: bring up a new pod, verify it is healthy, then terminate the old one. At no point should the service have zero available pods.

The problem is timing. When Kubernetes decides to terminate a pod, two things happen simultaneously — not sequentially:

1. The kubelet sends a SIGTERM to the container process.
2. The Endpoints controller removes the pod from the Service endpoints list, which propagates to kube-proxy, then to any ingress controller or external load balancer.

These two events are not coordinated by a shared lock. They are triggered by the same API object update (the pod entering Terminating state), but they propagate through different control loops at different speeds. The result: there is a window — usually a few hundred milliseconds, sometimes longer — where the load balancer still routes traffic to a pod that has already begun shutting down.

If your application responds to SIGTERM by immediately closing its listener, every in-flight request on that socket gets a connection reset. Every new request routed by a load balancer that has not yet received the updated endpoints list gets a 502.

This is not a theoretical edge case. It is the default behavior of most web frameworks.

SIGTERM Handling: What Most Apps Get Wrong

The SIGTERM signal is the standard Unix mechanism for requesting a graceful shutdown. Kubernetes sends it as the first step of pod termination, then waits for a configurable grace period (terminationGracePeriodSeconds, defaulting to 30 seconds) before sending SIGKILL.

The problem is that many applications either do not handle SIGTERM at all (letting the runtime's default handler kill the process immediately) or handle it by shutting down too eagerly — closing the listening socket, refusing new connections, and draining in-flight requests all in one step.

What you actually need is a two-phase shutdown:

Phase 1: Stop Accepting New Work, Keep Finishing Current Work

When your application receives SIGTERM, it should:

• Stop accepting new connections on its primary listener (or begin returning 503s to new health checks — more on this below).
• Continue processing all in-flight requests to completion.
• Continue responding to any active long-lived connections (WebSockets, gRPC streams, server-sent events) with a clean close sequence.

Phase 2: Exit After Drain Completes

Once all in-flight work is complete, the process exits cleanly with code 0. If the work is not complete within a reasonable deadline (say, 80% of your terminationGracePeriodSeconds), force-close remaining connections and exit anyway — you do not want to hit the SIGKILL wall, because that leaves no room for cleanup.

In practice, SIGTERM handling in web apps looks different depending on the runtime:

• In Node.js, you register a process.on('SIGTERM', ...) handler that calls server.close(), which stops accepting new connections but lets existing ones finish. You need to track active connections yourself if you want a hard deadline.
• In Go, the standard pattern is passing a context derived from signal.NotifyContext into http.Server.Shutdown(), which handles the drain natively.
• In Java Spring Boot, configuring server.shutdown=graceful and spring.lifecycle.timeout-per-shutdown-phase does most of the work, but you still need to verify that long-running request threads actually respect the interrupt.
• In .NET, IHostApplicationLifetime.ApplicationStopping gives you a cancellation token, and Kestrel will drain in-flight requests if configured correctly — but middleware that spawns background work can easily outlive the drain window if you are not deliberate about it.

The key principle across all of these: your application needs to participate in its own shutdown. The orchestrator cannot do it for you.

The PreStop Hook: Buying Time for Endpoint Propagation

Even if your application handles SIGTERM correctly, you still have the propagation-delay problem. The load balancer may keep sending traffic for several seconds after the pod begins terminating.

The most reliable fix is a preStop lifecycle hook that introduces a deliberate delay before the SIGTERM is delivered to your container. A simple sleep 5 in the preStop hook gives the endpoints controller, kube-proxy, and your ingress controller time to remove the pod from their routing tables before your application begins shutting down.

This is not a hack. It is the recommended pattern in the Kubernetes documentation and in every serious SRE team I have worked with over fifteen years of managing production infrastructure. The sequence becomes:

1. Pod enters Terminating state.
2. Endpoints controller begins removing the pod from Service endpoints.
3. PreStop hook runs: sleep 5 (pod is still running and serving traffic during this window, but new traffic is being redirected away).
4. After preStop completes, SIGTERM is sent to the container.
5. Application begins graceful shutdown — stops accepting new connections, drains in-flight requests.
6. Application exits.

The five-second sleep is not magic. In environments with external load balancers (AWS ALB, GCP HTTPS LB), propagation can take longer. You may need to tune the delay based on observed behavior in your specific environment. The point is to decouple the two timelines: endpoint removal and process shutdown.

Connection Draining at the Load Balancer

Connection draining is the load balancer's side of the same problem. When a backend is marked unhealthy or deregistered, the load balancer needs to stop sending new requests to it while allowing in-flight requests to complete.

Most managed load balancers support this, but the defaults vary and the interaction with Kubernetes health checks matters:

• AWS ALB/NLB: Deregistration delay (default 300 seconds) controls how long the load balancer waits before forcefully closing connections to a deregistered target. For short-lived HTTP requests, this is usually far longer than needed. For WebSocket connections, it may be too short. Right-size it to your actual request duration profile.
• GCP HTTPS Load Balancer: Connection draining timeout is configurable per backend service. The default is 300 seconds.
• NGINX Ingress Controller (in-cluster): The controller watches endpoint changes and updates its upstream configuration. The worker-shutdown-timeout annotation controls how long NGINX workers continue processing requests after a configuration reload.

The critical integration point: your Kubernetes readiness probe must fail promptly when the pod begins shutting down. If your readiness probe keeps returning 200 during the preStop sleep, the ingress controller has no signal to remove the pod from its upstream pool. One effective pattern is to have your SIGTERM handler set an internal flag that causes the readiness endpoint to return 503, but do this after the preStop delay — not before, so you do not prematurely drop traffic during the propagation window.

Putting It All Together: A Checklist

Zero downtime deployment best practices are not a single configuration change. They are a chain of behaviors that must be consistent across your application code, your pod spec, and your load balancer configuration. Here is what the full chain looks like:

1. Application-level SIGTERM handler: Stops accepting new connections, drains in-flight requests, exits cleanly within a bounded deadline.
2. PreStop hook: A short sleep (typically 5-15 seconds) to allow endpoint propagation before SIGTERM delivery.
3. terminationGracePeriodSeconds: Set this to preStop delay + maximum expected drain time + a small buffer. If your preStop sleeps for 5 seconds and your longest request takes 30 seconds, set this to at least 40.
4. Readiness probe: Returns failure once the application has committed to shutting down, signaling the Service and ingress controller to stop routing.
5. Load balancer connection draining: Configured with a timeout that matches your actual request duration profile — not the default 300 seconds, and not zero.
6. Rolling update strategy: maxUnavailable: 0 ensures at least full capacity is available throughout the deploy. maxSurge: 1 (or higher) controls how many new pods spin up ahead of old ones being terminated.

Every link in this chain matters. A correct SIGTERM handler with no preStop hook still drops requests during the propagation window. A correct preStop hook with an application that immediately exits on SIGTERM still resets in-flight connections. A correct application and pod spec with a load balancer that does not drain connections still produces 502s at the edge.

Observability: Proving It Actually Works

The frustrating thing about dropped requests during deploys is that they are easy to miss. A handful of 502s in a deploy window do not always trigger alerts, especially if your error budget is generous or your monitoring is aggregated at one-minute intervals.

To verify your graceful shutdown in Kubernetes is working, you need to observe it directly:

• Deploy-time error rate correlation: Overlay your deploy events on your error rate dashboard. If you see a spike — even a small one — at every deploy, you have a shutdown sequencing problem.
• Pod termination logs: Log the SIGTERM receipt, the start of the drain period, the number of in-flight requests at shutdown, and the clean exit. If you do not log it, you cannot debug it.
• Synthetic traffic during deploys: Run a steady stream of requests against the service during a rolling deploy and count failures. This is the most direct test.

An observability stack that surfaces these signals — metrics, structured logs, deploy-event annotations — is what turns a rolling deploy from an act of faith into a verifiable operation. This is the kind of foundational soil work that pays for itself on every subsequent deploy.

The Deeper Pattern

Graceful shutdown is really about a principle that applies across all of infrastructure engineering: the system and the application have to cooperate. Kubernetes can orchestrate pod lifecycle. Your load balancer can manage connection routing. But neither can reach inside your application and tell it how to finish what it started. That responsibility belongs to the code.

This is why rolling deploy dropped requests remain so common even in teams that have adopted Kubernetes and CI/CD pipelines. The platform gives you the tools for zero-downtime deployment, but the last mile — SIGTERM handling, drain logic, health check coordination — is application work that has to be designed and tested deliberately.

It is also why having an SRE perspective alongside software architecture matters. The deploy pipeline, the pod spec, and the application code are not three separate concerns. They are one system, and the seams between them are where requests get dropped.

Related service: Cloud infrastructure and DevOps services in Tampa Bay — what we build, how we work, and what it costs.

When the Foundation Needs an Architect

If your deploys are producing intermittent 502s that nobody can quite pin down, or if your team is scaling a Kubernetes environment that was stood up quickly and never hardened for production traffic, these are exactly the kinds of foundational problems FigTree Development is built to solve. We architect infrastructure that holds under real pressure — not just on the first deploy, but on every deploy after that.

If this post described a problem you recognize, book a free 20-minute discovery call with us. We will walk through your current deploy pipeline and shutdown behavior and identify where the gaps are — no pitch deck, just a grounded conversation about what your infrastructure needs to scale without cracking.