It is 2:47 AM on a Saturday. Your phone vibrates. Then it vibrates again. By the time you are awake enough to read the screen, three alerts have stacked up: your application is returning connection errors, users cannot reach the site, and the on-call channel is lighting up. You pull up the browser, click through the security warning, and there it is — the TLS certificate expired eleven minutes ago.
No code change caused this. No infrastructure failure triggered it. Someone provisioned a certificate months ago, maybe a year ago, and nobody built the mechanism to renew it before it lapsed. The fix itself takes minutes. The trust it erodes — from your team, your users, your own confidence in the system — takes longer to rebuild.
This scenario is not rare. It is one of the most common preventable outages in production systems. And the root cause is almost never technical complexity. It is that certificate lifecycle management was treated as a one-time task instead of an automated, observable process.
Why TLS Certificates Still Expire in Production
The mechanics of TLS certificate expiry are well understood. A certificate has a validity window. When that window closes, browsers and clients reject the connection. The question worth asking is not how certificates expire, but why teams with otherwise solid infrastructure still get bitten.
Three patterns show up repeatedly:
Manual Provisioning Without Lifecycle Tracking
Someone generates a certificate, installs it on a load balancer or ingress controller, and moves on. There is no automation wrapping that certificate. No renewal job. Sometimes not even a calendar reminder. The certificate becomes a piece of invisible infrastructure — it works silently until it does not.
Automation That Was Built but Never Verified
A renewal script exists, maybe a cron job or a CI step, but nobody has confirmed it actually runs. The job silently failed three months ago when a DNS provider changed its API. Or the credentials it uses to solve the ACME challenge rotated, and the renewal process lost access. Automation without observability is just a more sophisticated way to be surprised.
Scattered Certificate Management Across Environments
One certificate lives in AWS Certificate Manager. Another was issued manually through a CA portal. A third was provisioned by a contractor who left. There is no single inventory, no unified renewal process, and no way to answer the question: which certificates do we have, and when do they expire?
Each of these patterns shares a deeper problem. The certificate was treated as a static artifact instead of a living piece of infrastructure that requires the same automation, version control, and monitoring as everything else in the stack.
The Architecture of Automated TLS Renewal
Solving this well means designing a system, not writing a script. The goal is a certificate lifecycle that is fully automated, observable, and recoverable — one where renewal happens silently in the background, and the only time a human hears about it is when something deviates from the expected path.
cert-manager on Kubernetes: The Foundational Layer
For teams running workloads on Kubernetes, cert-manager is the grounded, battle-tested tool for this job. It is a native Kubernetes controller that watches for Certificate resources, issues them through configured providers (most commonly Let's Encrypt via the ACME protocol), and handles renewal automatically before expiry.
The architecture is declarative. You define a Certificate custom resource specifying the domain, the secret where the cert should be stored, and the issuer to use. cert-manager takes care of the rest: creating the ACME order, solving the challenge (HTTP-01 or DNS-01), storing the issued certificate as a Kubernetes Secret, and renewing it when the renewal window opens — by default, 30 days before expiry.
This is infrastructure as code applied to certificate management. The certificate definition lives in version control alongside your other manifests. It is reproducible, auditable, and not dependent on any single person remembering to act.
Let's Encrypt Automation and ACME Challenge Strategies
Let's Encrypt issues certificates with a 90-day validity window, deliberately short to encourage automation. This design choice is a feature, not a burden — it forces teams to build the renewal pipeline rather than rely on a comfortable 12-month buffer that breeds complacency.
The ACME protocol supports two primary challenge types, and choosing correctly matters:
HTTP-01 challenges require your server to respond to a specific HTTP request on port 80 for the domain being validated. This works well for publicly accessible services where your ingress controller can serve the challenge response directly. cert-manager handles this natively when paired with an ingress controller like NGINX or Traefik.
DNS-01 challenges require creating a specific TXT record in your domain's DNS zone. This is the right choice for wildcard certificates, for services not directly exposed on port 80, or for environments where DNS validation is operationally simpler. cert-manager integrates with major DNS providers — Route 53, Cloud DNS, Cloudflare, and others — to create and clean up challenge records automatically.
The trade-off is real. HTTP-01 is simpler to configure but limited to single-domain, publicly reachable endpoints. DNS-01 is more flexible but requires granting cert-manager credentials to modify DNS records, which introduces an IAM surface that needs to be scoped tightly using least-privilege policies. Granting broad DNS write access to a certificate controller is trading one risk for another.
Beyond Kubernetes: Automation Patterns for Other Environments
Not every environment runs Kubernetes. The principle of automated TLS renewal still applies — only the tooling changes.
For workloads behind AWS load balancers, AWS Certificate Manager handles issuance and renewal automatically for certificates attached to ALBs, NLBs, and CloudFront distributions. The key constraint: ACM certificates cannot be exported. They work only with AWS services. If your architecture requires the certificate on a non-AWS endpoint, ACM alone will not solve it.
For traditional VM-based deployments, tools like certbot (the original Let's Encrypt client) can automate renewal via systemd timers or cron, with deploy hooks that reload the web server after a new certificate is written. The same observability requirements apply: if certbot fails silently, you are back to the Saturday morning page.
For multi-cloud or hybrid architectures, the challenge multiplies. Each environment may need its own renewal mechanism, and the certificate inventory spans providers. This is where a unified approach to infrastructure as code — defining certificate requirements in Terraform or a similar declarative tool, with consistent monitoring across all environments — becomes essential rather than aspirational.
TLS Certificate Monitoring and Alerting: The Layer Most Teams Skip
Automating renewal solves the common case. Monitoring solves the failure case. And the failure case is the one that pages you at 2:47 AM.
A well-engineered observability stack for certificate lifecycle includes three layers:
Certificate Expiry Metrics
cert-manager exposes Prometheus metrics out of the box, including certmanager_certificate_expiration_timestamp_seconds. This metric tells you exactly when each managed certificate expires. From this single metric, you can build alerts at multiple thresholds: warn at 21 days, escalate at 7 days, page at 3 days.
For certificates not managed by cert-manager, external probes — tools like blackbox_exporter or purpose-built certificate checkers — can connect to your endpoints, read the presented certificate, and export the expiry timestamp as a metric. This catches certificates that exist outside your automation entirely: the one a vendor installed, the one on the legacy subdomain, the one nobody remembers provisioning.
Renewal Failure Alerts
Knowing a certificate will expire is necessary but not sufficient. You also need to know when a renewal attempt fails. cert-manager emits Kubernetes events and updates the Certificate resource's status conditions when an ACME order fails, a challenge times out, or an issuer is misconfigured. These signals should flow into your alerting pipeline.
A common failure mode: the DNS credentials cert-manager uses to solve DNS-01 challenges get rotated during a routine security review, and nobody updates the cert-manager secret. The certificate is still valid for another 60 days, so nothing breaks immediately. But the renewal attempt at the 60-day mark fails silently. Then at the 30-day mark. Then at the 7-day mark. Without renewal failure alerting, the first signal is the outage itself.
Certificate Inventory and Drift Detection
The third layer is knowing what you have. A certificate inventory — whether built from cert-manager CRDs, Terraform state, cloud provider APIs, or a combination — gives you the ability to answer the question every SRE should be able to answer in under a minute: how many certificates does this organization manage, where are they, and when does each one expire?
Drift detection adds a layer of accountability. If someone provisions a certificate manually — outside the automated pipeline — the drift check surfaces it before it becomes the next forgotten certificate waiting to expire unnoticed.
Building the System Right: A Practical Sequence
If you are starting from a patchwork of manually managed certificates, here is a grounded sequence for building toward full automation:
1. Inventory first. Before automating anything, catalog every certificate in your environment. Check load balancers, ingress controllers, API gateways, CDNs, mail servers. You cannot automate what you have not found.
2. Centralize issuance. Choose a primary issuance path — cert-manager for Kubernetes workloads, ACM for AWS-native services, certbot for traditional VMs — and migrate certificates to it methodically. Resist the temptation to automate everything simultaneously. Move one domain at a time, verify renewal works, then move the next.
3. Instrument the pipeline. Export expiry timestamps and renewal status as metrics. Build alerts at multiple thresholds. Test the alerts by temporarily lowering the threshold or simulating a renewal failure in a staging environment.
4. Harden the credentials. Review the IAM policies and secrets that your renewal automation depends on. Scope DNS write access to the minimum zones required. Store credentials in a secrets manager, not in a plaintext config map. Ensure credential rotation processes account for cert-manager's dependencies.
5. Document and review. Write a runbook for certificate renewal failures. Include the steps to manually renew a certificate when automation fails, because it will fail eventually. Review the certificate inventory quarterly — not because you distrust the automation, but because new services, new domains, and new integrations introduce new certificates that may not be covered.
This is soil work. It is not glamorous. It does not produce a visible feature. But it is the foundation that keeps your application reachable, your users' connections encrypted, and your weekends uninterrupted.
The Real Cost of Getting This Wrong
A certificate expiry outage is not just downtime. It is a specific kind of downtime that signals to anyone paying attention — users, partners, security auditors — that the foundational infrastructure is not being maintained. Browsers display aggressive, trust-destroying warnings. API integrations fail hard. HSTS-pinned domains become completely unreachable until the certificate is replaced.
The remedy is almost always fast. The reputational and operational cost lingers. And the irony is that the automation to prevent it is neither complex nor expensive. It requires intention, not invention.
Related service: Cloud infrastructure and DevOps services in Tampa Bay — what we build, how we work, and what it costs.
Infrastructure That Does Not Crack Under Pressure
Certificate lifecycle automation is one piece of a broader principle: foundational infrastructure should be designed to renew, heal, and alert without human intervention in the happy path, and surface clear, actionable signals in the unhappy path.
At FigTree Development, this is the kind of infrastructure architecture we build. From automated pipelines and container orchestration on Kubernetes to observability stacks that surface problems before users notice them, we design systems that are scalable from day one and engineered to hold under real pressure. Our own infrastructure runs on this philosophy — a self-hosted Kubernetes cluster with GitOps, automated pipelines, and the same certificate automation principles described in this post.
If your team is managing certificates manually, waking up to preventable outages, or inheriting infrastructure where nobody is sure what is automated and what is not, that is exactly the kind of problem a focused conversation can untangle. Book a free 20-minute discovery call and let us look at what is under the hood together.