Somewhere around month three of a growing deployment, something strange happens to the bill. Compute costs climb at a pace that roughly matches traffic, which is expected. But the observability line item — the sum of log ingestion, metrics storage, and trace retention — starts outpacing everything else. By month six it is not unusual for logging and monitoring costs to exceed the compute they are watching.
This is not a theoretical risk. It is one of the most common patterns in cloud infrastructure, and it is one of the quietest. Nobody sets out to spend more on watching their servers than running them. But without a deliberate log retention strategy and a clear understanding of metrics cardinality, that is exactly what happens.
The good news: you can cut observability costs significantly — often 40 to 60 percent — without losing the visibility that keeps your systems reliable. The work is not about turning things off. It is about being intentional about what you collect, how long you keep it, and where you store it.
Why Observability Costs Grow Faster Than Infrastructure
Most teams start observability the same way: turn on default logging for every service, ship everything to a centralized platform, and set retention to 30 or 90 days because that was the default. This approach works fine when you have three services and modest traffic. It breaks down in predictable ways as you scale.
Log volume compounds, it does not grow linearly
Every new microservice, every new endpoint, every new integration generates logs. But the compounding effect comes from the interactions between services. A single user request that touches eight services generates log entries in each one. Add a retry mechanism and you double or triple the output for the same logical operation. Add verbose error logging with full stack traces and the bytes-per-event climb fast.
The result: doubling your service count can quadruple your log volume. Doubling your traffic on top of that produces another multiplier. Meanwhile your ingestion pricing is per-gigabyte, and it does not offer volume discounts at the scale most growing teams operate.
Metrics cardinality explosion is the hidden multiplier
Cardinality — the number of unique time series your metrics system tracks — is where observability costs go from uncomfortable to alarming. Every unique combination of label values creates a new time series. A metric like http_request_duration with labels for method, path, status code, and service version sounds reasonable until you calculate the combinations.
Five HTTP methods, 200 unique paths, 15 status codes, and 10 active service versions produce 150,000 unique time series from a single metric. Add a user-ID or request-ID label (a common mistake) and you are looking at millions. Most managed metrics platforms charge per active time series, and the bill reflects every label combination whether anyone ever queries it or not.
The Framework: Collect With Intent, Not by Default
Reducing observability costs is not about choosing between visibility and budget. It is about replacing the default-everything approach with a tiered, intentional architecture. Think of it as soil work — preparing the foundation so that what you grow on top of it is healthy and sustainable, not a tangle of roots choking each other out.
Tier 1: What must be real-time and searchable
This tier is small and expensive, and it should stay small. Real-time, fully indexed, instantly searchable logs and metrics belong here — but only for the signals that drive active alerting and on-call response. Error logs from production services. Key business transaction traces. The golden signals: latency, traffic, errors, and saturation for each critical service.
Everything in Tier 1 should have a short retention window — 7 to 14 days in most cases. If an incident is not detected and investigated within two weeks, the real-time searchability of those logs is no longer the thing providing value.
Tier 2: Queryable but not instant
This is where the bulk of useful observability data should land after its Tier 1 window expires. Move logs to cheaper object storage with a lightweight indexing layer on top. Tools like Grafana Loki or a well-architected S3-plus-Athena setup let you query historical logs when you need them without paying real-time indexing prices every second of every day.
Tier 2 retention can run 30 to 90 days depending on your compliance needs. The cost difference between Tier 1 and Tier 2 storage is often 10:1 or greater per gigabyte.
Tier 3: Archive for compliance and forensics
Some logs need to exist for a year or more — audit trails, authentication events, data access records. These belong in cold storage: S3 Glacier, GCS Archive, or equivalent. They are not queryable in real time. They exist for the day you need to prove something happened or did not happen. The cost per gigabyte at this tier is negligible compared to Tiers 1 and 2.
Five Concrete Moves That Reduce Log Ingestion Costs
1. Drop debug and trace-level logs before they leave the node
If your application emits debug-level logs in production (and most do, because someone set the log level during development and never changed it), those logs are being shipped, ingested, indexed, and stored at full price. Configure your log pipeline — whether it is Fluentd, Fluent Bit, Vector, or an OpenTelemetry Collector — to filter at the source. Drop debug and trace levels before the data ever hits the network.
This single change regularly cuts log volume by 30 to 50 percent with zero loss of operational visibility.
2. Sample verbose but low-signal logs
Health check endpoints, keep-alive pings, and successful authentication events generate enormous volume and almost never surface in an investigation. Rather than logging every instance, sample them: keep one in every hundred, or one per minute, whichever is lower. You preserve the ability to detect anomalies in the pattern without paying for millions of identical log lines.
3. Enforce structured logging and strip bloat
A single unstructured log line with a full stack trace, a serialized request body, and three layers of nested exception context can weigh 10 to 50 KB. Multiply that by thousands of errors per minute during an incident and you are ingesting gigabytes of data that nobody will read past the first three lines of.
Structured logging — JSON with defined fields — lets you control exactly what ships. Log the exception type, the message, and the top frame of the stack. Store the full trace locally or in a sampling-based trace backend. The investigation workflow improves because the data is cleaner, and the ingestion cost drops because the payloads are smaller.
4. Tame cardinality at the metric definition layer
Audit your metric labels quarterly. Remove any label with unbounded values — user IDs, request IDs, full URL paths with query parameters. Replace high-cardinality path labels with normalized route patterns (/api/users/{id} instead of /api/users/48291). Set cardinality limits in your metrics pipeline so that a single misconfigured service cannot create millions of time series overnight.
This is not just a cost exercise. A metrics cardinality explosion degrades query performance for everyone on the team, making dashboards slow and alerts unreliable exactly when they matter most.
5. Separate the retention policy from the collection policy
Most teams treat these as one decision: collect everything, keep it for 30 days. Decoupling them is where the real savings live. You might collect verbose application logs for debugging but retain them for only 3 days. You might collect audit logs at lower volume but retain them for 365 days. Each log stream should have its own retention window based on its purpose, not a single global default applied to everything.
The Observability Stack That Scales With You
The architecture that holds under real pressure is not the one with the most data. It is the one where every signal has a purpose, every retention window is intentional, and the cost model is predictable as traffic grows.
Infrastructure as Code is foundational here. Your log routing rules, retention policies, sampling rates, and alerting thresholds should all be declarative and version-controlled. When a new service launches, it inherits the right observability configuration automatically — no manual setup, no forgotten defaults silently inflating the bill for months.
This is the kind of work I have done across enterprise environments managing 16 cloud accounts and budgets above $23 million. The pattern is always the same: observability costs that nobody questioned for a year turn out to be one of the largest optimization opportunities in the entire cloud bill. And the fix is never about flying blind. It is about engineering the right level of visibility for each layer of the system.
When to Worry: Signs Your Observability Costs Need Attention
A few signals that the quiet creep has become a real problem:
- Your observability line item is growing faster month-over-month than your compute or traffic.
- Dashboard queries are slow or timing out — a symptom of cardinality problems in your metrics backend.
- Nobody on the team can explain what a specific log stream is used for, but it ships tens of gigabytes per day.
- You have a single global retention policy applied identically to debug logs and audit trails.
- Your monthly ingestion bill from a managed observability vendor exceeds what a dedicated engineer would cost.
Any one of these is worth investigating. Two or more together suggest the observability architecture needs a deliberate redesign, not incremental tweaks.
Related service: Cloud infrastructure and DevOps services in Tampa Bay — what we build, how we work, and what it costs.
Build the Roots Before the Bill Forces Your Hand
Observability cost optimization is not a one-time cleanup. It is an architectural discipline — a set of decisions about what to collect, where to store it, and how long to keep it, revisited as your systems grow. Done well, it produces better operational visibility at a fraction of the cost. Done reactively, under budget pressure, it usually results in cutting the wrong signals and discovering the gap during the next incident.
The time to design this intentionally is before the bill forces a rushed decision.
At FigTree Development, we help engineering teams architect observability stacks that are scalable from day one — grounded in real operational experience, not vendor defaults. If your logging costs are climbing and you are not sure what is driving them, or if you are building a new platform and want to get the foundations right the first time, we would welcome the conversation.
Book a free 20-minute discovery call and let us walk through your observability architecture together. No sales pitch — just an honest look at where the waste is hiding and what a right-sized approach could look like for your environment.