Everyone is watching the observability bill. The real story is the computation behind it: telemetry that is collected, enriched, copied, indexed, and retained even though no alert, dashboard, investigation, or audit ever uses it.
That waste is usually discussed as a purchasing problem. It is an architecture problem. When every byte receives premium treatment, cost grows with traffic while decision quality barely moves.
The winners will stop asking how much pipeline capacity they can afford. They will ask what evidence the business actually needs the pipeline to produce.
The Burning Platform: Every Unused Transformation Compounds
The Volume Multiplier. A pipeline receiving one million 1 KB events each second processes 86.4 TB every day before replication, enrichment, or indexing. A three-copy storage policy alone turns that into 259.2 TB of daily writes, even if only a fraction of fields support a decision.
The Computation Tax. Filtering after parsing, enrichment, routing, and indexing does not eliminate waste. It merely deletes data after the expensive work is complete. The OpenTelemetry sampling guidance notes that high-traffic systems can often use sampling rates of 1 percent or less while retaining a representative view, provided the policy protects important events.
The Reliability Bill. More stages create more failure combinations. A Salesforce engineering analysis shows that a 12-component pipeline built from individually 99.99 percent available components delivers about 99.88 percent end-to-end availability, or roughly 10.5 hours of annual unavailability.
The bottom line: pipeline cost is the accumulated price of architectural decisions about what to compute, copy, and preserve.
The New Playbook: Design Backwards From the Required Output
1. The Outcome Architect: Name the Decision Before the Data
Start with the output contract. Every alert, dashboard, investigation path, capacity model, and compliance control should name the fields, resolution, freshness, and retention it requires. Data with no downstream consumer is not future insight; it is unowned inventory.
This changes the design review. Instead of approving a firehose and negotiating its cost later, teams approve specific evidence products and the minimum telemetry needed to create them.
2. The Signal Accountant: Price Work Per Useful Answer
Infrastructure bills group compute, network, and storage into separate lines. Architecture must reconnect them to a unit of value: cost per actionable alert, completed investigation, SLO calculation, or retained audit record.
Track bytes accepted, bytes parsed, attributes added, records queried, and records retained for each output. The counterintuitive insight is that storage is often not the first waste. CPU and network spend can already be locked in by the time a low-value event reaches the retention gate.
3. The Edge Curator: Reduce Data Before Expensive Work Begins
Put deterministic filtering, redaction, aggregation, and routing as close to the source as reliability permits. Repeated health checks can become counters. High-volume success traces can be sampled. Errors, unusual latency, and business-critical transactions can retain full detail.
This is not indiscriminate deletion. It is computation placement. The pipeline should spend the least effort on the data least likely to influence an outcome.
4. The Retention Governor: Match Fidelity to Signal Half-Life
Telemetry does not retain equal value forever. Fine-grained traces may matter for hours, searchable logs for days, aggregated trends for months, and audit evidence for years.
Create explicit hot, warm, and archival contracts with different query expectations. Selective indexing matters as much as retention: fields used in 95 percent of investigations deserve fast paths, while rarely queried payloads can remain recoverable without being continuously indexed.
5. The Feedback Mechanic: Make Cost Controls Adaptive
Static sampling rules age badly. Traffic changes, new services appear, and incidents make formerly ordinary events important. Feed alert state, error rate, latency, service criticality, and query demand back into collection policy.
Define two SLOs together: an evidence SLO for the signals that must remain available, and a cost SLO for the work required to preserve them. A pipeline is healthy only when it meets both.
Case Studies in the Wild: Architecture Beats Blanket Capacity
Uber. Its M3 metrics architecture aggregated 500 million incoming metrics per second into 20 million persisted metrics per second, a 96 percent reduction before storage. The engineering account also describes retention and granularity policies ranging from seconds to years. The crucial lesson is that aggregation and retention belong in the data model, not in a quarterly cost-cutting exercise.
Capital One. Engineers faced more than one petabyte of tracing data per day without sampling. By building tail-based sampling infrastructure, the organisation reported a 70 percent reduction in trace volume. The crucial lesson is that seeing the completed trace before deciding what to retain can preserve rare failures while discarding routine repetition.
Google. Dapper made distributed tracing practical across thousands of machines by combining low-overhead instrumentation with sampling in common libraries. The published system design shows the crucial lesson: ubiquitous visibility does not require exhaustive collection when sampling is treated as a first-class architectural control.
The Action Plan: Rebuild the Cost Curve in 90 Days
Days 0-15: Map Value and Waste. List every downstream output and its owner. Measure volume and compute at each stage, then identify data that is transformed but never queried, alerted on, or retained for a requirement.
Days 16-45: Move One Decision Upstream. Choose a high-volume stream with a stable output contract. Aggregate, filter, or sample before its most expensive transformation, while preserving errors and rare events. Compare evidence quality, incident coverage, and cost per useful answer.
Days 46-90: Productise the Control Plane. Publish reusable policies for routing, sampling, indexing, and tiered retention. Add evidence and cost SLOs to service reviews. Scale only the patterns that prove savings without weakening operational confidence.
The Inevitable Future: Pipelines Will Be Judged by Decisions, Not Throughput
AI agents and automated operations will generate more telemetry and consume more evidence than human teams can inspect. That makes brute-force collection less defensible, not more.
The structural shift is from provisioning pipelines for maximum input to engineering them for required output. Capacity remains important, but it becomes a consequence of the evidence contract rather than the starting assumption. The most valuable pipeline is not the one that processes everything; it is the one that knows what deserves work.
