Every engineering team has a version of the same story. Logs are often messy with inconsistent fields. Some services dump structured JSON while others emit freeform strings that made sense to the developer who wrote them and no one else. If you’re lucky, there’s a plan to clean it up, but the project usually remains on the backburner, and cleaning up data after the task is often a huge challenge.
Agents aren’t going to wait.
Agents occupy both ends of this problem. They write the code that produces the logs, and they are the ones who will have to read them. As agentic development accelerates, the volume and variety of log data will grow faster than any cleanup effort can keep pace with. The mess isn’t going away. It’s compounding.
The question isn’t how to stop that from happening. It’s how to build a data layer that can handle the mess you have today and incrementally make it more useful, without stalling the systems that depend on it.
Do the fast work at ingest, the smart work behind it
When a human engineer investigates a production issue, they bring context. They know roughly what they’re looking for, they recognize familiar patterns, and they can mentally filter noise. An agent doesn’t automatically have that context. It queries, interprets, and acts, and the quality of its output is a direct function not just of the skills and context you build but also of the quality of the data it’s reasoning against.
Unstructured logs don’t just slow queries down. They degrade the agent’s reasoning. A field named C1 containing a UUID tells an agent nothing. That same field, understood to be a media ID with known access patterns and a known relationship to downstream tables, provides the necessary context for the agent. The difference between those two states isn’t the data. It’s the context that’s provided.
This is already a problem for human operators. At the throughput levels that agentic systems will generate, it becomes a reliability problem. An agent that has to wait for accurate query results before acting in production will make decisions on stale information. An agent reasoning over data it doesn’t understand or that’s incomplete is an agent that will be confidently wrong, and only engineers experienced with the underlying data will be able to tell the difference.
The instinct, at this point, is usually to impose more structure earlier. Enforce schema at ingest. Get developers to clean up their logs before they ship. And this is part of the solution, but on its own isn’t a panacea.
Impose structure at ingest and adapt over time
The appeal of schema-on-write is obvious. If you know what the data looks like before it arrives, you can store it efficiently, query it fast, and reason over it reliably. Where possible, standardizing and normalizing data is a best practice, and many modern solutions provide fast streaming ETL pipelines that make transformation at ingest possible.
However, you don’t always know what the data looks like before it arrives, and in an agentic development environment, where code is being generated and deployed faster than any schema review process can track, you increasingly can’t.
There’s a deeper problem, too. The goal of schema-on-write is to make the data useful at query time. But the intelligence required to make data truly useful, such as understanding that a string is not just a UUID but an AWS ARN with specific access patterns, or that a log field or that a log line is one of a few dozen shapes an OpenSSH server ever emits, often isn’t available at ingest time. That intelligence accumulates. You learn it from the data over time, not before the data arrives.
Trying to front-load that intelligence doesn’t just create friction for producers. It puts the smart work on the path where speed matters most. That’s the wrong place for it. You should standardize at ingest where possible, but remain flexible and ready to add context as data changes over time.
Stateless ingest is the foundation, not a limitation
The systems that handle extreme-scale data workloads well share a design principle: ingest pipelines are fast and stateless. They should be capable of real-time streaming ETL, use massive parallelism to scale up and down depending on data needs, and prepare data as much as possible for fast writes and reads.
When ingest is stateless, it can absorb a firehose of data without slowdowns or timeouts. Partition timestamps can overlap, and late-arriving data isn’t a failure condition. The system doesn’t need to hold state in order to accept the next row, which means throughput scales without the overhead that stateful ingest would require.
Think of it as a receptacle that accepts exactly the shapes it was built for, and is made of something that won’t deform under pressure. The shapes it accepts are simple. The volume it can handle is not. That combination is what lets a system serve real-time queries against petabyte-scale ingest under real-world conditions.
The enrichment layer that doesn’t slow anything down
That somewhere else is an asynchronous enrichment layer, decoupled entirely from the ingest path.
Because it runs behind the scenes, it can afford to be slower, thorough, and occasionally wrong in ways that get corrected later. It doesn’t have to make decisions in milliseconds. It can look at data after the fact, apply statistical and information-theoretic methods to find patterns, and progressively build up an understanding of what the data is and how it’s accessed.
The output of that process isn’t just metadata. It informs storage. Once you know that a field is a UUID used as a high-cardinality identifier, you can apply a bloom filter to it. Or if you recognize that a log field matches a small set of recurring templates, such as the way user agent strings, despite their apparent variety, resolve to a few thousand distinct patterns, you can store the patterns in highly-compressed summary tables. If you know the access patterns for a dataset, you can build materialized aggregates that make the common queries fast without touching the raw data.
The LLM involved in this layer doesn’t read every row. At petabyte scale, that’s not a viable approach. It reads what the enrichment process has surfaced as worthy of attention, such as patterns and anomalies. It brings domain knowledge to that sample that no purely statistical method has. It knows that an OpenSSH log has a set of known templates. It knows how fields in a security log tend to be queried in practice. That knowledge shapes how the data gets stored and labeled, which shapes how fast it can be queried and ultimately determines how reliably an agent can reason over it.
The key constraint is that none of this touches the hot path. The moment any of the smart work moves to ingest, you’ve traded the thing that makes the system fast for the thing that makes it smart. You need both. The architecture keeps them separate.
Every step toward structure pays before you reach the end
The progression from unstructured to structured isn’t a cliff you fall off if you don’t reach the top. It’s a spectrum with returns at every point along the way.
A blob you can’t query at all becomes useful the moment you know it’s JSON. JSON becomes more useful when you’ve established data types for its fields. Typed fields become more useful when you’ve added semantic understanding. For example, once you know a string is an AWS ARN, or a request path with a few thousand recurring shapes. Semantic context makes fields more useful because the system knows how they should be accessed and has organized storage accordingly.
Each of those transitions is a step that a system can take incrementally, asynchronously, and without requiring the data producer to change anything. Each one makes queries faster, storage cheaper, and agent reasoning more reliable. None of them requires completing the journey before you start getting value from it.
There’s also a compounding effect that matters for cost. Storage cost and query cost aren’t the same. A system that accepts everything and structures nothing charges you at query time for the work it didn’t do at enrichment time. Volume pushes both costs up. Structure pushes them down. A system that adds structure over time is the one whose cost curve bends the right way as volume grows. The system that can support data along the spectrum, even partway, will have costs and performance that scale more favorably than the one that treats every byte the same.
There are also benefits when it comes to data hygiene. As code changes, new fields appear and old ones disappear. An enrichment layer that tracks the velocity of structure change can flag when a field that was reliably sending integers starts sending floats instead, or when a known template starts producing anomalies. The system should be able to maintain semantic hygiene on a dataset that’s always in motion.
Every step pays off along the journey
You don’t need to immediately have a fully structured, semantically rich, access-pattern-optimized data layer before agents start delivering value. The spectrum is the point. Each step along it makes agents faster, cheaper to run, and less likely to reason incorrectly.
The organizations that will get the most out of agentic systems aren’t necessarily the ones with the cleanest data today. They’re the ones that build a data layer that keeps every row, learns from the rows it holds, and improves how it stores and serves them over time. You cannot learn from what you threw away.
