Every experienced data engineer has seen this scene before. You open your BI dashboard first thing in the morning. The queries execute quickly. The visualizations render without errors. The platform is healthy. Then comes the disappointment. The numbers are from yesterday. Nothing is technically broken, yet the dashboard has already failed its purpose.
Most engineering discussions would describe this as a data freshness problem. I think that diagnosis misses something more fundamental. We often confuse how fast data is produced with how fast data becomes usable. These are not the same problem.
Suppose your operational database receives only a few hundred new transactions every hour. From a throughput perspective, the workload is trivial. The database is barely under pressure. Pipelines should have no difficulty keeping up. Yet the business may require those transactions to appear in a dashboard within five minutes. The challenge is no longer throughput. It is latency.
That distinction changes almost every architectural decision. One of the most common mistakes in modern data platforms is sizing the architecture according to data generation speed instead of data availability latency. Volume is visible. Latency is structural.
Imagine a PostgreSQL database feeding BigQuery.
The obvious solution may be to build a custom CDC service running inside a Docker container on a virtual machine. At first glance, the architecture appears inexpensive and under complete control. Until it isn’t.
What happens if the VM stops on a Saturday morning?
Who notices before the business notices?
How does the pipeline behave during an unexpected spike in writes?
How much engineering effort is spent maintaining infrastructure that was never part of the original business problem?
The opposite decision is equally interesting. Instead of operating your own CDC infrastructure, you adopt a managed platform such as Fivetran, Stitch, Informatica or Matillion. Maintenance decreases. Operational burden decreases. But new questions immediately appear.
How is pricing calculated?
Does a complete backfill count against monthly quotas?
Can the platform perform partial backfills?
How does it behave when source tables lack proper indexing?
How transparent is its compliance with financial and privacy regulations?
The architectural problem did not disappear. It simply changed shape.
Cloud-native services introduce another layer of decisions. Within Google Cloud alone, DataStream, Data Fusion and Dataflow each represent different assumptions about movement, transformation and orchestration.
Should transformations happen before ingestion? Or after data reaches BigQuery?
Should one replication stream feed multiple analytical datasets?
Should ingestion remain completely raw while transformations execute independently?
How much customization is actually necessary?
These questions are often treated as implementation details. I think they are architectural questions. Because none of them is fundamentally about moving data. They are about minimizing the time between reality and decision. Perhaps this is why latency deserves a more central place in platform architecture than it usually receives.
We measure rows per second. We benchmark query execution. We estimate storage growth. Yet we rarely begin architecture discussions with a much simpler question: How long can the organization tolerate waiting before this information becomes actionable? Everything else follows from that answer.
Maybe the most important characteristic of a modern data platform is not how much data it can process. Maybe it is how little time it allows information to remain trapped between the source system and the decision.
I have been exploring this question through a broader research idea called Minimum Context Signals. It starts from an uncomfortable observation: operational systems rarely fail because they lack information. More often, they fail because the right information arrives too late — or arrives surrounded by context that delays action.
Latency, then, is not simply a performance metric. It is a design philosophy. However, if two architectures produce exactly the same data, but one delivers it five minutes earlier, which one is actually more intelligent? Let me know what you think and how you would resolve this issue.
One question. One idea. Every week.