Every time I join a company that's already 2–3 years in, I see the same pattern.
The engineering team picked a stack — maybe it's event-driven, maybe it's a specific message queue, maybe they went all-in on a NoSQL database because it was hot on Hacker News. The product shipped. Investors were happy.
But now someone wants a dashboard. And suddenly we're trying to reverse-engineer events back into a relational model. The JSON blobs in production have no schema. The "real-time" streaming platform they chose doesn't actually play nice with batch processing.
That's the data pipeline trap.
The problem isn't the tools. It's when the tools were chosen.
Buzzword tech stacks are almost always selected for the product use case — low latency, high throughput, whatever the frontend needs. Nobody asks: "How will this data be consumed six months from now?"
By the time a data team enters the picture, the architecture is already baked. You end up duct-taping connectors, writing custom exporters, and paying for middleware that exists only to translate between incompatible systems. Operational overhead balloons. Costs creep up. And the data team spends 80% of its time on plumbing instead of insights.
The fix is simple: involve data earlier
When CTOs and founders evaluate a tech stack, they should ask one more question: "Can we export this data efficiently for analytics?"
Not "does it have an API" — everything has an API. But can we get clean, structured, timestamped data out without writing a custom pipeline? If the answer involves "we'll figure that out later," you're already digging the trap.
The best time to design your data infrastructure is before you write your first production query. The second best time is right now.
In the age of AI, this is no longer optional
Every company I talk to wants to "do AI." But AI runs on data — clean, documented, well-modeled data. If your data pipeline is a pile of workarounds, your AI initiatives will fail before they start.
You can't train on garbage. You can't get insights from a firehose of unstructured events that were never designed to be queried.
First principles for data-ready architecture
- Choose storage that supports both transactional and analytical access patterns. PostgreSQL with foreign data wrappers, or a dedicated OLAP store alongside your OLTP database. Don't force one system to do both.
- Define schemas early, even if you expect them to change. An imperfect schema you iterate on beats a schemaless blob you can never query.
- Give the data team a seat at the architecture table before the stack is set in stone. A 30-minute conversation about data access patterns can save months of pipeline work.
- Think about how data will be used, not just how it will be stored. Who will query it? What questions will they ask? What latency do they need? The answers should shape your infrastructure choices.
The companies that will win in the next five years are the ones that design their data infrastructure from day one — not as an afterthought, but as a first-class concern.
The data pipeline trap is expensive to escape. The best time to avoid it is before you fall in.