The air in the server room hums, a low thrumming that usually signals diligent computation. But beneath the veneer of this digital symphony, a quiet rebellion is brewing—one that challenges the very foundations of how we build AI agents.
I remember it all too well. Weeks spent wrestling with Pinecone, finessing embedding models, painstakingly chunking terabytes of data, all to build a support agent. And what did it do? Most of the time, it still ended up hitting the app’s own SQL database with a plain old SELECT query. The vector store, that shiny centerpiece of the RAG stack, felt more like an expensive paperweight. Tearing it out, replacing it with a simple index file and a directory of markdown notes that the agent could load on demand—that’s when things finally clicked. Same results, exponentially fewer moving parts.
Look, RAG isn’t dead. Hamel Husain correctly batted away that notion last year. What’s shifting, dramatically, is which retrieval mechanism you should default to. If you’ve been tinkering with Claude Code or Cursor, you’ve already been using this simpler, file-based memory pattern without even realizing it.
Open up any recent tutorial on building an AI agent, and you’ll see the same predictable dance: pick a vector database (ChromaDB, pgvector, you name it), set up an embedding pipeline, chunk your documents, write the retrieval logic, add a reranker, then feed the top-k results into the model. Each of those steps represents a complex system to manage, maintain, and, critically, pay for.
This whole elaborate setup made a certain kind of sense back when frontier models were capped at 8K or 32K context windows and function calling was still a novelty. But in 2026? With Claude Sonnet 4.6 boasting a staggering 1 million token context window and universal function calling, that 2023-era stack feels like bringing a bazooka to a knife fight. Most SaaS data already lives in a structured database; agents are supposed to reach it via precise tool calls, not fuzzy similarity searches. Trying to force it through a vector store is, frankly, over-engineering for most of today’s tasks.
So, before you dive headfirst into the RAG deep end, let’s name the scenarios where it actually makes sense. Because they’re real, and they’re important.
When Does the Full RAG Stack Actually Win?
There are indeed corners of the AI agent universe where the complexity of RAG is justified. If your use case hits these marks, then yes, build the full pipeline. The rest of us can probably take a shortcut.
Large unstructured corpora. When your agent needs to sift through tens of thousands of documents—think sprawling product manuals, ancient legal archives, or massive internal wikis—where titles are often unhelpful and exact-match lookups miss the point, similarity search is your best bet. Trying to list every single document in a massive index simply won’t fit into context windows anymore, and keyword searches will fall short.
Regulated, multi-tenant isolation. For SaaS applications in highly sensitive sectors like healthcare, finance, or defense, where strict per-tenant data boundaries are non-negotiable, vector databases often provide strong, out-of-the-box row-level access controls and audit trails. While filesystem memory can achieve this, you’ll be building those primitives yourself, a significant undertaking.
Frequently-refreshed external knowledge. If your agent needs to stay on top of rapidly changing information—hourly news feeds, volatile market data, evolving regulatory landscapes—a vector index’s incremental update capabilities are invaluable. Filesystem memory can become stale quickly unless you build out a similar incremental update mechanism.
Agentic search over structured tool responses. Jason Liu cuts right to the chase: “Good search is the ceiling on your RAG quality. If recall is poor, no prompt engineering or model upgrade will save you.” When an agent needs to reason across thousands of structured records and intelligently decide what to ask next, a true retrieval infrastructure with faceted metadata becomes essential.
If your use case aligns with any of these, then by all means, invest in the RAG stack. For everyone else, let’s talk about what’s next.
The Simpler Path: File-Based Memory and Tool Calls
The vast majority of SaaS agents operate on your own structured data: user accounts, order histories, support tickets, audit logs. You don’t need fuzzy similarity search to find a specific user record; you need a tool call that executes SELECT * FROM users WHERE id = ?. And here’s why tool calls crush vector retrieval for these scenarios:
- Precision: Tool calls deliver precise, structured records that models handle far more reliably than raw chunks of prose. There’s no ambiguity.
- Freshness: You get real-time data the moment it’s written. No more waiting for an embedding pipeline to re-run.
- Integration: You use your existing database’s built-in access controls, transaction management, and audit trails. Why build a parallel system when your existing one is already battle-tested?
And for the contextual information that isn’t sitting in your main database—system instructions, established conventions, accumulated insights about a specific user, summaries of past conversations, or your product documentation—the game has changed. With a 1 million token context window, you can carry an astonishing amount of state directly within the prompt. The need to offload this to a separate retrieval system diminishes drastically.
Think about Claude Code’s internal architecture: it’s built around an index file, MEMORY.md, and then specific markdown files that are loaded only when needed. This isn’t some fringe experiment; this is a production-ready pattern that scales. For many SaaS agents, this approach—combining strong tool calls for structured data with file-based memory for unstructured context—is the sweet spot. It’s faster, cheaper, and significantly simpler to implement and maintain.
We’re at a crossroads. The default thinking around AI agents has been heavily influenced by the constraints of older models and less capable tooling. Now, with the leap in context window sizes and the ubiquity of sophisticated tool-calling capabilities, we have the opportunity to simplify our architectures dramatically. The question isn’t whether RAG is effective; it’s whether it’s the right tool for the job, every time. For most SaaS agents, the answer is increasingly becoming a resounding ‘no’.
A Blast from the Past, A Glimpse of the Future
It’s almost poetic, isn’t it? We spent years optimizing for distributed systems, for microservices, for abstracting away underlying infrastructure. Now, with AI agents, we’re finding that sometimes, the most direct path—a well-structured file, a direct database query—is the most performant and efficient. It’s a reminder that technological progress often involves a cyclical return to fundamental principles, albeit with vastly more powerful tools.
This shift isn’t just about cutting costs or reducing engineering overhead, though those are certainly compelling benefits. It’s about building more responsive, more accurate, and ultimately, more useful AI agents. Agents that don’t get bogged down in the complexities of vector similarity but instead use their intelligence directly where the data—and the user’s intent—resides.
**
🧬 Related Insights
- Read more: Dockerizing React: Nginx Slashes Size to 23MB
- Read more: Observability Explained: Logs, Metrics, and Traces for Modern Applications
Frequently Asked Questions**
Will this simplify my agent development? Yes, for many typical SaaS use cases, this approach significantly reduces complexity by eliminating the need for vector databases, embedding pipelines, and reranking systems.
When should I still use RAG? RAG remains the best choice for large, unstructured document sets, highly regulated multi-tenant environments requiring built-in isolation, and situations needing constant updates from frequently changing external knowledge sources.
How does Claude Code’s internal pattern work?
Claude Code uses a filesystem-based memory system with an index file (MEMORY.md) and on-demand loading of per-topic markdown files, avoiding traditional vector databases for its core functionality.
