Moving to Agentic RAG | Our Agentic RAG Architecture that outperforms 2025 RAG in production

Chatbots and agentic ai systems with access to company defined knowledgebases and internal tools are continuing to grow in popularity, as users are finding out that popular models like ChatGPT and Claude are trained on public data up to a cutoff date and don’t have access to your internal documentation, databases, or continuously changing workflows. That creates a gap between what users ask and what the model can reliably answer. The result is familiar: outdated responses, missing details, or confident hallucinations when the model “fills in” unknowns.

‍

A question to GPT 5.4 about a specific product.

‍

‍At this point you probably know Retrieval-Augmented Generation (RAG) solves this by retrieving relevant information from your company’s knowledge sources at runtime and supplying it to the model as grounded context for generation. But RAG development projects fall apart all the time, either not working at all, or not being able to scale as the use case or the knowledge base grows. Most of the people we work with have at least tried a RAG build, but cannot achieve the results they want for a number of reasons. 

Our framework has been developed over 100s of iterations as the space has changed. We started building “RAG” (it wasn’t called that back then) in 2021 and focus on a few core principles for success that not only work out of the box but scale with the use cases. Every use case is a bit different, but if you understand the pieces of the pipeline that really drive accurate answers you can focus on the stuff that matters. 

‍

‍

What you’ll learn:

• RAG in 60 seconds: the old school basic pipeline and how it works
• The Width.ai Agentic Retrieval Augmented Generation Framework: the modules we use to make RAG reliable in production
• What we retrieve (and why): documents, external tools, guiding examples, system prompts, and successful past conversations
• Production considerations: routing, chunking, embeddings, vector DB choices, and tool/action retrieval with external data sources.
• The methods we use for complex tasks, multi agent collaboration, and multi agent orchestration
• Where it’s working today: examples from document summarization and customer-service chatbots

‍Let’s start with the standard RAG workflow. 

‍

What Is Retrieval-Augmented Generation?

Large language models (LLMs) are trained on large amounts of text from websites, books, research papers, and other such public content. The semantics and real-world knowledge present in all that text are dispersed across the weights of the LLM's neural network. This representation as network weights is called parametric memory.

But what if you want accurate answers based on your own private documents and data? Since they weren't included in the LLM static training data, you aren't likely to get the answers you want.

One option is to convert your private documents and data into a private dataset and finetune the LLM on it. But the problem is that you'll have to finetune the LLM frequently if new documents and data are being created all the time.

So you need an approach that can dynamically supply information to your LLM on demand. Retrieval-augmented generation is a solution that retrieves relevant information from your documents and data to supply to your LLM.

‍

The Original End-to-End Differentiable Approach

The original RAG approach, proposed in 2020 by Lewis et al., worked a bit differently than the modern approach described below. 

A key difference was that the entire RAG was a single complex model where every step was a differentiable part of the whole, including the vectorization and similarity search steps as shown below.

Differentiability meant that you could train the entire model on a language dataset and knowledge base. The weight adjustments to minimize the final loss could then be backpropagated to every step of the model. This effectively created custom functions for vectorization, retrieval, and text generation that were highly fine-tuned for relevance.

The benefits of this approach are:

• High relevance
• End-to-end trainability
• Replaceable knowledge base

The cons are:

• Limited knowledge base due to computation and memory constraints
• Usually requires fine-tuning
• The system focused on single pass information retrieval

‍

The Modern(ish) High Level RAG Pipeline

RAG pipelines are split into various models and tools to go from user query, to relevant context, to answer. The components include:

1. Knowledge base: The knowledge base (KB) is the information from which relevant sections are retrieved and provided to the LLM as inputs. Unlike the internal knowledge dispersed throughout the LLM's weights (parametric memory), the KB acts like external, long term memory that people can easily read, modify, and replace without retraining the LLM.
2. Data sources: Your KB can consist of a variety of data sources and formats, like unstructured documents, structured data in relational databases, application programming interfaces (APIs), or web content.
3. Large language model: The LLM acts based on the details provided in a prompt which includes a primary task or question, contextual information, and optional few-shot examples to demonstrate desired results. Any of these details may come from the KB.
4. Embeddings: Embeddings are vector representations of the information in the KB & the query. These are used to be able to do the comparison between the input query and relevant information in the KB. Usually these are smaller chunks of documents.
5. Vector database: Depending on the size of your KB, vectorization can take a long time and produce millions of embeddings that are difficult to search quickly. A vector database alleviates this by saving the embeddings on storage media for later reuse and efficiently searching millions of embeddings with fast similarity algorithms.

A typical modern RAG workflow with these components operates as follows:

1. Vectorize your KB: All the information in your KB is vectorized using an embedding model or API, and the generated embeddings are stored in the vector database. The embeddings may cover multiple levels of information organization, like sentences, paragraphs, sections, chapters, entire documents, database rows, subsets of database columns, and so on.
2. Set up your LLM: Typically, you just use a pre-trained LLM like GPT-5 or Claude. Some specialized domains or knowledge bases, you may have to fine-tune your LLM. 
3. Generate queries: More modern RAG systems generate augmented search queries based on the provided user query. In multi step ai systems (chatbots) this also allows you to to create search queries based on the entire conversation.
4. Create embeddings for the search queries: For each query, vectorize it using the same embedding model or API.
5. Retrieve information relevant to the prompts: Pass the prompt's embedding to the vector database. Using vector similarity algorithms, it finds information in the KB whose embeddings are semantically relevant to the prompt. The information may be useful contextual details, better task descriptions, detailed system prompts, or relevant few-shot examples.
6. Augment the prompt with the retrieved information: The retrieved context is combined with the original prompt according to LLM-specific prompt engineering rules. Sometimes, it may even replace the original prompt entirely. You may also augment system prompts this way.
7. Send the prompts to the LLM: Each augmented prompt consisting of a task and optional contextual information, few-shot examples, and system prompt, is sent to the LLM for processing. 
8. LLM generation process: The LLM generates one or more responses for each prompt with the focus being context aware responses. You can optionally evaluate and rerank them.

The architecture and workflow above became popular as it was very straightforward and worked as a single agent RAG architecture. The focus was a single pass where the one lookup you do drove the results you achieved in answering. We’ll dive into the issues with RAG that lead to people moving away from this static approach, and how it works now. 

‍

The Width.ai Framework - What we do differently that works (updated 2026)

There are a few key concepts we think about a bit differently that have led to really strong success at scale as models, tools, and use cases change. We’ve deployed RAG conversational systems for fortune 500 customers at scale and have seen how these builds can shift over time with user growth. 

This has drastically changed in 2026 with newer methods of RAG. We have completely overhauled the way that we think about RAG and have pushed these new methods to production. 

Here are some concepts we focus on:

‍

Retrieval Recall is the foundation of everything

If you cannot retrieve the relevant information needed to answer the query, how are you going to get the answer right? How is your accuracy of a question/answer system ever going to be higher than the accuracy of the tool used to get information to answer? While in practice it's not this black and white, as there are chunks of information that are not the “exact” right chunk that can definitely still answer the question, retrieval recall is the foundation for everything else built on top of RAG. Lots of work goes into prompt, task definition etc that should be dedicated to just improving getting the right information from the knowledgebase. 

‍

Retrieval has completely changed some use cases

RAG started as a simple embedding similarity lookup based on the query and knowledgebase. Then it evolved to generating augmented queries based on domain specific knowledge about how to find the best information. Then it evolved to hybrid approaches like embedding + keyword search to rank specific terms higher in the results. Newer approaches have completely changed this by using ultra fast tools like grep, glop, or file search. 

‍

Model context size has changed chunking and information sorting

Most of the work around chunking documents in the knowledgebase was due to context window limitations in llms. When llms had limits of 4k, 8k, or even 100k tokens, you had to take those limits into consideration when using large documents or many documents in the knowledgebase. Now with model context windows reaching up to a million tokens we can completely rethink our strategy for storing data. 

‍

Evaluation is growing in complexity

Single pass RAG systems were easy to evaluate for accuracy. Recall scoring, hallucinations, top-k as a baseline metric for those, and log probabilities. Dynamic systems like autonomous ai agents are much more complex to evaluate and require additional metrics and steps to track. 

‍

Agentic frameworks are the future, but you need to understand how they work

Agentic frameworks, specifically ones that scale up and down in complexity dynamically based on the query are the future. The scalability of agent + tool + validation frameworks actually makes these systems easier to manage over time. These also allow for parallelization of required tasks much easier than traditional workflows. 

‍

These are all over the place, but are much more complex than they are made out to be. The dynamic nature they can provide through ReAct patterns and recursions is awesome, but the added components that aren’t used every single time you run make testing, evaluation, and iteration more complex. 

‍

Our Generalized Agentic RAG Architecture

Our RAG architecture is straightforward once you understand the pieces used and what they accomplish. Our full architecture is built on the concepts of:

• Top level Planning with Files agentic architecture
• ReAct agent pattern used for agents with tool calling ability
• Validation at agent level and final response level. Final response level validation can take a step back and reuse agents
• Context and memory management with the plan and agent level

‍

At a high level this architecture allows us to easily do these things across industries and use cases:

• Keep a consistent agent pattern that allows us to easily scale to new agents
• Each prompt step is simple and repeatable, each one has simple goals and simple task definition. 
• Plug and play pieces of this architecture based on the requirements are easy, need a query augmentation step due to a more complex knowledgebase? Just add it before the agent kickoff step. Need a new embedding model? Just trade it out in the agent tool library.  

‍

Let’s walk through the most used pieces in production.

‍

Planning with Files as the backbone for agentic rag systems

Planning with files is an agentic framework focused on building a “plan” at the beginning of the search process to outline which agents to use and how they should operate. The plan is given:

‍

• What the task is (the query, chat conversation, etc)
• What tools are available via the tool library
• What agents are available and how they operate
• Any file structure, database structure, or knowledgebase outline
• Current context or running memory. This can include previous runs or guided examples
• Knowledge about how to outline success 

‍

And the plan outlines:

The step by step workflow for how to complete this task. What agents are used, what order they are used, any dependencies the agents have, per agent task outlines the agents used to perform their tasks, and how to validate the final output. 

Inside the plan file creation are these subtasks to build a plan, run the plan, and interact with the agents. Think of it as a master agent in charge of downstream intelligent agents.

‍

This system is commonly set up with quick routing built in so simple queries or follow up questions we already have the context for do not need to go through this whole process. This is the same framework that Claude Code uses to manage complex and deep retrieval tasks (reading a codebase) vs replying quickly. Users expect a level of correlation between how much complexity they assume a task has and how long the answer takes, so routing is what lets one architecture handle complex queries efficiently without making every simple one pay the planning cost.

‍

ReAct agent pattern for agentic retrieval and tool calling

I’ve been using the ReAct prompting pattern since 2022 in production, and is the primary framework used for agents in a larger architecture. Most out of the box agentic frameworks like LangChain, LangGraph, and Microsoft Agent support this pattern natively. 

Using a framework that uses multiple task specialized agents instead of one monolithic approach has a few key value propositions:

• Each ai agent is very isolated in what it has to accomplish which reduces hallucinations and allows for easier management of information between recursions.
• Working memory can be task specific, so less confusion.
• Multi-step reasoning for cleaner decision making
• Tasks are clearly defined with a task, goal, output schema, and tool library. Not only does this improve accuracy, but makes evaluation easier. 

‍

The ReAct ai agent can be set up to have access to all retrieval tools, or just tools specific to the “type” of agent you want to define. This comes down to your preferred architecture approach and level of control you want to have over the dynamic nature of your system. 

‍

A key part of this pattern in relation to retrieval is the ability to perform recursions. These recursions allow you to do multiple retrievals and searches instead of a single pass like old school RAG systems. This greatly improves the retrieval recall before the user ever sees the answer. 

‍

Plan validation to ensure coverage and plan goals

Plan validation is a single prompt in charge of validating the results from the agents against the original query and plan to ensure we’re ready to respond. This is a critical step that can retry agents if needed, create citations/references if the task requires them, and generate the response. Traditional RAG systems just return a response based on the context. 

‍

Context merging for multi agent systems

In some use cases using multiple agents for retrieval you will receive a ton of context back from various sources. While we’re happy that we were able to find a bunch of relevant information to answer the query we now might have a few issues:

‍

• Overlapping information from different sources
• Same document multiple times
• Irrelevant context that ranked high due to semantics, keywords etc
• Contradicting information from two different sources

‍

We can use specific algorithms to solve this problem based on scoring algorithms from how the information was retrieved. This should be something you’re familiar with to an extent, like ranking sources retrieved using embeddings by cosine similarity. This becomes a bit different when we use different retrieval methods in the same agentic RAG framework. 

‍

The tool for this is Reciprocal Rank Fusion, which merges ranked lists using each document's position rather than its score. That matters here because your agents are not returning comparable numbers: one comes back with cosine similarities, another with BM25 scores, another with a grep hit that has no score at all. The mechanics are covered in the Hybrid Search section below, where the technique originates.

‍

What changes in a multi-agent RAG setting is the assumption underneath it. Hybrid search fuses two lists that ranked the same query, so a document's absence from one list is meaningful evidence against it. Agents run different sub-queries, so absence usually means the document was never in scope for that agent rather than that it was rejected. Fuse naively and you penalize exactly the documents your plan decomposition was designed to surface.

‍

Three adjustments handle the failure modes above. Deduplicate on document identity before fusion rather than after, or a chunk three agents found gets credited three times, once by RRF and twice by duplication. Fuse within a sub-query first and then across, because lists answering the same sub-question are directly comparable, while merging across sub-questions is a coverage decision that belongs to plan validation. And treat contradiction as its own step, since RRF ranks but does not adjudicate: two credible sources that disagree will both rank highly precisely because both are relevant, and resolving them needs recency, source precedence, or an explicit reconciliation turn in the agent loop.

‍

Retrieval Methods and Data Sources

How we retrieve our relevant data, and what the tools look like to decide what systems we access to retrieve data, are the most important parts of RAG systems. It’s critical we pull context from the proper sources to even have a chance to answer the query!

‍

Chunking for Unstructured Documents

Chunking breaks up the information in your KB into fragments for retrieval. Ideally, you don't want to lose important information or context while doing so. Chunking is required in most systems that have expansive documents across multiple domains. Additionally, Liu et al. showed that the accuracies of LLM-generated answers degrade with longer context lengths and when relevant information is in the middle of long contexts.

‍

Some chunking techniques are:

• Naive limit-based chunking: You just split the information based on the token limit, ignoring any kind of syntactic or semantic consistency.
• Overlapped chunking: This is slightly better than naive chunking because you keep some context from the previous and next chunks by overlapping some of their starting and ending tokens.
• Chunking based on document structure: The chunks here follow the inherent structure of your documents. For example, structural elements like paragraphs, sections, chapters, tables, lists, and code blocks are treated as separate chunks.
• Hierarchical chunking using summaries: Another context-preserving technique is similar to overlapped chunking but instead of the original tokens, a summary of the previous chunk is included in the current chunk. Goal is deeper contextual understanding.

‍Chunking decisions really depends on:

• The type of documents you have and the layout of them. You will extract and chunk information differently in tabular documents like medical records vs blog posts, as the extraction process to map relevant information in a page is different. I wrote a ton about this in my healthcare guide.
• Your chosen retrieval method. These two play into each other a bit, so its not entirely straightforward as one retrieval method decides one chunking method. 
• The use case. Conversational, generation focused, or simply retrieving information. 
• How will this content merge with other data sources in dynamic environments? Is this content being used alongside structured tool calls like model context protocol requests, web searches, or DB lookups.

File Search with Grep & Glob

Every other retrieval strategy in this piece assumes you build something first: chunk the corpus, embed it, store the vectors, keep them fresh. Grep and glob assume nothing — point them at a directory and they work. On its own that was never enough. Grep's fatal flaw in classic RAG was vocabulary: the user types "auth is broken," the document says ERROR_AUTH_INVALID_TOKEN, grep returns nothing, and the query is lost. The index had to be smart because the query was dumb, just raw user text, one shot, no recovery. Agentic retrieval removes that constraint. The agent doesn't hand grep the user's words; it reads intent, generates a pattern, evaluates what came back, and generates a better one. Query generation per recursion is the vocabulary bridge and is the one capability grep lacked, now supplied by the loop instead of the index. How far the bridge carries depends on the domain: it's strongest where terminology is consistent or where the model already knows the vocabulary, and thinnest in idiosyncratic prose.

‍

What grep owns is the class of queries where the literal string carries the meaning. Exact identifiers: ERROR_4001 and ERROR_4002 are neighbors in vector space and opposites in fact. Numerics: $45.2M and $45,200,000 embed differently, and tables of figures have famously poor semantic representations. Blurring near-identical strings is precisely what embeddings are for, and precisely wrong when the string is the answer. The same property makes grep's failures cheap. Top-k returns k chunks whether or not the answer exists in the corpus, so a miss is indistinguishable from a hit until the agent has read and rejected the results — a full turn, plus thousands of tokens that stay in context and slow every turn after it. Grep's null is instant, definitive, and free. And because grep reads disk rather than a snapshot, an agent that writes during its own loop sees its own edits; an index is stale the moment recursion three modifies what recursion five will search.

‍

The strongest case is documents that cross-reference themselves, and it's why this transfers beyond code. "See Note 12" is a literal string. So is "as defined in Section 4.2" and so is an import statement. In a corpus with explicit reference structure — SEC filings, contracts, regulatory submissions, technical manuals — the previous result contains the next query, and the agent doesn't have to invent a synonym to continue. It reads the pointer and greps it. Tracing a lease obligation runs: find "lease" in the statements, hit "See Note 12," navigate there, hit "excluding discontinued operations (Note 23)," check Note 23 for the additional $2B, cross-reference MD&A, then search subsequent events for a $500M termination. Five recursions, each one handed its query by the document itself. Semantic search has no concept of following a pointer — it finds text resembling your query, which is a different operation entirely. The vocabulary problem doesn't arise, because the corpus is doing the query generation.

‍

The honest accounting is that both approaches pay per recursion, and only one of the costs scales with your corpus. A production pipeline with a reranker adds roughly 300–2000ms per query before the LLM sees anything, and rerankers cap out around 4096 tokens, so larger retrievals split into parallel calls and merge. Grep pays a scan: roughly corpus size ÷ 5GB/s against a warm cache. Both multiply by recursion count, so recursion depth doesn't decide the winner — it amplifies whoever's already ahead. Corpus size decides the sign. Under ~1.5GB, grep's scan undercuts even the fastest reranker; past ~10GB it exceeds the slowest, on every step, forever. In between it's close enough to test rather than argue. So reach for grep and glob when the corpus is small enough to scan, changes faster than you can index it, references itself, or answers to exact identifiers, and reach for the index when it's large, stable, flat, and conceptual. These aren't competing for the same job.

Embeddings Design

Embedding design and structure matters a ton in systems where semantic similarity is a huge chunk of the retrieval process. 

Embedding Length, Corpus Variety, and Model Size

These three factors influence the quality of semantic similarity matching.

Generally, longer embeddings are better because they can embed more context from the information they were trained on. So the 1,536-dimensional embeddings produced by the OpenAI API are likely to be qualitatively better than the 768-dimensional embeddings from a BERT model.

The volume of data that a model is trained on also matters. The OpenAI models or the T5 model are known to have trained on far more variety than BERT. Since larger models hold more information in their parameters, their embeddings too are of better quality.

Lastly, the nature of the training data influences the quality of the embeddings. A specialized model like Med-BERT may produce better embeddings for medical tasks than general embeddings even when the embeddings are shorter.

We are huge fans of fine-tuning embedding models to be more domain specific. Fine-tuning embedding models allows you to refine what it means to be “similar” for your specific use case, which can be very different in specific domains compared to the general knowledge a model has. To a generalized embedding model, any text with legal jargon is “somewhat” similar, but an embedding model used in a legal chatbot needs to have a more granular understanding. 

SentenceTransformer Models

The SentenceTransformers framework provides a large number of embedding models for both symmetric and asymmetric searching. These models differ in their model architectures, model sizes, relevance quality, performance, training corpora, language capabilities, training approaches, and more.

Llama 2 Embeddings

An open-source LLM like the 70-billion-parameter model of Llama 2 is far more versatile and powerful than anything available with SentenceTransformers. Its embeddings will yield much better quality.

OpenAI Embeddings

OpenAI embeddings API is another good choice to generate your embeddings. Since it's a managed and metered API, performance will be slow and incur expenses over time. However, it's a good choice if your knowledge base is small.

‍

Hybrid Search

Hybrid search runs two retrievers over the same query: BM25 for lexical matching, and vector search for semantic similarity, then merges the two ranked lists into one. That merge is harder than it sounds, because the two systems don't speak the same language. BM25 scores are unbounded and depend on corpus statistics, document length, and term rarity; a score of 14.2 means nothing on its own. Cosine similarity is bounded roughly [0,1] and says nothing about term rarity. You can't add them, and normalizing them is fragile: min-max scaling depends entirely on the result set you happened to retrieve, so a single outlier at the top compresses everything below it and the weights shift from query to query.

‍

Reciprocal Rank Fusion sidesteps the problem by throwing the scores away. It keeps only the order:

RRF(d) = Σ 1 / (k + rank(d))

‍

Each document scores the sum of its reciprocal rank across every list it appears in, with k a constant conventionally set to 60. That's the entire algorithm, a few lines of code, no training, no calibration, nothing to tune per corpus.

‍

The effect is that documents both retrievers like rise to the top, even when neither ranked them first. A document at rank 5 in both lists scores 2/65 = 0.031; a document at rank 1 in one list and absent from the other scores 1/61 = 0.016. Agreement beats a single strong opinion by roughly 2x. That's the behavior you want when the two retrievers have complementary blind spots: BM25 misses paraphrased or abstract information, vectors miss exact terms and weighting more useful terms,  because a document that both systems surface is one that's relevant on both lexical and semantic grounds.

‍

Why use it over semantic search alone

Vector search fails predictably on exact strings. Error codes, SKUs, ticker symbols, clause numbers, part numbers — ERROR_4001 and ERROR_4002 sit next to each other in embedding space and mean opposite things. Same for numerics: $45.2M and $45,200,000 embed differently. BM25 handles all of that natively and weights rare terms correctly, which is exactly what jargon and identifiers are. Running both and fusing gets you paraphrase tolerance from the vector side and exact-term precision from the lexical side, in one list, without picking which failure mode you'd rather have.

The practical case for RRF over weighting or normalizing scores is that it just works without a fine-tuning impl. It's the default hybrid ranking method in Elasticsearch, OpenSearch, Azure AI Search, MongoDB Atlas, Weaviate, and Qdrant, which means for most stacks it's a config flag rather than an implementation. It's also cheap operationally: ranks can be summed one system at a time, so you never hold both full lists in memory.

‍

Retrieve Knowledge, Prompts, or Both?

As you know an LLM's response is guided by multiple pieces of information in the prompt like:

1. The primary task or question
2. The system prompt that guides the LLM's behavior, tone, and personality
3. Contextual information and its relevance to the task or question
4. Few-shot examples to demonstrate desired processing or results

Conventional RAG focuses only on retrieving contextual information that is relevant to the task.

However, there's nothing special about context. You can use the same retrieval approach to select relevant tasks, questions, system prompts, or few-shot examples. We demonstrate the need for this and the outcomes in the sections below.

Retrieval of use case specific prompt information

All promptable LLMs are sensitive to the structures and semantics of prompts relative to the query. That's why there are so many prompt engineering tips and tricks in circulation.

RAG can be used to improve and standardize the language used in prompts for specific inputs, retrieved document types, and use cases by maintaining a knowledge base of predefined task prompts that are known to work well for specific uses. Instead of forcing users or systems to send well-formed prompts, use RAG to select predefined task prompt language that are semantically similar to the requested tasks but work better.

For example, in healthcare record Q&A systems we use document classification as a step in document extraction, and can use that classification for specific document types (admission records, transcripts, dosage report etc) to provide specific rules that improve quality of answers. 

Retrieval of guided examples

Some steps of the agentic RAG process can be enhanced by retrieving relevant examples of how to successfully complete that specific step. Things like:

‍

• Successful plans for the planning step to use
• Successful tool calls for the agent to use to speed up search and reduce recursions. This is probably the most used instance of this idea as the ROI on recursion and accuracy is very high
• Examples of validating generated answers

‍

Vector Database Selection

When selecting a vector database for your RAG pipeline, keep the following aspects in mind.

In-Memory, Standalone, and Managed Databases

There are multiple ways to deploy a vector database. Some run as components inside your application process and are thus limited by the system's memory. Some can be deployed as standalone distributed processes. Others are managed databases with APIs.

You must select a database that is suited to the scale and quality of service you need for your RAG workflows:

• If you're deploying RAG agents in production and require high scalability for the volume of information or users, choose a managed database service like Pinecone, Weaviate Cloud Service, or Qdrant Cloud.
• If you prefer self-managed solutions, go for open-source, production-ready, standalone, distributed, highly scalable databases like Weaviate, Mlivus, or Qdrant.
• If you're just prototyping RAG or implementing it as a minor component and don't expect a high scale of information or users, a simple in-memory database like Chroma or FAISS will be best.

‍

Case Study: Sales Search & Insights Agentic Framework

Built with our agentic framework to stay on top of sales and pipeline management, with Slack wired in directly for team communication and integrating with real time data. 

Every team knows the pattern. A deal quietly loses momentum. An email sits in someone's inbox for a week. A warm lead cools off because the thread of the last conversation got dropped. A renewal comes and goes because the date was tucked inside a CRM record nobody opened. None of these are catastrophic on their own — but across a quarter, these small leaks are exactly where revenue goes missing.

This system is designed to seal those leaks. It ties HubSpot, Gmail, and Google Docs together behind one conversational interface, so anyone on the team can ask a question in plain language and get back an instant, sourced answer:

‍

"Did we ever circle back with Jackson Corp after the demo?" — scans emails, deals, and contact activity in seconds rather than digging through CRM records one by one.

"Which deals are stuck in negotiation?" — flags deals that haven't advanced a stage, along with who owns them and when they were last touched.

"When did we last connect with Sarah at Width and did we send a proposal?" — retrieves the most recent email, meeting, or note spanning both HubSpot and Gmail.

"How many proposals did Matt send out last month? And list a summary of each" — locates files in Google Drive with no need to recall names or folder paths.

‍

The insights pipeline takes it a step further, actively combing through CRM data to surface patterns you'd never have thought to query — contacts going quiet, deals with no recent movement, or support tickets drifting in a direction worth a closer look. This system runs every day with a working memory of what was provided before so the insights are fresh. 

The bottom line: fewer follow-ups slipping away, quicker answers on where deals stand, and a clear view of what's really moving through your pipeline — all without turning every team member into a CRM expert. (Full writeup)

Case Study: Agentic RAG to Chat Depositions & Full Cases

The Challenge

Litigation teams are buried in paper. A single matter might hinge on one deposition — or an entire folder of them, running to thousands of transcript pages that are frequently scanned and rarely formatted the same way twice. And buried somewhere in that stack is the answer to a straightforward question:

• How much overtime did Ariana actually work?
• Which witnesses brought up social media activity?
• Do the Johnson and Matt depositions line up on the supervisor's schedule?

Getting to that answer typically means hours of reading by hand, fragile keyword searches that surface stray lines rather than real answers, or heavyweight enterprise search platforms that take weeks to stand up and a dedicated specialist to keep running.

Not one of those options manages to be fast, affordable, and reliable all at once.

The Solution

We built an agentic framework that takes plain-English questions about a folder of documents and hands back a clear, written answer — complete with citations to the exact pages it drew from. Pose a question the way you would to a colleague, and it does the reading on your behalf.

It runs in two straightforward stages:

1. Preparation. Every PDF is split into individual pages, and each page is automatically assigned a descriptive, human-readable label reflecting the people, organizations, and topics it covers. The page dealing with Ariana's overtime turns into something anyone can spot at a glance: page 77 — Ariana Arellano, Molly Chuen, overtime, employee We lean on our state-of-the-art document processor to convert the deposition pages into a text format better suited to agentic tool calling. It also handles embedded images and cross-references between pages, and scanned documents are processed automatically with no extra effort.
2. Answering. When a question comes in, the system reads only the pages that matter — not the entire stack — and replies in everyday language, always citing the source pages so every claim can be checked. Response length is configurable, too: dial it up for more detail, down for something shorter, or switch on a brief-style setting for quick answers.

Full case study

‍

Ready to fix your Agentic RAG System?

In this article, you studied various design and implementation aspects of RAG. At Width, we have implemented and deployed RAG in production for banking clients and law firms where retrieving the latest information is an absolute necessity. If you want to streamline your workflows using LLMs on your company's private documents and data, contact us!

References

• Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, Douwe Kiela (2020). "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks." arXiv:2005.11401 [cs.CL]. https://arxiv.org/abs/2005.11401

‍

Additional Notes: Ability Trained LLMs

The idea behind ability trained LLMs vs knowledge trained LLMs focuses on what the goal of the LLM is in RAG. We want to use it to understand and contextualize information provided to the model for generation, not pull information from its training for generation. These are two very different use cases, and two very different prompting structures. Here’s a better way to think about it.

All understanding of the task and what information is available to perform the task is based on what is provided. This way the model only generates responses based on this specific information, and none of its underlying knowledge, which can lead to hallucinations.This generally requires some level of extraction from the model to understand what is relevant. Although you might not actually perform an extraction step, the model has to do this with larger context.

This is what it looks like when we rely on the LLM for the knowledge used to answer the query. Everything is focused on the prompt and the knowledge the model is trained on.

This means the key focus of the LLM in RAG is natural language understanding: comprehending the context provided and how it correlates with the query, not retrieving facts from its weights. What that means for fine-tuning LLMs is the focus should be on improving the LLMs ability to extract and understand provided context, not fine-tuning the LLM to improve its knowledge. This is how we best improve RAG systems by minimizing the data variance that causes hallucinations or poor responses. Our LLM better understands how to handle context from multiple sources and sizes which becomes more common as these systems move to production use cases. This means we can spend less time trying to over optimize chunking algorithms and preprocessing to fit a specific data variance as our model is better at understanding the inputs and how they correlate to a goal state output.

‍