Skip to main content

RAG's

Updated 5 min readMarkdown

Why was RAG introduced

LLM's are trained on large amounts of data which allows them to answer questions, write code, summarize documents and more such tasks. However, once LLM is trained it's knowledge becomes fixed. If new information becomes available such as updated company policies, the latest product documentation, or today's news the model doesn't automatically know it.

This is where RAG's was introduced. Rather than training LLM again Retrieval-Augmented Generation (RAG) is used where additional information is passed and used to generate the relevant answers about the additional information passed.

RAG Pipeline

1. Data Ingestion

Firstly, raw data which might be in form of pdf, text files, csv ,outlook ,emails and more is ingested to the RAG system.

2. Data pre-processing

Once data is loaded ,the large data is transformed into small parts using techniques like text-splitting, chunking and others.

3. Generating vector embeddings

The pre-processed data is then used to generate vector embeddings, which represents the data in a numerical format.

4. Storing embeddings in Vector Databases

The processed data and generated embeddings are stored in specialized databases known as vector databases. These databases are optimized to handle vectorized data, enabling rapid search and retrieval operations.

5.LLMs

LLMs are used to generate fully formed responses based on the user query and contextual information retrieved from the vector DBs during user queries.

6. User Query

When a user submits a query, the RAG system uses the indexed data and vectors. The system identifies relevant information by comparing the query vector with the stored vectors in the vector DB . The LLMs then use the retrieved data to answer the query.

Scenarios where RAG works well

  • Chatbots: Many companies already use AI chatbots to interact with customers ion their websites. With RAG, companies can build a chat experience that’s highly specific to their product. For example, questions about product specifications could easily be answered.

  • Customer service: Companies can empower live service representatives to easily answer customer questions with precise and latest information.

  • Enterprise search: Businesses have a wealth of knowledge across the organization, including technical documentation, company policies, IT support articles, and code repositories. Employees could query an internal search engine to retrieve information faster and more efficiently.

Why RAG Sometimes Gives Incorrect Answers

Feeding data to RAG improves the chances of a correct answer, but it doesn't guarantee one.

LLM knows and answers only from the data it is received. It fails to answer properly and gives incorrect answers when the retrieved information is incomplete or irrelevant.

The quality of answer not only depends on LLM but also on the every stage of the RAG pipeline

Example

Imagine asking:

"What is the company's maternity leave policy?"

If the retriever finds the Employee Benefits document instead of the Leave Policy document, the LLM will answer using incomplete information.

Even though the LLM is working correctly, the retrieval stage failed.

This shows that the retrieval stage plays a critical role in the overall quality of a RAG system.

Poor Retrieval and Missing Context

Retrieval is not about finding the correct answer but its about finding the documents that may contain the correct answer and passing it to the LLM.

Retrieval plays a huge role in answering the write answer. If retrieval selects the wrong chunks important and required information will be missed, unrelated information will be added and LLM has to make assumptions

Poor Chunking and Its Impact

Lets understand the impact of poor chunking with an example.

Imagine this document:

Refunds are available within 30 days.

Products must be unused.

Digital products cannot be refunded.

Now suppose chunking splits it into:

Chunk 1

Refunds are available

Chunk 2

within 30 days.
Products...

The sentence is broken.

When retrieval returns Chunk 1, the LLM only sees:

Refunds are available

It never sees

within 30 days.

The answer becomes incomplete.

Now compare it with proper chunking:

Chunk 1

Refunds are available within 30 days.
Products must be unused.

The meaning stays intact.

Context window limitations

Even after retrieving relevant documents, the LLM cannot process unlimited information.

Every model has a context window, which determines how much text it can consider while generating a response.

Imagine trying to study using ten textbooks while your desk only has space for three. Even though all ten books contain useful information, you can only keep a few open at once.

Similarly, if too many document chunks are retrieved, some may need to be discarded before reaching the model. If the omitted chunks contain critical details, the quality of the final answer may suffer.

Hallucinations even with RAG

A RAG system is only as useful as the information stored in its knowledge base.

If documents become outdated, are removed, or are never indexed, the retriever cannot provide accurate context. Consequently, the LLM will generate responses based on outdated or incomplete information.

Maintaining an up-to-date knowledge base through regular indexing and document management is therefore an essential part of any production RAG system.

When RAG is not the right solution

Although RAG is powerful, it is not the answer to every AI problem.

Tasks such as basic arithmetic, language translation, creative writing, or general reasoning typically do not require retrieving external documents. In these situations, the LLM's existing knowledge is often sufficient.

RAG is most valuable when answers depend on information that is private, frequently changing, or too specialized to have been included during the model's original training.

No comments yet. Be the first to comment.