Why Does AI Sometimes "Get Lost"? — Engineering Practices for Retrieval-Augmented Generation (RAG)

A Real-World Problem

When your AI system answers questions, it often cites outdated information, fabricates non-existent details, or simply says, "I'm not sure." This isn't due to a lack of model capability; rather, it's being asked to do the impossible—memorize all information and then extract it accurately.

Enterprise data changes daily in the real world: product descriptions are updated, compliance terms are revised, and employees change their contact information. The only reliable way to keep AI up-to-date with all this information in real time is not to retrain the model, but to have it query the latest materials dynamically when generating answers. This is the core problem that RAG (Retrieval-Augmented Generation) aims to solve.

The Basic Architecture of RAG: A Two-Step Process

A RAG system consists of two independent stages:

**Step 1: Index Construction**

- Extract text from sources such as document repositories, databases, and APIs.

- Split the text into appropriately sized chunks (typically 500–1000 characters).

- Convert each chunk into a vector using an embedding model (dimensions ranging from 768 to 3072).

- Store these vectors in a vector database (e.g., Milvus, Pinecone, Chroma).

**Step 2: Real-Time Retrieval and Generation**

- When a user asks a question, first convert the query into a vector of the same dimension.

- Perform an Approximate Nearest Neighbor (ANN) search in the vector database to find the most relevant chunks.

- Send the original question along with the retrieved chunks to the Large Language Model (LLM) to generate an answer.

Key point: The retrieval and generation stages are completely decoupled. The model no longer needs to recall facts from its training data; instead, it gathers evidence from current data.

Practical Engineering Challenges

1. Document Chunking Strategy

Splitting documents by fixed length can lead to semantic fragmentation. Better approaches include:

- Splitting naturally by paragraphs or headings.

- Using hierarchical chunking for long documents (Document level → Chapter level → Paragraph level).

- Preserving contextual links between chunks (e.g., using the last 100 characters of the previous chunk as the header for the next one).

2. Retrieval Quality

Not all retrieved results are relevant. Best practices include:

- Setting a retrieval threshold to filter out chunks with similarity scores below a certain value.

- Reranking retrieved results: Using a smaller model to evaluate the true relevance of each chunk to the query.

- Adjusting `top_k` (typically 5–15 chunks) to avoid information overload.

3. Index Update Latency

When source documents change, the vector database needs to be synchronized. For frequently updated content:

- Adopt an incremental update strategy, updating vectors only for the changed parts.

- Or use batch processing windows (e.g., bulk updates every hour).

Why RAG Is Not a Panacea

While RAG can significantly improve factual accuracy, it has several inherent limitations:

1. **Retrieval Errors**: If the vector database does not contain any chunks with the answer, the system will return errors or fabricate content.

2. **Context Limits**: Even if relevant chunks are retrieved, the LLM may struggle to correctly handle multiple conflicting sources of information.

3. **Computational Overhead**: Each request requires embedding and vector search, typically increasing latency by 50–500ms.

Practical Configuration Recommendations

For small to medium-sized enterprise systems (<100,000 documents):

- Use open-source solutions: LangChain + Chroma + any LLM.

- Embedding models: `bge-large-zh` (optimized for Chinese) or `text-embedding-3-small`.

- Retrieval strategy: `top_k=8` + simple threshold filtering.

For large-scale systems (>500,000 documents):

- Use commercial vector databases (Pinecone, Weaviate).

- Multi-layer indexing strategy: Use fast embeddings for coarse filtering and a reranker for fine filtering.

- Caching layer: Cache results for common queries to reduce vector search calls.

RAG is not magic that makes AI smarter; it is engineering that makes AI more honest. It acknowledges the limitations of AI and compensates for them with verifiable evidence.

---

*Original link: https://www.smallfiredragon.com/science/rag-engineering-practices-20260727*