the signal

Why Your RAG Isn’t Working

Written by

Arthur Mello

in

And what to do about it

Retrieval-Augmented Generation (RAG) was supposed to fix everything. More accurate answers. Less hallucination. Instant access to domain-specific knowledge.

But in real-world deployments, the results often disappoint.

Answers feel off. Retrievals are irrelevant. Context seems lost. So what’s going wrong?

Let’s break down the core problems, and how to fix them.

1. Vector embeddings aren’t magic

RAG relies on vector embeddings to find semantically similar documents. But embeddings aren’t perfect.

They compress language into fixed-length vectors and, in that compression, nuance gets lost.

The issues:

  • Polysemy: One word, multiple meanings. Embeddings may pick the wrong sense.
  • Synonymy: Different words, same meaning — but often not close enough in vector space.
  • Dense mapping: Common terms dominate, drowning out contextually relevant ones.
  • Lossy compression: Some information simply disappears during vectorization.
  • Fragmented meaning: Chunking documents too finely can split up important context.

Bottom line: vector similarity ≠ true relevance.

2. Your query might be the problem

If your query is too vague, you’ll get shallow matches.

Too specific? You risk missing relevant documents that use slightly different phrasing.

Fix it with:

  • Query rephrasing: Reformulate user queries before embedding them, to better align with how data is structured.
  • Disambiguation: Make sure the model understands what you actually mean (replacing acronyms, etc.)
  • Context tuning: Keep queries focused and information-rich — not bloated or ambiguous.

Your retrieval is only as smart as your query.

3. Your chunking strategy is hurting you

Chunking is more than just splitting text. It’s a balancing act.

Too small, and you lose context. Too large, and you overload the model.

Strategies to explore:

  • Sliding window: Maintains continuity across chunks.
  • Recursive chunking: Uses document structure (headings, paragraphs) to guide splits.
  • Semantic chunking: Groups based on meaning, not just tokens.
  • Hybrid chunking: Combines multiple methods, customized per use case.

The right chunking strategy depends on your data and your goals.

4. You’re missing Named Entity Filtering (NEF)

Named Entity Recognition (NER) isn’t just for tagging people and places.

It can drastically sharpen your retrievals by filtering documents based on entity-level relevance.

Use it to:

  • Filter results to only include documents with relevant entities.
  • Refine embeddings by focusing on entity-rich sentences.
  • Reduce noise and boost relevance, especially for technical or domain-specific content.

Pair this with noun-phrase disambiguation, and you’ll see a big drop in hallucinations.

5. You’re using embeddings too early

Embeddings are great, but don’t make them do all the work upfront.

Sometimes, traditional keyword matching or metadata filtering gives a cleaner first pass. Use vector embeddings to rerank or refine after that.

Think hybrid:

  • Start with keyword or synonym-based retrieval.
  • Apply vector search as a second pass.
  • Fine-tune embeddings for your domain for even better alignment.

Precision + semantic recall = better results.

6. You’re not using advanced RAG techniques

RAG has evolved. Basic setups won’t cut it anymore.

Try these:

  • Reranking: Cross-encoders to reassess document relevance.
  • Query expansion: Add synonyms, related terms, or constraints.
  • Prompt compression: Strip irrelevant content before feeding it to the model.
  • Corrective RAG (CRAG): Evaluate and refine retrieved documents dynamically.
  • RAG Fusion: Generate multiple queries and fuse their results for broader coverage.
  • Contextual Metadata Filtering RAG (CMF-RAG): Automatically generate metadata filters from the user query.
  • Enrich documents with context: When chunking, add a summary of the page or document, for context.

Use what fits your data and needs. There’s no one-size-fits-all.

Putting it all together

How do you know what will work for your use case?

Set up automated tests:

  1. Define a batch of 50–100 relevant questions
  2. Use an LLM as evaluator
  3. Iterate through different chunking strategies, hyperparameters, type of search, etc. and store results
  4. Analyze results and choose the best setup

Final thoughts

RAG isn’t broke, it’s just misunderstood.

It’s easy to slap vector search on top of an LLM and call it a day.

But building a high-performance RAG system takes more.

  • Tune your queries.
  • Chunk your documents wisely.
  • Filter with entities.
  • Rerank with smarter models.
  • Layer retrieval techniques strategically.

In short: stop treating retrieval as an afterthought.

It’s half the battle.

And often the one you’re losing.

Feel free to reach out to me if you would like to discuss further, it would be a pleasure (honestly):

Comments

Leave a comment

More posts