Skip to main content

The Answer Was in the Database. Why Couldn't RAG Find It?

Updated 31 min readMarkdown

From keyword matching to semantic, hybrid, graph, and agentic search — understanding how AI applications find the right information.

Imagine building an AI assistant for an online shopping platform.

A customer asks:

"I received a damaged product. Can I get my money back?"

Somewhere in your database, there's a document that says:

Customers may request a full refund for defective or damaged items within 30 days of delivery.

The answer exists. The document is stored correctly. Your embedding pipeline ran successfully.

Yet your AI assistant responds:

"I couldn't find any information about that."

What went wrong?

Not necessarily the LLM. Not necessarily the prompt.

The problem could be retrieval.

We spend a lot of time discussing which LLM to use, how to write better prompts, or which vector database performs best. But there's another question worth asking:

How does our application actually find the information the LLM needs?

That's where search strategies come in.

And there are far more of them than simply choosing between keyword search and vector search.

Let's explore them, starting with traditional search and gradually moving into the techniques used in more sophisticated RAG systems.


First, Where Does Search Fit into RAG?

RAG stands for Retrieval-Augmented Generation.

Instead of expecting an LLM to know everything, we retrieve relevant information from external sources and provide that information as context before generating a response.

A typical pipeline looks like this:

          User Question
                |
                v
         Query Processing
                |
                v
         Search / Retrieval
                |
                v
          Relevant Chunks
                |
                v
        Reranking / Filtering
                |
                v
         Context Assembly
                |
                v
          LLM Generation
                |
                v
          Final Response

The interesting part is the retrieval stage.

Consider three questions:

  • "What is the refund policy?"
  • "Can I return a broken item?"
  • "What does policy RF-2026-04 say?"

All three relate to refunds, but they require slightly different search behavior.

The first is a straightforward information request. The second relies heavily on understanding meaning. The third contains an exact identifier.

One search technique may handle all three reasonably well, but different techniques have different strengths.

One important distinction before we begin: not everything discussed below is a standalone search algorithm.

Some techniques retrieve documents. Others improve the query, combine search results, navigate related information, or rerank what has already been found.

They're all part of designing a strong RAG retrieval system.

Part 1: Traditional Search — Finding Words, Phrases, and Records

1. Exact Match Search — When Every Character Matters

Let's start with the simplest approach.

Exact match search retrieves information by matching a specified value.

Suppose an engineer asks:

"Find the documentation for error AUTH_403."

Your database contains:

AUTH_401 - Invalid authentication credentials
AUTH_403 - Insufficient user permissions
AUTH_429 - Too many authentication attempts

An exact match on AUTH_403 returns the second record.

No embeddings. No language model. No semantic interpretation.

Just a precise match.

Where it's useful: Error codes, transaction IDs, order numbers, document IDs, product SKUs, and other identifiers.

Where it struggles: Someone searching for "permission denied error" may not know the exact code.

This is one reason semantic search shouldn't automatically replace traditional search.

Sometimes understanding exactly what someone typed is more important than understanding what they meant.

2. Full-Text Search — Searching Across the Words in a Document

Now imagine an employee searches:

"Annual leave policy for new employees"

Your knowledge base contains hundreds of HR documents.

A basic string-matching approach might look for that exact phrase. Full-text search goes further.

It typically processes documents into searchable terms using techniques such as tokenization, normalization, and sometimes stemming or stop-word removal.

For example:

Document:
"New employees become eligible for annual leave
after completing their probation period."

Search terms:
annual, leave, new, employee

A full-text search engine can find the relevant document even when the query doesn't match the original sentence word for word.

Common technologies include PostgreSQL full-text search, Elasticsearch, OpenSearch, and Apache Solr.

Where it's useful: Policy documents, knowledge bases, documentation portals, legal text, and large collections of articles.

Full-text search is a family of techniques. How results are ranked depends on the engine and configuration.

That brings us to BM25.

3. BM25 — Not Just Finding Documents, but Ranking Them

Suppose you search:

"Refund policy for damaged products"

Three documents match:

Document A: Mentions refunds once in a 20-page report.

Document B: Has a dedicated section explaining refunds for damaged products.

Document C: Repeats the word "refund" dozens of times but focuses on an unrelated issue.

Which should come first?

A simple keyword match doesn't necessarily provide a useful ranking.

BM25 is a widely used lexical ranking algorithm designed to estimate how relevant a document is to a query.

It considers signals including:

  • How frequently a search term appears in the document.
  • How rare or informative that term is across the collection.
  • The document's length, with adjustments so repetition doesn't dominate indefinitely.

In our example, Document B may rank highest because its content is more focused on the searched terms.

BM25 doesn't truly understand meaning the way a semantic model attempts to. It's a clever, mathematically grounded way to rank textual matches.

Where it's useful: Technical documentation, support tickets, research repositories, and product catalogs.

A small implementation detail worth knowing: PostgreSQL has native full-text ranking functions such as ts_rank and ts_rank_cd, but they are not the same as BM25. A BM25 implementation requires an appropriate search engine or extension.

4. Boolean, Phrase, and Proximity Search — Giving Search More Rules

Not every search needs to be open-ended.

Sometimes we know exactly which conditions the result should satisfy.

Imagine a legal assistant searching contracts for:

"Confidentiality agreements that mention employee data but exclude external vendors."

Boolean search allows combinations such as:

confidentiality AND employee AND NOT vendor

Phrase search can look for words appearing together:

"confidential information"

Proximity search goes further by finding words that occur close to one another.

For instance, a document where "security" appears near "incident" may be more relevant than one where those words appear 15 pages apart.

Where it's useful: Legal research, compliance, scientific literature, and investigative search.

The exact syntax varies by search engine, but the idea is the same: the searcher can express more precise constraints.

5. Fuzzy Search — Because Humans Make Typos

Think about how people actually search.

A customer types:

"How to cancle my subcription?"

The documentation uses:

"How to cancel your subscription."

The user has misspelled both cancel and subscription.

A fuzzy search implementation can find similar terms using measures such as edit distance — the number of character changes needed to transform one word into another.

cancle -> cancel

subcription -> subscription

Autocomplete and prefix search are closely related search features, although they operate differently.

For example, someone typing subscr might see suggestions for "subscription," "subscriber," and "subscribe."

Where it's useful: Customer support portals, product search, names, and applications with frequent typing errors.

An important distinction: fuzzy search handles textual variation. It doesn't inherently understand that "money back" and "refund" have similar meanings.

That's a different problem.

6. Metadata Filtering — Searching Only Where You're Supposed To

Imagine an organization with 50,000 policy documents.

An employee asks:

"What is the leave policy for Indian employees in 2026?"

A semantic search over every document might retrieve policies from the US, UK, or previous years.

Even if those documents are conceptually similar, they're not necessarily relevant.

Metadata filtering helps restrict the search.

{
  "country": "India",
  "year": 2026,
  "document_type": "HR Policy"
}

The retrieval system searches the appropriate subset of documents.

Metadata might include department, tenant, document owner, creation date, product category, access permissions, or document version.

Where it's useful: Enterprise RAG, multi-tenant SaaS platforms, healthcare records, financial documents, and region-specific policies.

A particularly important detail: metadata filters must also enforce access control. A document being semantically relevant does not mean the user is authorized to read it.

7. Structured Database Search — Sometimes the Answer Is in a Table

Not every RAG question should be answered by retrieving text chunks.

Consider:

"How many orders were cancelled yesterday?"

Your PostgreSQL database contains an orders table with order statuses and timestamps.

Rather than searching embeddings, the application could execute an authorized SQL query:

SELECT COUNT(*)
FROM orders
WHERE status = 'cancelled'
  AND cancelled_at >= :day_start
  AND cancelled_at < :day_end;

The query parameters represent the relevant day's boundaries.

This retrieves the factual count directly.

An LLM may help interpret the request, but the actual calculation belongs in the database.

Where it's useful: Sales dashboards, finance assistants, inventory management, customer account systems, and analytics.

For production applications, database access should use read-only permissions where appropriate, validated query patterns, and user-specific authorization.

A vector database is not a replacement for SQL.

Part 2: Semantic and Vector Search — Finding Meaning, Not Just Words

8. Dense Vector Search — When Different Words Mean Similar Things

Let's return to our original example.

The user asks:

"I received a damaged product. Can I get my money back?"

The document says:

"Customers are eligible for a full refund on defective items."

Notice the vocabulary:

User:      damaged product, money back

Document:  defective item, full refund

The words are different. The meanings overlap.

This is where dense vector search becomes useful.

An embedding model converts text into a numerical representation called an embedding.

Conceptually:

"I want my money back"
          |
          v
     Embedding Model
          |
          v
 [0.12, -0.36, 0.81, ...]

Documents are converted into embeddings too.

The system compares the query vector with stored document vectors and retrieves the closest matches.

Similarity can be measured using cosine similarity, dot product, or Euclidean distance, depending on the model and system.

Where it's useful: Conversational assistants, RAG chatbots, knowledge discovery, and questions where users don't know the exact terminology.

But vector search has limitations.

It can confuse closely related concepts, retrieve vaguely similar passages, or overlook exact identifiers such as RF-2026-04.

Semantic similarity is useful evidence of relevance — not proof that a document contains the answer.

When discussing vector search, you'll often hear these terms:

KNN (k-nearest neighbors): Find the closest vectors based on a similarity or distance metric.

Exact KNN: Evaluate the candidates exhaustively to find the nearest matches.

ANN (approximate nearest neighbors): Use specialized indexes to retrieve likely nearest neighbors faster, potentially missing some exact matches.

Two common ANN indexing techniques are:

  • HNSW: Uses a layered graph structure to navigate toward nearby vectors efficiently.
  • IVF / IVFFlat: Groups vectors into partitions and searches selected partitions.

These aren't alternatives to semantic search. They're different ways of executing vector retrieval.

For example, a small HR knowledge base might work perfectly well with exact vector search. A system containing millions of embeddings may benefit from ANN indexing.

Libraries such as pgvector support both exact search and approximate indexes.

9. Sparse Vector Search — Giving Important Terms More Weight

Dense vectors capture broader semantic information, but there's another family of vector representations: sparse vectors.

Sparse vectors contain many zero-valued dimensions, with relatively few nonzero values representing important terms or features.

Traditional sparse retrieval methods include TF-IDF and BM25-style lexical representations.

Modern learned sparse retrieval methods, such as SPLADE, use neural models to produce weighted term representations and may expand relevant vocabulary.

Imagine a customer asks:

"How can I request parental leave?"

Your HR documents discuss:

  • Parental leave
  • Maternity leave
  • Paternity leave
  • Family leave benefits

A learned sparse model may assign weights to related terms, helping retrieval beyond literal word overlap.

However, the amount of vocabulary expansion depends on the specific model.

Where it's useful: Specialized terminology, technical documentation, hybrid retrieval systems, and collections where important words need strong representation.

Dense and sparse retrieval aren't enemies. They often complement one another.

10. Hybrid Search — Combining Keywords and Meaning

This is one of the most practical techniques to understand.

Imagine an engineer asks:

"Why does the payment API return PAY_409 when retrying a transaction?"

Keyword search is valuable because PAY_409 is an exact error identifier.

Vector search is valuable because a document explaining duplicate payment attempts might not use the phrase "retrying a transaction."

Hybrid search combines retrieval signals from both approaches.

                 User Query
                     |
            +--------+--------+
            |                 |
            v                 v
       Keyword/BM25      Vector Search
            |                 |
            v                 v
       Exact Matches     Semantic Matches
            |                 |
            +--------+--------+
                     |
                     v
                Result Fusion
                     |
                     v
              Ranked Documents

Hybrid search often uses Reciprocal Rank Fusion (RRF), although weighted score fusion and other approaches are also possible.

The result is a combination of lexical precision and semantic matching.

Where it's useful: Enterprise knowledge bases, technical support, e-commerce search, and most applications containing both natural-language questions and exact terminology.

One caution: hybrid search isn't automatically better for every dataset. It should still be evaluated against a keyword-only and vector-only baseline.

11. Multi-Vector Search — One Document, Several Ways to Understand It

A document can contain multiple kinds of information.

Imagine a product manual:

Title: Smart Air Purifier X200

Summary: Troubleshooting and maintenance

Body: Detailed instructions

FAQ: Why is the red indicator blinking?

You could generate separate embeddings for its title, summary, and FAQ.

When someone asks:

"Why does my purifier keep blinking red?"

The FAQ embedding might provide a stronger match than an embedding representing the entire manual.

Multi-vector or multi-target search can search across these different representations and combine their results.

Where it's useful: Product catalogs, technical manuals, research papers, and documents with rich structure.

Note that different systems use the phrase multi-vector differently. Some mean multiple embeddings per document; others mean token-level vector representations or searches across multiple vector fields.

That brings us to the next technique.

12. Late-Interaction Search — A More Detailed Kind of Semantic Matching

A typical dense retrieval system represents a text chunk using one vector.

That is fast, but a single vector compresses a lot of information.

Late-interaction models, such as ColBERT, retain multiple token-level representations and compare them during scoring.

Consider:

"Does the premium plan allow offline access on Android?"

Two documents discuss:

  • Premium-plan benefits
  • Android offline-access restrictions

A single-vector search might consider both highly relevant.

A late-interaction model can compare the query's important components more directly against the document's token representations.

This can help distinguish documents that merely discuss related subjects from those matching the specific information need.

Where it's useful: Detailed technical documentation, specialized research collections, and retrieval tasks demanding higher precision.

The trade-off is increased indexing, storage, and computational complexity compared with ordinary single-vector retrieval.

13. Hierarchical and Parent–Child Retrieval — Finding a Small Answer Without Losing the Big Picture

Imagine uploading a 150-page insurance policy.

You split it into small chunks for retrieval.

One chunk says:

Coverage applies only after the waiting period.

That sentence might rank highly for a user's question.

But what waiting period? Which policy? Are there exceptions?

The answer might be in the surrounding paragraphs.

Parent–child retrieval addresses this by maintaining relationships between smaller searchable chunks and larger document sections.

       Insurance Policy
              |
              v
       Coverage Section
              |
       +------+------+
       |      |      |
     Chunk 1 Chunk 2 Chunk 3
              |
              v
       Relevant Match
              |
              v
     Retrieve Parent Section

The system searches small chunks for precision, then fetches the larger parent section to preserve context.

Example: Someone asks, "Does this policy cover dental surgery?"

The search finds a sentence about dental coverage, while the parent section explains exclusions and waiting periods.

Where it's useful: Legal agreements, medical guidelines, academic papers, insurance documents, and long technical manuals.

This is a reminder that good retrieval isn't only about finding the right sentence. Sometimes it's about retrieving enough surrounding information to interpret that sentence correctly.

Part 3: Search Beyond Individual Text Chunks

14. Knowledge Graph Search — When the Answer Depends on Relationships

Suppose you're building an assistant for a large organization.

Someone asks:

"Which projects are affected by the supplier that delayed our battery shipment?"

The information is scattered across several systems:

  • A shipment record identifies the delayed supplier.
  • A procurement document links the supplier to battery components.
  • A project database identifies products using those components.

A regular vector search might retrieve individual documents about the shipment.

But the question requires following relationships.

A knowledge graph represents entities and their connections.

Supplier A
    |
  supplies
    |
    v
Battery Component
    |
  used in
    |
    v
Project Alpha
    |
  affects
    |
    v
Product Launch

Graph-based retrieval can traverse these connections to gather the evidence needed for an answer.

Where it's useful: Supply chains, fraud detection, enterprise knowledge management, biomedical research, and other relationship-heavy domains.

GraphRAG is especially interesting when questions require combining relationships with information from the original documents.

15. Global Graph Search — Understanding an Entire Collection

Now consider a different question:

"What were the major problems reported across all our projects this quarter?"

This isn't asking for one fact or one relationship.

It asks for themes across a large collection.

Ordinary top-k vector search might retrieve a handful of relevant chunks but miss the overall picture.

Global graph-based approaches can organize information into communities, generate summaries of those communities, and use those summaries to answer broad questions.

For example, reports from different projects might collectively reveal recurring themes:

  • Supplier delays
  • Unclear ownership
  • Deployment failures
  • Changing requirements

Microsoft's GraphRAG global search is one example of this approach.

Where it's useful: Organization-wide analysis, research synthesis, risk reporting, and questions about trends across large document collections.

It can be more computationally expensive than standard retrieval, so it's generally better suited to questions that genuinely require collection-wide reasoning.

16. Multimodal Search — When the Question Isn't Just Text

Imagine someone uploads a picture of an unfamiliar machine part and asks:

"Find the installation manual for this component."

A text-only retrieval system may struggle because the user doesn't know the part's name.

Multimodal retrieval can work with images, text, and, depending on the architecture, audio or video.

A suitable vision-language embedding model can represent images and text in a compatible search space.

The system may then match the uploaded image against indexed product images or documents.

Another example:

A user uploads a screenshot of a dashboard graph and asks:

"Where is this trend discussed in our quarterly reports?"

A multimodal retrieval system can use the visual information to locate relevant pages or charts.

Where it's useful: Manufacturing, retail, visual document search, healthcare imaging workflows, and multimedia libraries.

Importantly, multimodal search requires suitable models and indexing pipelines. Simply storing images in a vector database doesn't automatically make them searchable by meaning.

17. Federated Search — Searching Across Multiple Systems

Real companies rarely store all their knowledge in one place.

Consider an employee asking:

"What was decided about the product launch, and what tasks are still pending?"

Relevant information might live in:

  • SharePoint documents
  • Meeting transcripts
  • Jira tickets
  • Internal databases
  • Team conversations

Federated retrieval searches across multiple sources and combines the results.

                 User Query
                     |
          +----------+----------+
          |          |          |
          v          v          v
        Jira     SharePoint    SQL DB
          |          |          |
          +----------+----------+
                     |
                     v
            Combine & Rerank
                     |
                     v
                LLM Answer

Where it's useful: Enterprise AI assistants, workflow automation platforms, consulting knowledge systems, and internal research tools.

Federated search has an additional challenge: different sources have different permissions, ranking systems, update frequencies, and document formats.

The retrieval layer must respect those differences rather than blindly merging everything.

18. Time-Aware Search — When Newer Information Matters

Imagine an employee asking:

"What is our current remote-work policy?"

Your documents contain:

2023 - Remote work permitted five days per week

2025 - Hybrid attendance policy introduced

2026 - Updated location-specific attendance rules

A semantic search might retrieve the 2023 policy because the wording closely matches the question.

But the relevant answer should come from the applicable, current policy.

Time-aware retrieval can combine relevance with signals such as effective dates, document versions, and recency.

Where it's useful: News assistants, regulations, pricing information, company policies, and fast-changing knowledge bases.

However, newer doesn't always mean more authoritative. An effective date, approval status, and source hierarchy may matter more than upload time.


Part 4: Making the Question Easier to Search

So far, we've focused mainly on how information is stored and retrieved.

But what if the user's question itself makes retrieval difficult?

That's where query transformation techniques help.

19. Query Rewriting — Turning Messy Questions into Searchable Ones

People rarely ask perfectly structured questions.

Imagine a conversation:

User: "Tell me about the premium plan."

Assistant: Explains the premium plan.

User: "Does it include that feature on mobile too?"

If the retrieval system searches only the last sentence, it may have no idea what "it" or "that feature" refers to.

A query-rewriting component can use the conversation history to create a standalone question.

For example:

"Does the premium subscription include offline document access on mobile devices?"

This revised question is much easier to search.

Where it's useful: Conversational RAG, support chatbots, multi-turn assistants, and systems with follow-up questions.

A good rewrite preserves the user's intent. A bad rewrite can accidentally introduce assumptions that were never stated.

20. Multi-Query Retrieval — Asking the Same Question in Different Ways

Imagine a user asks:

"How can I reduce cloud infrastructure costs?"

Relevant documents might discuss:

  • Cloud cost optimization
  • Reducing unused resources
  • Rightsizing virtual machines
  • Reserved capacity
  • Managing storage expenses

A single search query may miss some of these.

Multi-query retrieval generates alternative search formulations.

Original:
How can I reduce cloud infrastructure costs?

Alternative 1:
Cloud cost optimization strategies

Alternative 2:
Reducing unused compute resources

Alternative 3:
Rightsizing and reserved capacity guidance

Each query retrieves candidate documents, and the results are merged and deduplicated.

Where it's useful: Broad research questions, varied terminology, and document collections where the same concept appears under different names.

The trade-off is additional searches, which may increase latency and cost.

21. HyDE — Searching Using a Hypothetical Answer

HyDE stands for Hypothetical Document Embeddings.

The idea is surprisingly creative.

Instead of embedding only the user's question, an LLM first generates a hypothetical passage that might answer it.

Suppose someone asks:

"How do companies reduce customer churn?"

The model might generate a hypothetical passage:

Companies can reduce churn through improved onboarding, proactive customer support, personalized engagement, and analysis of cancellation patterns.

That passage is converted into an embedding and used to search the real document collection.

Why?

Because a passage written in the style of an answer may be closer in embedding space to actual explanatory documents than a short question is.

Where it's useful: Research-oriented RAG, short or abstract questions, and collections where questions and source passages have very different wording.

There's an important catch: the hypothetical passage is not evidence.

The system should retrieve and use actual documents, not present the generated hypothetical text as a verified answer.

22. Query Decomposition — Breaking Big Questions into Smaller Ones

Some questions are difficult because they ask several things at once.

Consider:

"How did our revenue change after the marketing campaign, and which customer segments contributed the most?"

This requires multiple pieces of information.

The retrieval system could decompose it into:

  1. What was revenue before the campaign?
  2. What was revenue after the campaign?
  3. Which customer segments generated revenue during that period?
  4. How did their contributions change?

Each sub-question can be answered using appropriate data sources.

The results are then combined.

Where it's useful: Financial analysis, research assistants, consulting applications, and multi-hop questions.

Query decomposition becomes particularly valuable when one retrieval call cannot reasonably gather all the evidence.

23. Step-Back Retrieval — Searching for the Bigger Idea First

Sometimes a question is so specific that direct retrieval misses the broader context.

Imagine:

"Why did our application slow down after increasing the number of workers?"

A direct search might focus narrowly on worker configuration.

A step-back approach asks a broader question first:

"What factors affect application throughput and latency when worker concurrency increases?"

That search might retrieve documentation explaining CPU contention, database connection limits, queue behavior, and shared-resource bottlenecks.

The original question can then be investigated with a better understanding of the underlying concepts.

Where it's useful: Technical troubleshooting, scientific questions, complex reasoning, and educational assistants.

Step-back retrieval doesn't replace specific evidence. It helps locate the general principles needed to interpret that evidence.

24. Self-Query Retrieval — Translating Natural Language into Search Conditions

Imagine asking a movie recommendation assistant:

"Find science-fiction movies released after 2020 with ratings above 8."

This contains two types of information.

The semantic concept:

Genre: Science fiction

And structured conditions:

Release year > 2020
Rating > 8

A self-query retriever uses a language model or another parser to turn natural language into a semantic query and structured filters.

The same idea applies to RAG.

For example:

"Find security reports written by the infrastructure team after March 2026."

The system extracts the topic, authoring team, and date constraint.

Where it's useful: Document repositories, product catalogs, research databases, and enterprise search systems with rich metadata.

As with SQL generation, the extracted fields and operators should be validated against an approved schema.

25. Agentic Retrieval — When Search Becomes an Investigation

A traditional RAG system often follows a fixed sequence:

Search -> Retrieve -> Generate

An agentic retrieval system can make decisions during the process.

Imagine a manager asks:

"Why did Project Orion miss its deadline?"

The system might:

  1. Search the project report.
  2. Discover a reference to delayed API integration.
  3. Search issue-tracking records for the integration problem.
  4. Find a dependency on an external vendor.
  5. Retrieve relevant meeting notes.
  6. Compare the evidence before generating an explanation.
        User Question
              |
              v
         Plan Search
              |
              v
        Retrieve Evidence
              |
              v
       Is More Evidence Needed?
          /           \
        Yes            No
         |              |
         v              v
    Search Again     Generate Answer

Where it's useful: Complex enterprise research, technical investigations, workflow assistants, and questions involving multiple systems.

Agentic retrieval can be powerful, but it can also become expensive, slow, and difficult to control.

Good implementations limit search iterations, validate tool permissions, retain citations, and have clear stopping conditions.

Not every question needs an agent.


Part 5: Finding Documents Is Only Half the Job

Suppose your search returns 30 documents.

How do you decide which five deserve to reach the LLM?

That's where result fusion, reranking, and diversity come in.

26. Reciprocal Rank Fusion — Combining Different Search Rankings

Let's say BM25 and vector search return these results:

Position BM25 Vector Search
1 Document A Document B
2 Document C Document A
3 Document B Document D

Document A ranks highly in both lists.

RRF combines the rankings using a formula like:

[ RRF(d)=\sum_i\frac{1}{k+rank_i(d)} ]

Here, rank is the document's position in each result list, and k is a smoothing constant, commonly 60.

The exact raw BM25 and vector similarity scores don't need to be directly comparable.

Instead, RRF uses positions in the ranked lists.

Example: Document A ranks first for keywords and second for vectors. That consistent relevance can give it an advantage in the fused results.

RRF is especially useful when combining different retrieval systems.

27. Reranking — Taking a Second, Closer Look

First-stage search is often designed to retrieve a reasonably broad set of relevant candidates quickly.

But its ranking may not be perfect.

A reranker evaluates those candidate documents more carefully against the original query.

Suppose the user asks:

"Can contractors claim travel reimbursement?"

The initial retrieval returns:

  • General employee travel reimbursement policy
  • Contractor payment policy
  • Contractor-specific travel reimbursement rules

A reranker may place the third document first because it directly addresses the question.

Cross-encoder rerankers are one common implementation. They examine the query and candidate document together to estimate relevance.

Retrieve Top 30 Documents
             |
             v
          Reranker
             |
             v
        Select Top 5
             |
             v
             LLM

Where it's useful: Enterprise RAG, compliance assistants, legal research, and applications where a small number of highly relevant passages matters more than raw retrieval speed.

The trade-off is extra inference cost and latency.

28. Maximum Marginal Relevance — Avoiding Five Versions of the Same Answer

Imagine asking:

"What are the main challenges of adopting AI in healthcare?"

A vector search returns five highly similar paragraphs about data privacy.

They're relevant, but collectively they provide a narrow perspective.

Maximum Marginal Relevance, or MMR, is a selection technique that balances relevance with diversity.

Instead of selecting only the most similar passages, it also considers how redundant each new passage is compared with those already selected.

A more diverse result set might cover:

  • Data privacy
  • Regulatory compliance
  • Model reliability
  • Clinical workflow integration
  • Infrastructure costs

Where it's useful: Research assistants, report generation, broad exploratory questions, and summarization across varied sources.

MMR shouldn't be used to manufacture diversity where the evidence doesn't support it. Its purpose is to reduce redundancy while preserving relevance.

29. Contextual Compression — Keeping Only What the Answer Needs

Sometimes a retrieved document is relevant but unnecessarily long.

Imagine searching a 30-page onboarding guide for:

"What documents must new employees submit?"

The relevant information occupies just two paragraphs.

Contextual compression attempts to extract or retain the portions most useful to the question.

It might use rules, relevance scoring, or an LLM-based extractor.

The final context contains the document requirements rather than the entire onboarding guide.

Where it's useful: Long-form documents, expensive context windows, high-volume RAG systems, and applications where reducing irrelevant information improves answer generation.

Compression introduces its own risk: removing an exception, qualifier, or condition can change the meaning.

For high-stakes documents, retain source references and consider fetching surrounding context before finalizing an answer.


So, Which Search Method Should You Actually Use?

After reading about all these approaches, it's tempting to add every technique to your RAG pipeline.

That's rarely the right move.

More components create more opportunities for latency, cost, maintenance overhead, and unexpected retrieval failures.

The better approach is to match the retrieval technique to the problem.

Your problem Good starting point
Searching exact codes, IDs, or names Exact / keyword search
Searching technical documentation BM25 or hybrid search
Understanding differently worded questions Dense vector search
Combining exact terms and semantic meaning Hybrid search
Searching within departments or regions Metadata-filtered retrieval
Answering questions from database tables Structured SQL retrieval
Preserving context in long documents Parent–child retrieval
Following connections between entities Graph-based retrieval
Understanding collection-wide themes Global GraphRAG
Searching images and documents together Multimodal retrieval
Handling vague queries Query rewriting or multi-query
Answering multi-part questions Query decomposition
Investigating complex problems Agentic retrieval
Improving result ordering Reranking
Reducing duplicate context MMR
Reducing unnecessary context Contextual compression

These are starting points, not universal rules.

Actual performance depends on the corpus, the embedding model, the search engine, the questions users ask, and the quality of your documents.

What Would I Build for a Production RAG Application?

For a typical enterprise document assistant, I wouldn't begin with GraphRAG, autonomous search agents, and five different embedding models.

I'd start with a smaller architecture that is easy to understand, measure, and improve.

During document ingestion

         Documents / PDFs / URLs
                    |
                    v
          Extract & Clean Text
                    |
                    v
          Structure-Aware Chunking
                    |
                    v
           Attach Metadata
                    |
             +------+------+
             |             |
             v             v
       Text Index     Embedding Model
             |             |
             v             v
         BM25          Vector Index
             |             |
             +------+------+
                    |
                    v
              Data Storage

When a user asks a question

              User Question
                    |
                    v
            Authenticate User
                    |
                    v
             Query Rewriting
                (if needed)
                    |
                    v
        Authorization & Filters
                    |
           +--------+--------+
           |                 |
           v                 v
       BM25 Search       Vector Search
           |                 |
           +--------+--------+
                    |
                    v
            Rank Fusion (RRF)
                    |
                    v
                Reranking
                    |
                    v
          Parent Context Retrieval
               (if needed)
                    |
                    v
              Context Assembly
                    |
                    v
              LLM Generation
                    |
                    v
          Answer with Citations

For implementation, PostgreSQL + pgvector can be a practical starting point.

PostgreSQL can handle application data, document metadata, access relationships, and native full-text search. pgvector adds vector similarity search.

Depending on scale and requirements, dedicated search platforms such as Elasticsearch, OpenSearch, Azure AI Search, Qdrant, Vespa, or Weaviate may also make sense.

The important thing is that a database product is not a retrieval strategy. It's part of the infrastructure used to execute one.

How I'd improve it over time

Start with a simple retrieval baseline and test it.

If users ask about exact error codes, strengthen lexical search.

If they paraphrase document terminology, evaluate semantic retrieval.

If relevant chunks appear but rank too low, try reranking.

If the answer is missing important surrounding details, inspect chunking and parent–child retrieval.

If a question involves multiple facts or systems, consider decomposition or agentic retrieval.

And if users ask broad questions about patterns across an entire organization, graph-based approaches may be worth exploring.

Each addition should solve a measured problem.

One More Thing: How Do You Know Retrieval Is Working?

A search result can look convincing without being correct.

That's why retrieval evaluation matters.

Build a test set of realistic user questions and identify which documents or passages actually contain the required evidence.

Then measure metrics such as:

Recall@k: Did the system retrieve the relevant evidence among its top k results?

Precision@k: How much of the retrieved set is actually relevant?

MRR (Mean Reciprocal Rank): How early does the first relevant result appear?

nDCG: Are highly relevant documents appearing near the top, taking graded relevance into account?

And beyond retrieval, evaluate whether the final answer is faithful to its sources, whether citations support its claims, and whether the system can recognize when evidence is insufficient.

Measure latency and cost alongside quality.

An approach that improves retrieval slightly but makes every user wait several additional seconds may not be the right trade-off.

Remember, an LLM can only reason over the information it actually receives.

And even good retrieval cannot guarantee a correct answer if the information is outdated, contradictory, or incomplete.


Final Thoughts: Search Is More Than Finding Similar Text

The biggest lesson I've taken from studying RAG retrieval is that relevance has different meanings in different situations.

Sometimes the most relevant result is the document containing an exact error code.

Sometimes it's a paragraph expressing the same idea in completely different words.

Sometimes it's a record that matches a date and customer ID.

Sometimes the answer isn't in any single document at all — it exists across relationships, conversations, and multiple systems.

That's why there is no single search algorithm that solves every RAG problem.

Keyword search gives us precision.

Semantic search helps us bridge differences in language.

Hybrid search brings those strengths together.

Graph retrieval helps us follow relationships.

Query transformation helps us ask better questions.

Reranking helps us decide which evidence deserves attention.

And agentic retrieval allows a system to investigate when one search isn't enough.

The real engineering challenge isn't learning the names of all these techniques.

It's knowing which retrieval problem you're trying to solve — and choosing the simplest method that solves it reliably.

Because in a RAG application, having the answer somewhere in your database is only the beginning.

The system still has to find it.


Further Reading

For anyone interested in exploring the technical implementations:

Thanks for reading. If you're building RAG systems, I'd be interested to know which retrieval approach has worked best for your use case.

About the author

  • I’m a AI engineer who enjoys exploring ideas and making complex topics easier to understand. I write about technology, things I build, and lessons I learn—with practical examples and honest perspectives.

No comments yet. Be the first to comment.