<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0"><channel><title><![CDATA[The AI Notebook]]></title><description><![CDATA[The AI Notebook]]></description><link>https://ainotebook.hashnode.dev</link><generator>RSS for Node</generator><lastBuildDate>Sat, 21 Nov 2026 17:18:46 GMT</lastBuildDate><atom:link href="https://ainotebook.hashnode.dev/rss.xml" rel="self" type="application/rss+xml"/><language><![CDATA[en]]></language><ttl>60</ttl><item><title><![CDATA[The Answer Was in the Database. Why Couldn't RAG Find It?]]></title><description><![CDATA[From keyword matching to semantic, hybrid, graph, and agentic search — understanding how AI applications find the right information. Imagine building an AI…]]></description><link>https://ainotebook.hashnode.dev/rag-search-retrieval-techniques-explained</link><guid isPermaLink="true">https://ainotebook.hashnode.dev/rag-search-retrieval-techniques-explained</guid><category><![CDATA[RAG  Artificial Intelligence  Machine Learning  Vector Database  LLM]]></category><dc:creator><![CDATA[Varshitha]]></dc:creator><pubDate>Sun, 15 Nov 2026 16:44:27 GMT</pubDate><content:encoded><![CDATA[<p><em>From keyword matching to semantic, hybrid, graph, and agentic search — understanding how AI applications find the right information.</em></p>
<p>Imagine building an AI assistant for an online shopping platform.</p>
<p>A customer asks:</p>
<p><strong>"I received a damaged product. Can I get my money back?"</strong></p>
<p>Somewhere in your database, there's a document that says:</p>
<blockquote>
<p>Customers may request a full refund for defective or damaged items within 30 days of delivery.</p>
</blockquote>
<p>The answer exists. The document is stored correctly. Your embedding pipeline ran successfully.</p>
<p>Yet your AI assistant responds:</p>
<p><em>"I couldn't find any information about that."</em></p>
<p>What went wrong?</p>
<p>Not necessarily the LLM. Not necessarily the prompt.</p>
<p><strong>The problem could be retrieval.</strong></p>
<p>We spend a lot of time discussing which LLM to use, how to write better prompts, or which vector database performs best. But there's another question worth asking:</p>
<p><em>How does our application actually find the information the LLM needs?</em></p>
<p>That's where search strategies come in.</p>
<p>And there are far more of them than simply choosing between keyword search and vector search.</p>
<p>Let's explore them, starting with traditional search and gradually moving into the techniques used in more sophisticated RAG systems.</p>
<hr />
<h2>First, Where Does Search Fit into RAG?</h2>
<p>RAG stands for <strong>Retrieval-Augmented Generation</strong>.</p>
<p>Instead of expecting an LLM to know everything, we retrieve relevant information from external sources and provide that information as context before generating a response.</p>
<p>A typical pipeline looks like this:</p>
<pre><code class="language-text">          User Question
                |
                v
         Query Processing
                |
                v
         Search / Retrieval
                |
                v
          Relevant Chunks
                |
                v
        Reranking / Filtering
                |
                v
         Context Assembly
                |
                v
          LLM Generation
                |
                v
          Final Response
</code></pre>
<p>The interesting part is the retrieval stage.</p>
<p>Consider three questions:</p>
<ul>
<li>"What is the refund policy?"</li>
<li>"Can I return a broken item?"</li>
<li>"What does policy RF-2026-04 say?"</li>
</ul>
<p>All three relate to refunds, but they require slightly different search behavior.</p>
<p>The first is a straightforward information request. The second relies heavily on understanding meaning. The third contains an exact identifier.</p>
<p>One search technique may handle all three reasonably well, but different techniques have different strengths.</p>
<p>One important distinction before we begin: <strong>not everything discussed below is a standalone search algorithm.</strong></p>
<p>Some techniques retrieve documents. Others improve the query, combine search results, navigate related information, or rerank what has already been found.</p>
<p>They're all part of designing a strong RAG retrieval system.</p>
<h1>Part 1: Traditional Search — Finding Words, Phrases, and Records</h1>
<h2>1. Exact Match Search — When Every Character Matters</h2>
<p>Let's start with the simplest approach.</p>
<p>Exact match search retrieves information by matching a specified value.</p>
<p>Suppose an engineer asks:</p>
<p><strong>"Find the documentation for error AUTH_403."</strong></p>
<p>Your database contains:</p>
<pre><code class="language-text">AUTH_401 - Invalid authentication credentials
AUTH_403 - Insufficient user permissions
AUTH_429 - Too many authentication attempts
</code></pre>
<p>An exact match on <code>AUTH_403</code> returns the second record.</p>
<p>No embeddings. No language model. No semantic interpretation.</p>
<p>Just a precise match.</p>
<p><strong>Where it's useful:</strong> Error codes, transaction IDs, order numbers, document IDs, product SKUs, and other identifiers.</p>
<p><strong>Where it struggles:</strong> Someone searching for "permission denied error" may not know the exact code.</p>
<p>This is one reason semantic search shouldn't automatically replace traditional search.</p>
<p>Sometimes understanding <em>exactly what someone typed</em> is more important than understanding what they meant.</p>
<h2>2. Full-Text Search — Searching Across the Words in a Document</h2>
<p>Now imagine an employee searches:</p>
<p><strong>"Annual leave policy for new employees"</strong></p>
<p>Your knowledge base contains hundreds of HR documents.</p>
<p>A basic string-matching approach might look for that exact phrase. Full-text search goes further.</p>
<p>It typically processes documents into searchable terms using techniques such as tokenization, normalization, and sometimes stemming or stop-word removal.</p>
<p>For example:</p>
<pre><code class="language-text">Document:
"New employees become eligible for annual leave
after completing their probation period."

Search terms:
annual, leave, new, employee
</code></pre>
<p>A full-text search engine can find the relevant document even when the query doesn't match the original sentence word for word.</p>
<p>Common technologies include PostgreSQL full-text search, Elasticsearch, OpenSearch, and Apache Solr.</p>
<p><strong>Where it's useful:</strong> Policy documents, knowledge bases, documentation portals, legal text, and large collections of articles.</p>
<p>Full-text search is a family of techniques. How results are ranked depends on the engine and configuration.</p>
<p>That brings us to BM25.</p>
<h2>3. BM25 — Not Just Finding Documents, but Ranking Them</h2>
<p>Suppose you search:</p>
<p><strong>"Refund policy for damaged products"</strong></p>
<p>Three documents match:</p>
<p><strong>Document A:</strong> Mentions refunds once in a 20-page report.</p>
<p><strong>Document B:</strong> Has a dedicated section explaining refunds for damaged products.</p>
<p><strong>Document C:</strong> Repeats the word "refund" dozens of times but focuses on an unrelated issue.</p>
<p>Which should come first?</p>
<p>A simple keyword match doesn't necessarily provide a useful ranking.</p>
<p>BM25 is a widely used lexical ranking algorithm designed to estimate how relevant a document is to a query.</p>
<p>It considers signals including:</p>
<ul>
<li>How frequently a search term appears in the document.</li>
<li>How rare or informative that term is across the collection.</li>
<li>The document's length, with adjustments so repetition doesn't dominate indefinitely.</li>
</ul>
<p>In our example, Document B may rank highest because its content is more focused on the searched terms.</p>
<p>BM25 doesn't truly understand meaning the way a semantic model attempts to. It's a clever, mathematically grounded way to rank textual matches.</p>
<p><strong>Where it's useful:</strong> Technical documentation, support tickets, research repositories, and product catalogs.</p>
<p>A small implementation detail worth knowing: PostgreSQL has native full-text ranking functions such as <code>ts_rank</code> and <code>ts_rank_cd</code>, but they are not the same as BM25. A BM25 implementation requires an appropriate search engine or extension.</p>
<h2>4. Boolean, Phrase, and Proximity Search — Giving Search More Rules</h2>
<p>Not every search needs to be open-ended.</p>
<p>Sometimes we know exactly which conditions the result should satisfy.</p>
<p>Imagine a legal assistant searching contracts for:</p>
<p><strong>"Confidentiality agreements that mention employee data but exclude external vendors."</strong></p>
<p>Boolean search allows combinations such as:</p>
<pre><code class="language-text">confidentiality AND employee AND NOT vendor
</code></pre>
<p>Phrase search can look for words appearing together:</p>
<pre><code class="language-text">"confidential information"
</code></pre>
<p>Proximity search goes further by finding words that occur close to one another.</p>
<p>For instance, a document where "security" appears near "incident" may be more relevant than one where those words appear 15 pages apart.</p>
<p><strong>Where it's useful:</strong> Legal research, compliance, scientific literature, and investigative search.</p>
<p>The exact syntax varies by search engine, but the idea is the same: the searcher can express more precise constraints.</p>
<h2>5. Fuzzy Search — Because Humans Make Typos</h2>
<p>Think about how people actually search.</p>
<p>A customer types:</p>
<p><strong>"How to cancle my subcription?"</strong></p>
<p>The documentation uses:</p>
<p>"How to cancel your subscription."</p>
<p>The user has misspelled both <em>cancel</em> and <em>subscription</em>.</p>
<p>A fuzzy search implementation can find similar terms using measures such as edit distance — the number of character changes needed to transform one word into another.</p>
<pre><code class="language-text">cancle -&gt; cancel

subcription -&gt; subscription
</code></pre>
<p>Autocomplete and prefix search are closely related search features, although they operate differently.</p>
<p>For example, someone typing <code>subscr</code> might see suggestions for "subscription," "subscriber," and "subscribe."</p>
<p><strong>Where it's useful:</strong> Customer support portals, product search, names, and applications with frequent typing errors.</p>
<p>An important distinction: fuzzy search handles textual variation. It doesn't inherently understand that "money back" and "refund" have similar meanings.</p>
<p>That's a different problem.</p>
<h2>6. Metadata Filtering — Searching Only Where You're Supposed To</h2>
<p>Imagine an organization with 50,000 policy documents.</p>
<p>An employee asks:</p>
<p><strong>"What is the leave policy for Indian employees in 2026?"</strong></p>
<p>A semantic search over every document might retrieve policies from the US, UK, or previous years.</p>
<p>Even if those documents are conceptually similar, they're not necessarily relevant.</p>
<p>Metadata filtering helps restrict the search.</p>
<pre><code class="language-json">{
  "country": "India",
  "year": 2026,
  "document_type": "HR Policy"
}
</code></pre>
<p>The retrieval system searches the appropriate subset of documents.</p>
<p>Metadata might include department, tenant, document owner, creation date, product category, access permissions, or document version.</p>
<p><strong>Where it's useful:</strong> Enterprise RAG, multi-tenant SaaS platforms, healthcare records, financial documents, and region-specific policies.</p>
<p>A particularly important detail: metadata filters must also enforce access control. A document being semantically relevant does not mean the user is authorized to read it.</p>
<h2>7. Structured Database Search — Sometimes the Answer Is in a Table</h2>
<p>Not every RAG question should be answered by retrieving text chunks.</p>
<p>Consider:</p>
<p><strong>"How many orders were cancelled yesterday?"</strong></p>
<p>Your PostgreSQL database contains an <code>orders</code> table with order statuses and timestamps.</p>
<p>Rather than searching embeddings, the application could execute an authorized SQL query:</p>
<pre><code class="language-sql">SELECT COUNT(*)
FROM orders
WHERE status = 'cancelled'
  AND cancelled_at &gt;= :day_start
  AND cancelled_at &lt; :day_end;
</code></pre>
<p>The query parameters represent the relevant day's boundaries.</p>
<p>This retrieves the factual count directly.</p>
<p>An LLM may help interpret the request, but the actual calculation belongs in the database.</p>
<p><strong>Where it's useful:</strong> Sales dashboards, finance assistants, inventory management, customer account systems, and analytics.</p>
<p>For production applications, database access should use read-only permissions where appropriate, validated query patterns, and user-specific authorization.</p>
<p>A vector database is not a replacement for SQL.</p>
<h1>Part 2: Semantic and Vector Search — Finding Meaning, Not Just Words</h1>
<h2>8. Dense Vector Search — When Different Words Mean Similar Things</h2>
<p>Let's return to our original example.</p>
<p>The user asks:</p>
<p><strong>"I received a damaged product. Can I get my money back?"</strong></p>
<p>The document says:</p>
<p>"Customers are eligible for a full refund on defective items."</p>
<p>Notice the vocabulary:</p>
<pre><code class="language-text">User:      damaged product, money back

Document:  defective item, full refund
</code></pre>
<p>The words are different. The meanings overlap.</p>
<p>This is where dense vector search becomes useful.</p>
<p>An embedding model converts text into a numerical representation called an embedding.</p>
<p>Conceptually:</p>
<pre><code class="language-text">"I want my money back"
          |
          v
     Embedding Model
          |
          v
 [0.12, -0.36, 0.81, ...]
</code></pre>
<p>Documents are converted into embeddings too.</p>
<p>The system compares the query vector with stored document vectors and retrieves the closest matches.</p>
<p>Similarity can be measured using cosine similarity, dot product, or Euclidean distance, depending on the model and system.</p>
<p><strong>Where it's useful:</strong> Conversational assistants, RAG chatbots, knowledge discovery, and questions where users don't know the exact terminology.</p>
<p>But vector search has limitations.</p>
<p>It can confuse closely related concepts, retrieve vaguely similar passages, or overlook exact identifiers such as <code>RF-2026-04</code>.</p>
<p>Semantic similarity is useful evidence of relevance — not proof that a document contains the answer.</p>
<h3>A Quick Detour: Exact vs. Approximate Vector Search</h3>
<p>When discussing vector search, you'll often hear these terms:</p>
<p><strong>KNN (k-nearest neighbors):</strong> Find the closest vectors based on a similarity or distance metric.</p>
<p><strong>Exact KNN:</strong> Evaluate the candidates exhaustively to find the nearest matches.</p>
<p><strong>ANN (approximate nearest neighbors):</strong> Use specialized indexes to retrieve likely nearest neighbors faster, potentially missing some exact matches.</p>
<p>Two common ANN indexing techniques are:</p>
<ul>
<li><strong>HNSW:</strong> Uses a layered graph structure to navigate toward nearby vectors efficiently.</li>
<li><strong>IVF / IVFFlat:</strong> Groups vectors into partitions and searches selected partitions.</li>
</ul>
<p>These aren't alternatives to semantic search. They're different ways of executing vector retrieval.</p>
<p>For example, a small HR knowledge base might work perfectly well with exact vector search. A system containing millions of embeddings may benefit from ANN indexing.</p>
<p>Libraries such as <code>pgvector</code> support both exact search and approximate indexes.</p>
<h2>9. Sparse Vector Search — Giving Important Terms More Weight</h2>
<p>Dense vectors capture broader semantic information, but there's another family of vector representations: sparse vectors.</p>
<p>Sparse vectors contain many zero-valued dimensions, with relatively few nonzero values representing important terms or features.</p>
<p>Traditional sparse retrieval methods include TF-IDF and BM25-style lexical representations.</p>
<p>Modern learned sparse retrieval methods, such as SPLADE, use neural models to produce weighted term representations and may expand relevant vocabulary.</p>
<p>Imagine a customer asks:</p>
<p><strong>"How can I request parental leave?"</strong></p>
<p>Your HR documents discuss:</p>
<ul>
<li>Parental leave</li>
<li>Maternity leave</li>
<li>Paternity leave</li>
<li>Family leave benefits</li>
</ul>
<p>A learned sparse model may assign weights to related terms, helping retrieval beyond literal word overlap.</p>
<p>However, the amount of vocabulary expansion depends on the specific model.</p>
<p><strong>Where it's useful:</strong> Specialized terminology, technical documentation, hybrid retrieval systems, and collections where important words need strong representation.</p>
<p>Dense and sparse retrieval aren't enemies. They often complement one another.</p>
<h2>10. Hybrid Search — Combining Keywords and Meaning</h2>
<p>This is one of the most practical techniques to understand.</p>
<p>Imagine an engineer asks:</p>
<p><strong>"Why does the payment API return PAY_409 when retrying a transaction?"</strong></p>
<p>Keyword search is valuable because <code>PAY_409</code> is an exact error identifier.</p>
<p>Vector search is valuable because a document explaining duplicate payment attempts might not use the phrase "retrying a transaction."</p>
<p>Hybrid search combines retrieval signals from both approaches.</p>
<pre><code class="language-text">                 User Query
                     |
            +--------+--------+
            |                 |
            v                 v
       Keyword/BM25      Vector Search
            |                 |
            v                 v
       Exact Matches     Semantic Matches
            |                 |
            +--------+--------+
                     |
                     v
                Result Fusion
                     |
                     v
              Ranked Documents
</code></pre>
<p>Hybrid search often uses Reciprocal Rank Fusion (RRF), although weighted score fusion and other approaches are also possible.</p>
<p>The result is a combination of lexical precision and semantic matching.</p>
<p><strong>Where it's useful:</strong> Enterprise knowledge bases, technical support, e-commerce search, and most applications containing both natural-language questions and exact terminology.</p>
<p>One caution: hybrid search isn't automatically better for every dataset. It should still be evaluated against a keyword-only and vector-only baseline.</p>
<h2>11. Multi-Vector Search — One Document, Several Ways to Understand It</h2>
<p>A document can contain multiple kinds of information.</p>
<p>Imagine a product manual:</p>
<pre><code class="language-text">Title: Smart Air Purifier X200

Summary: Troubleshooting and maintenance

Body: Detailed instructions

FAQ: Why is the red indicator blinking?
</code></pre>
<p>You could generate separate embeddings for its title, summary, and FAQ.</p>
<p>When someone asks:</p>
<p><strong>"Why does my purifier keep blinking red?"</strong></p>
<p>The FAQ embedding might provide a stronger match than an embedding representing the entire manual.</p>
<p>Multi-vector or multi-target search can search across these different representations and combine their results.</p>
<p><strong>Where it's useful:</strong> Product catalogs, technical manuals, research papers, and documents with rich structure.</p>
<p>Note that different systems use the phrase <em>multi-vector</em> differently. Some mean multiple embeddings per document; others mean token-level vector representations or searches across multiple vector fields.</p>
<p>That brings us to the next technique.</p>
<h2>12. Late-Interaction Search — A More Detailed Kind of Semantic Matching</h2>
<p>A typical dense retrieval system represents a text chunk using one vector.</p>
<p>That is fast, but a single vector compresses a lot of information.</p>
<p>Late-interaction models, such as ColBERT, retain multiple token-level representations and compare them during scoring.</p>
<p>Consider:</p>
<p><strong>"Does the premium plan allow offline access on Android?"</strong></p>
<p>Two documents discuss:</p>
<ul>
<li>Premium-plan benefits</li>
<li>Android offline-access restrictions</li>
</ul>
<p>A single-vector search might consider both highly relevant.</p>
<p>A late-interaction model can compare the query's important components more directly against the document's token representations.</p>
<p>This can help distinguish documents that merely discuss related subjects from those matching the specific information need.</p>
<p><strong>Where it's useful:</strong> Detailed technical documentation, specialized research collections, and retrieval tasks demanding higher precision.</p>
<p>The trade-off is increased indexing, storage, and computational complexity compared with ordinary single-vector retrieval.</p>
<h2>13. Hierarchical and Parent–Child Retrieval — Finding a Small Answer Without Losing the Big Picture</h2>
<p>Imagine uploading a 150-page insurance policy.</p>
<p>You split it into small chunks for retrieval.</p>
<p>One chunk says:</p>
<blockquote>
<p>Coverage applies only after the waiting period.</p>
</blockquote>
<p>That sentence might rank highly for a user's question.</p>
<p>But what waiting period? Which policy? Are there exceptions?</p>
<p>The answer might be in the surrounding paragraphs.</p>
<p>Parent–child retrieval addresses this by maintaining relationships between smaller searchable chunks and larger document sections.</p>
<pre><code class="language-text">       Insurance Policy
              |
              v
       Coverage Section
              |
       +------+------+
       |      |      |
     Chunk 1 Chunk 2 Chunk 3
              |
              v
       Relevant Match
              |
              v
     Retrieve Parent Section
</code></pre>
<p>The system searches small chunks for precision, then fetches the larger parent section to preserve context.</p>
<p><strong>Example:</strong> Someone asks, "Does this policy cover dental surgery?"</p>
<p>The search finds a sentence about dental coverage, while the parent section explains exclusions and waiting periods.</p>
<p><strong>Where it's useful:</strong> Legal agreements, medical guidelines, academic papers, insurance documents, and long technical manuals.</p>
<p>This is a reminder that good retrieval isn't only about finding the right sentence. Sometimes it's about retrieving enough surrounding information to interpret that sentence correctly.</p>
<h1>Part 3: Search Beyond Individual Text Chunks</h1>
<h2>14. Knowledge Graph Search — When the Answer Depends on Relationships</h2>
<p>Suppose you're building an assistant for a large organization.</p>
<p>Someone asks:</p>
<p><strong>"Which projects are affected by the supplier that delayed our battery shipment?"</strong></p>
<p>The information is scattered across several systems:</p>
<ul>
<li>A shipment record identifies the delayed supplier.</li>
<li>A procurement document links the supplier to battery components.</li>
<li>A project database identifies products using those components.</li>
</ul>
<p>A regular vector search might retrieve individual documents about the shipment.</p>
<p>But the question requires following relationships.</p>
<p>A knowledge graph represents entities and their connections.</p>
<pre><code class="language-text">Supplier A
    |
  supplies
    |
    v
Battery Component
    |
  used in
    |
    v
Project Alpha
    |
  affects
    |
    v
Product Launch
</code></pre>
<p>Graph-based retrieval can traverse these connections to gather the evidence needed for an answer.</p>
<p><strong>Where it's useful:</strong> Supply chains, fraud detection, enterprise knowledge management, biomedical research, and other relationship-heavy domains.</p>
<p>GraphRAG is especially interesting when questions require combining relationships with information from the original documents.</p>
<h2>15. Global Graph Search — Understanding an Entire Collection</h2>
<p>Now consider a different question:</p>
<p><strong>"What were the major problems reported across all our projects this quarter?"</strong></p>
<p>This isn't asking for one fact or one relationship.</p>
<p>It asks for themes across a large collection.</p>
<p>Ordinary top-k vector search might retrieve a handful of relevant chunks but miss the overall picture.</p>
<p>Global graph-based approaches can organize information into communities, generate summaries of those communities, and use those summaries to answer broad questions.</p>
<p>For example, reports from different projects might collectively reveal recurring themes:</p>
<ul>
<li>Supplier delays</li>
<li>Unclear ownership</li>
<li>Deployment failures</li>
<li>Changing requirements</li>
</ul>
<p>Microsoft's GraphRAG global search is one example of this approach.</p>
<p><strong>Where it's useful:</strong> Organization-wide analysis, research synthesis, risk reporting, and questions about trends across large document collections.</p>
<p>It can be more computationally expensive than standard retrieval, so it's generally better suited to questions that genuinely require collection-wide reasoning.</p>
<h2>16. Multimodal Search — When the Question Isn't Just Text</h2>
<p>Imagine someone uploads a picture of an unfamiliar machine part and asks:</p>
<p><strong>"Find the installation manual for this component."</strong></p>
<p>A text-only retrieval system may struggle because the user doesn't know the part's name.</p>
<p>Multimodal retrieval can work with images, text, and, depending on the architecture, audio or video.</p>
<p>A suitable vision-language embedding model can represent images and text in a compatible search space.</p>
<p>The system may then match the uploaded image against indexed product images or documents.</p>
<p>Another example:</p>
<p>A user uploads a screenshot of a dashboard graph and asks:</p>
<p><strong>"Where is this trend discussed in our quarterly reports?"</strong></p>
<p>A multimodal retrieval system can use the visual information to locate relevant pages or charts.</p>
<p><strong>Where it's useful:</strong> Manufacturing, retail, visual document search, healthcare imaging workflows, and multimedia libraries.</p>
<p>Importantly, multimodal search requires suitable models and indexing pipelines. Simply storing images in a vector database doesn't automatically make them searchable by meaning.</p>
<h2>17. Federated Search — Searching Across Multiple Systems</h2>
<p>Real companies rarely store all their knowledge in one place.</p>
<p>Consider an employee asking:</p>
<p><strong>"What was decided about the product launch, and what tasks are still pending?"</strong></p>
<p>Relevant information might live in:</p>
<ul>
<li>SharePoint documents</li>
<li>Meeting transcripts</li>
<li>Jira tickets</li>
<li>Internal databases</li>
<li>Team conversations</li>
</ul>
<p>Federated retrieval searches across multiple sources and combines the results.</p>
<pre><code class="language-text">                 User Query
                     |
          +----------+----------+
          |          |          |
          v          v          v
        Jira     SharePoint    SQL DB
          |          |          |
          +----------+----------+
                     |
                     v
            Combine &amp; Rerank
                     |
                     v
                LLM Answer
</code></pre>
<p><strong>Where it's useful:</strong> Enterprise AI assistants, workflow automation platforms, consulting knowledge systems, and internal research tools.</p>
<p>Federated search has an additional challenge: different sources have different permissions, ranking systems, update frequencies, and document formats.</p>
<p>The retrieval layer must respect those differences rather than blindly merging everything.</p>
<h2>18. Time-Aware Search — When Newer Information Matters</h2>
<p>Imagine an employee asking:</p>
<p><strong>"What is our current remote-work policy?"</strong></p>
<p>Your documents contain:</p>
<pre><code class="language-text">2023 - Remote work permitted five days per week

2025 - Hybrid attendance policy introduced

2026 - Updated location-specific attendance rules
</code></pre>
<p>A semantic search might retrieve the 2023 policy because the wording closely matches the question.</p>
<p>But the relevant answer should come from the applicable, current policy.</p>
<p>Time-aware retrieval can combine relevance with signals such as effective dates, document versions, and recency.</p>
<p><strong>Where it's useful:</strong> News assistants, regulations, pricing information, company policies, and fast-changing knowledge bases.</p>
<p>However, <em>newer</em> doesn't always mean <em>more authoritative</em>. An effective date, approval status, and source hierarchy may matter more than upload time.</p>
<hr />
<h1>Part 4: Making the Question Easier to Search</h1>
<p>So far, we've focused mainly on how information is stored and retrieved.</p>
<p>But what if the user's question itself makes retrieval difficult?</p>
<p>That's where query transformation techniques help.</p>
<h2>19. Query Rewriting — Turning Messy Questions into Searchable Ones</h2>
<p>People rarely ask perfectly structured questions.</p>
<p>Imagine a conversation:</p>
<p><strong>User:</strong> "Tell me about the premium plan."</p>
<p><strong>Assistant:</strong> Explains the premium plan.</p>
<p><strong>User:</strong> "Does it include that feature on mobile too?"</p>
<p>If the retrieval system searches only the last sentence, it may have no idea what "it" or "that feature" refers to.</p>
<p>A query-rewriting component can use the conversation history to create a standalone question.</p>
<p>For example:</p>
<blockquote>
<p>"Does the premium subscription include offline document access on mobile devices?"</p>
</blockquote>
<p>This revised question is much easier to search.</p>
<p><strong>Where it's useful:</strong> Conversational RAG, support chatbots, multi-turn assistants, and systems with follow-up questions.</p>
<p>A good rewrite preserves the user's intent. A bad rewrite can accidentally introduce assumptions that were never stated.</p>
<h2>20. Multi-Query Retrieval — Asking the Same Question in Different Ways</h2>
<p>Imagine a user asks:</p>
<p><strong>"How can I reduce cloud infrastructure costs?"</strong></p>
<p>Relevant documents might discuss:</p>
<ul>
<li>Cloud cost optimization</li>
<li>Reducing unused resources</li>
<li>Rightsizing virtual machines</li>
<li>Reserved capacity</li>
<li>Managing storage expenses</li>
</ul>
<p>A single search query may miss some of these.</p>
<p>Multi-query retrieval generates alternative search formulations.</p>
<pre><code class="language-text">Original:
How can I reduce cloud infrastructure costs?

Alternative 1:
Cloud cost optimization strategies

Alternative 2:
Reducing unused compute resources

Alternative 3:
Rightsizing and reserved capacity guidance
</code></pre>
<p>Each query retrieves candidate documents, and the results are merged and deduplicated.</p>
<p><strong>Where it's useful:</strong> Broad research questions, varied terminology, and document collections where the same concept appears under different names.</p>
<p>The trade-off is additional searches, which may increase latency and cost.</p>
<h2>21. HyDE — Searching Using a Hypothetical Answer</h2>
<p>HyDE stands for <strong>Hypothetical Document Embeddings</strong>.</p>
<p>The idea is surprisingly creative.</p>
<p>Instead of embedding only the user's question, an LLM first generates a hypothetical passage that might answer it.</p>
<p>Suppose someone asks:</p>
<p><strong>"How do companies reduce customer churn?"</strong></p>
<p>The model might generate a hypothetical passage:</p>
<blockquote>
<p>Companies can reduce churn through improved onboarding, proactive customer support, personalized engagement, and analysis of cancellation patterns.</p>
</blockquote>
<p>That passage is converted into an embedding and used to search the real document collection.</p>
<p>Why?</p>
<p>Because a passage written in the style of an answer may be closer in embedding space to actual explanatory documents than a short question is.</p>
<p><strong>Where it's useful:</strong> Research-oriented RAG, short or abstract questions, and collections where questions and source passages have very different wording.</p>
<p>There's an important catch: the hypothetical passage is not evidence.</p>
<p>The system should retrieve and use actual documents, not present the generated hypothetical text as a verified answer.</p>
<h2>22. Query Decomposition — Breaking Big Questions into Smaller Ones</h2>
<p>Some questions are difficult because they ask several things at once.</p>
<p>Consider:</p>
<p><strong>"How did our revenue change after the marketing campaign, and which customer segments contributed the most?"</strong></p>
<p>This requires multiple pieces of information.</p>
<p>The retrieval system could decompose it into:</p>
<ol>
<li>What was revenue before the campaign?</li>
<li>What was revenue after the campaign?</li>
<li>Which customer segments generated revenue during that period?</li>
<li>How did their contributions change?</li>
</ol>
<p>Each sub-question can be answered using appropriate data sources.</p>
<p>The results are then combined.</p>
<p><strong>Where it's useful:</strong> Financial analysis, research assistants, consulting applications, and multi-hop questions.</p>
<p>Query decomposition becomes particularly valuable when one retrieval call cannot reasonably gather all the evidence.</p>
<h2>23. Step-Back Retrieval — Searching for the Bigger Idea First</h2>
<p>Sometimes a question is so specific that direct retrieval misses the broader context.</p>
<p>Imagine:</p>
<p><strong>"Why did our application slow down after increasing the number of workers?"</strong></p>
<p>A direct search might focus narrowly on worker configuration.</p>
<p>A step-back approach asks a broader question first:</p>
<p><strong>"What factors affect application throughput and latency when worker concurrency increases?"</strong></p>
<p>That search might retrieve documentation explaining CPU contention, database connection limits, queue behavior, and shared-resource bottlenecks.</p>
<p>The original question can then be investigated with a better understanding of the underlying concepts.</p>
<p><strong>Where it's useful:</strong> Technical troubleshooting, scientific questions, complex reasoning, and educational assistants.</p>
<p>Step-back retrieval doesn't replace specific evidence. It helps locate the general principles needed to interpret that evidence.</p>
<h2>24. Self-Query Retrieval — Translating Natural Language into Search Conditions</h2>
<p>Imagine asking a movie recommendation assistant:</p>
<p><strong>"Find science-fiction movies released after 2020 with ratings above 8."</strong></p>
<p>This contains two types of information.</p>
<p>The semantic concept:</p>
<pre><code class="language-text">Genre: Science fiction
</code></pre>
<p>And structured conditions:</p>
<pre><code class="language-text">Release year &gt; 2020
Rating &gt; 8
</code></pre>
<p>A self-query retriever uses a language model or another parser to turn natural language into a semantic query and structured filters.</p>
<p>The same idea applies to RAG.</p>
<p>For example:</p>
<p><strong>"Find security reports written by the infrastructure team after March 2026."</strong></p>
<p>The system extracts the topic, authoring team, and date constraint.</p>
<p><strong>Where it's useful:</strong> Document repositories, product catalogs, research databases, and enterprise search systems with rich metadata.</p>
<p>As with SQL generation, the extracted fields and operators should be validated against an approved schema.</p>
<h2>25. Agentic Retrieval — When Search Becomes an Investigation</h2>
<p>A traditional RAG system often follows a fixed sequence:</p>
<pre><code class="language-text">Search -&gt; Retrieve -&gt; Generate
</code></pre>
<p>An agentic retrieval system can make decisions during the process.</p>
<p>Imagine a manager asks:</p>
<p><strong>"Why did Project Orion miss its deadline?"</strong></p>
<p>The system might:</p>
<ol>
<li>Search the project report.</li>
<li>Discover a reference to delayed API integration.</li>
<li>Search issue-tracking records for the integration problem.</li>
<li>Find a dependency on an external vendor.</li>
<li>Retrieve relevant meeting notes.</li>
<li>Compare the evidence before generating an explanation.</li>
</ol>
<pre><code class="language-text">        User Question
              |
              v
         Plan Search
              |
              v
        Retrieve Evidence
              |
              v
       Is More Evidence Needed?
          /           \
        Yes            No
         |              |
         v              v
    Search Again     Generate Answer
</code></pre>
<p><strong>Where it's useful:</strong> Complex enterprise research, technical investigations, workflow assistants, and questions involving multiple systems.</p>
<p>Agentic retrieval can be powerful, but it can also become expensive, slow, and difficult to control.</p>
<p>Good implementations limit search iterations, validate tool permissions, retain citations, and have clear stopping conditions.</p>
<p>Not every question needs an agent.</p>
<hr />
<h1>Part 5: Finding Documents Is Only Half the Job</h1>
<p>Suppose your search returns 30 documents.</p>
<p>How do you decide which five deserve to reach the LLM?</p>
<p>That's where result fusion, reranking, and diversity come in.</p>
<h2>26. Reciprocal Rank Fusion — Combining Different Search Rankings</h2>
<p>Let's say BM25 and vector search return these results:</p>
<table>
<thead>
<tr>
<th>Position</th>
<th>BM25</th>
<th>Vector Search</th>
</tr>
</thead>
<tbody><tr>
<td>1</td>
<td>Document A</td>
<td>Document B</td>
</tr>
<tr>
<td>2</td>
<td>Document C</td>
<td>Document A</td>
</tr>
<tr>
<td>3</td>
<td>Document B</td>
<td>Document D</td>
</tr>
</tbody></table>
<p>Document A ranks highly in both lists.</p>
<p>RRF combines the rankings using a formula like:</p>
<p>[ RRF(d)=\sum_i\frac{1}{k+rank_i(d)} ]</p>
<p>Here, <code>rank</code> is the document's position in each result list, and <code>k</code> is a smoothing constant, commonly 60.</p>
<p>The exact raw BM25 and vector similarity scores don't need to be directly comparable.</p>
<p>Instead, RRF uses positions in the ranked lists.</p>
<p><strong>Example:</strong> Document A ranks first for keywords and second for vectors. That consistent relevance can give it an advantage in the fused results.</p>
<p>RRF is especially useful when combining different retrieval systems.</p>
<h2>27. Reranking — Taking a Second, Closer Look</h2>
<p>First-stage search is often designed to retrieve a reasonably broad set of relevant candidates quickly.</p>
<p>But its ranking may not be perfect.</p>
<p>A reranker evaluates those candidate documents more carefully against the original query.</p>
<p>Suppose the user asks:</p>
<p><strong>"Can contractors claim travel reimbursement?"</strong></p>
<p>The initial retrieval returns:</p>
<ul>
<li>General employee travel reimbursement policy</li>
<li>Contractor payment policy</li>
<li>Contractor-specific travel reimbursement rules</li>
</ul>
<p>A reranker may place the third document first because it directly addresses the question.</p>
<p>Cross-encoder rerankers are one common implementation. They examine the query and candidate document together to estimate relevance.</p>
<pre><code class="language-text">Retrieve Top 30 Documents
             |
             v
          Reranker
             |
             v
        Select Top 5
             |
             v
             LLM
</code></pre>
<p><strong>Where it's useful:</strong> Enterprise RAG, compliance assistants, legal research, and applications where a small number of highly relevant passages matters more than raw retrieval speed.</p>
<p>The trade-off is extra inference cost and latency.</p>
<h2>28. Maximum Marginal Relevance — Avoiding Five Versions of the Same Answer</h2>
<p>Imagine asking:</p>
<p><strong>"What are the main challenges of adopting AI in healthcare?"</strong></p>
<p>A vector search returns five highly similar paragraphs about data privacy.</p>
<p>They're relevant, but collectively they provide a narrow perspective.</p>
<p>Maximum Marginal Relevance, or MMR, is a selection technique that balances relevance with diversity.</p>
<p>Instead of selecting only the most similar passages, it also considers how redundant each new passage is compared with those already selected.</p>
<p>A more diverse result set might cover:</p>
<ul>
<li>Data privacy</li>
<li>Regulatory compliance</li>
<li>Model reliability</li>
<li>Clinical workflow integration</li>
<li>Infrastructure costs</li>
</ul>
<p><strong>Where it's useful:</strong> Research assistants, report generation, broad exploratory questions, and summarization across varied sources.</p>
<p>MMR shouldn't be used to manufacture diversity where the evidence doesn't support it. Its purpose is to reduce redundancy while preserving relevance.</p>
<h2>29. Contextual Compression — Keeping Only What the Answer Needs</h2>
<p>Sometimes a retrieved document is relevant but unnecessarily long.</p>
<p>Imagine searching a 30-page onboarding guide for:</p>
<p><strong>"What documents must new employees submit?"</strong></p>
<p>The relevant information occupies just two paragraphs.</p>
<p>Contextual compression attempts to extract or retain the portions most useful to the question.</p>
<p>It might use rules, relevance scoring, or an LLM-based extractor.</p>
<p>The final context contains the document requirements rather than the entire onboarding guide.</p>
<p><strong>Where it's useful:</strong> Long-form documents, expensive context windows, high-volume RAG systems, and applications where reducing irrelevant information improves answer generation.</p>
<p>Compression introduces its own risk: removing an exception, qualifier, or condition can change the meaning.</p>
<p>For high-stakes documents, retain source references and consider fetching surrounding context before finalizing an answer.</p>
<hr />
<h1>So, Which Search Method Should You Actually Use?</h1>
<p>After reading about all these approaches, it's tempting to add every technique to your RAG pipeline.</p>
<p>That's rarely the right move.</p>
<p>More components create more opportunities for latency, cost, maintenance overhead, and unexpected retrieval failures.</p>
<p>The better approach is to match the retrieval technique to the problem.</p>
<table>
<thead>
<tr>
<th>Your problem</th>
<th>Good starting point</th>
</tr>
</thead>
<tbody><tr>
<td>Searching exact codes, IDs, or names</td>
<td>Exact / keyword search</td>
</tr>
<tr>
<td>Searching technical documentation</td>
<td>BM25 or hybrid search</td>
</tr>
<tr>
<td>Understanding differently worded questions</td>
<td>Dense vector search</td>
</tr>
<tr>
<td>Combining exact terms and semantic meaning</td>
<td>Hybrid search</td>
</tr>
<tr>
<td>Searching within departments or regions</td>
<td>Metadata-filtered retrieval</td>
</tr>
<tr>
<td>Answering questions from database tables</td>
<td>Structured SQL retrieval</td>
</tr>
<tr>
<td>Preserving context in long documents</td>
<td>Parent–child retrieval</td>
</tr>
<tr>
<td>Following connections between entities</td>
<td>Graph-based retrieval</td>
</tr>
<tr>
<td>Understanding collection-wide themes</td>
<td>Global GraphRAG</td>
</tr>
<tr>
<td>Searching images and documents together</td>
<td>Multimodal retrieval</td>
</tr>
<tr>
<td>Handling vague queries</td>
<td>Query rewriting or multi-query</td>
</tr>
<tr>
<td>Answering multi-part questions</td>
<td>Query decomposition</td>
</tr>
<tr>
<td>Investigating complex problems</td>
<td>Agentic retrieval</td>
</tr>
<tr>
<td>Improving result ordering</td>
<td>Reranking</td>
</tr>
<tr>
<td>Reducing duplicate context</td>
<td>MMR</td>
</tr>
<tr>
<td>Reducing unnecessary context</td>
<td>Contextual compression</td>
</tr>
</tbody></table>
<p>These are starting points, not universal rules.</p>
<p>Actual performance depends on the corpus, the embedding model, the search engine, the questions users ask, and the quality of your documents.</p>
<h1>What Would I Build for a Production RAG Application?</h1>
<p>For a typical enterprise document assistant, I wouldn't begin with GraphRAG, autonomous search agents, and five different embedding models.</p>
<p>I'd start with a smaller architecture that is easy to understand, measure, and improve.</p>
<h3>During document ingestion</h3>
<pre><code class="language-text">         Documents / PDFs / URLs
                    |
                    v
          Extract &amp; Clean Text
                    |
                    v
          Structure-Aware Chunking
                    |
                    v
           Attach Metadata
                    |
             +------+------+
             |             |
             v             v
       Text Index     Embedding Model
             |             |
             v             v
         BM25          Vector Index
             |             |
             +------+------+
                    |
                    v
              Data Storage
</code></pre>
<h3>When a user asks a question</h3>
<pre><code class="language-text">              User Question
                    |
                    v
            Authenticate User
                    |
                    v
             Query Rewriting
                (if needed)
                    |
                    v
        Authorization &amp; Filters
                    |
           +--------+--------+
           |                 |
           v                 v
       BM25 Search       Vector Search
           |                 |
           +--------+--------+
                    |
                    v
            Rank Fusion (RRF)
                    |
                    v
                Reranking
                    |
                    v
          Parent Context Retrieval
               (if needed)
                    |
                    v
              Context Assembly
                    |
                    v
              LLM Generation
                    |
                    v
          Answer with Citations
</code></pre>
<p>For implementation, <strong>PostgreSQL + pgvector</strong> can be a practical starting point.</p>
<p>PostgreSQL can handle application data, document metadata, access relationships, and native full-text search. pgvector adds vector similarity search.</p>
<p>Depending on scale and requirements, dedicated search platforms such as Elasticsearch, OpenSearch, Azure AI Search, Qdrant, Vespa, or Weaviate may also make sense.</p>
<p>The important thing is that a database product is not a retrieval strategy. It's part of the infrastructure used to execute one.</p>
<h3>How I'd improve it over time</h3>
<p>Start with a simple retrieval baseline and test it.</p>
<p>If users ask about exact error codes, strengthen lexical search.</p>
<p>If they paraphrase document terminology, evaluate semantic retrieval.</p>
<p>If relevant chunks appear but rank too low, try reranking.</p>
<p>If the answer is missing important surrounding details, inspect chunking and parent–child retrieval.</p>
<p>If a question involves multiple facts or systems, consider decomposition or agentic retrieval.</p>
<p>And if users ask broad questions about patterns across an entire organization, graph-based approaches may be worth exploring.</p>
<p>Each addition should solve a measured problem.</p>
<h1>One More Thing: How Do You Know Retrieval Is Working?</h1>
<p>A search result can look convincing without being correct.</p>
<p>That's why retrieval evaluation matters.</p>
<p>Build a test set of realistic user questions and identify which documents or passages actually contain the required evidence.</p>
<p>Then measure metrics such as:</p>
<p><strong>Recall@k:</strong> Did the system retrieve the relevant evidence among its top k results?</p>
<p><strong>Precision@k:</strong> How much of the retrieved set is actually relevant?</p>
<p><strong>MRR (Mean Reciprocal Rank):</strong> How early does the first relevant result appear?</p>
<p><strong>nDCG:</strong> Are highly relevant documents appearing near the top, taking graded relevance into account?</p>
<p>And beyond retrieval, evaluate whether the final answer is faithful to its sources, whether citations support its claims, and whether the system can recognize when evidence is insufficient.</p>
<p>Measure latency and cost alongside quality.</p>
<p>An approach that improves retrieval slightly but makes every user wait several additional seconds may not be the right trade-off.</p>
<p>Remember, an LLM can only reason over the information it actually receives.</p>
<p>And even good retrieval cannot guarantee a correct answer if the information is outdated, contradictory, or incomplete.</p>
<hr />
<h1>Final Thoughts: Search Is More Than Finding Similar Text</h1>
<p>The biggest lesson I've taken from studying RAG retrieval is that <strong>relevance has different meanings in different situations</strong>.</p>
<p>Sometimes the most relevant result is the document containing an exact error code.</p>
<p>Sometimes it's a paragraph expressing the same idea in completely different words.</p>
<p>Sometimes it's a record that matches a date and customer ID.</p>
<p>Sometimes the answer isn't in any single document at all — it exists across relationships, conversations, and multiple systems.</p>
<p>That's why there is no single search algorithm that solves every RAG problem.</p>
<p>Keyword search gives us precision.</p>
<p>Semantic search helps us bridge differences in language.</p>
<p>Hybrid search brings those strengths together.</p>
<p>Graph retrieval helps us follow relationships.</p>
<p>Query transformation helps us ask better questions.</p>
<p>Reranking helps us decide which evidence deserves attention.</p>
<p>And agentic retrieval allows a system to investigate when one search isn't enough.</p>
<p>The real engineering challenge isn't learning the names of all these techniques.</p>
<p>It's knowing <strong>which retrieval problem you're trying to solve — and choosing the simplest method that solves it reliably.</strong></p>
<p>Because in a RAG application, having the answer somewhere in your database is only the beginning.</p>
<p>The system still has to find it.</p>
<hr />
<h2>Further Reading</h2>
<p>For anyone interested in exploring the technical implementations:</p>
<ul>
<li><a href="https://www.postgresql.org/docs/current/textsearch.html">PostgreSQL — Full-Text Search</a></li>
<li><a href="https://github.com/pgvector/pgvector">pgvector — Vector Similarity Search for PostgreSQL</a></li>
<li><a href="https://learn.microsoft.com/en-us/azure/search/hybrid-search-overview">Azure AI Search — Hybrid Retrieval</a></li>
<li><a href="https://qdrant.tech/documentation/hybrid-queries/">Qdrant — Hybrid and Multi-Stage Queries</a></li>
<li><a href="https://microsoft.github.io/graphrag/">Microsoft GraphRAG</a></li>
<li><a href="https://learn.vespa.ai/reranking-ltr/neural-rerankers/">Vespa — Neural Rerankers and ColBERT</a></li>
<li><a href="https://docs.cohere.com/docs/rerank">Cohere — Reranking</a></li>
<li><a href="https://www.langchain.com/blog/deconstructing-rag">LangChain — Deconstructing RAG</a></li>
</ul>
<p><em>Thanks for reading. If you're building RAG systems, I'd be interested to know which retrieval approach has worked best for your use case.</em></p>
]]></content:encoded></item><item><title><![CDATA[Inside the Kitchen of an AI App: How Celery Works]]></title><description><![CDATA[Learn how Celery handles background tasks in AI applications, with simple examples of Redis, workers, queues, retries, scheduling, and workflows.]]></description><link>https://ainotebook.hashnode.dev/inside-the-kitchen-of-an-ai-app-how-celery-works</link><guid isPermaLink="true">https://ainotebook.hashnode.dev/inside-the-kitchen-of-an-ai-app-how-celery-works</guid><category><![CDATA[Python]]></category><category><![CDATA[Artificial Intelligence]]></category><category><![CDATA[AI]]></category><category><![CDATA[celery]]></category><category><![CDATA[System Design]]></category><dc:creator><![CDATA[Varshitha]]></dc:creator><pubDate>Wed, 04 Nov 2026 18:06:44 GMT</pubDate><content:encoded><![CDATA[<p><em>From taking orders to running AI tasks—understanding Celery through a restaurant.</em></p>
<p>Imagine a waiter taking your order, cooking the meal, serving it, and only then attending to the next customer.</p>
<p>That restaurant would struggle on a busy evening.</p>
<p>An application faces a similar problem when it keeps a request waiting while processing a document or generating a video. Instead, it can accept the job, return an acknowledgment, and let a worker handle the processing.</p>
<p>Celery helps organize that work. Let’s follow an order through a restaurant to understand how.</p>
<h2>1. The waiter, the counter, and the kitchen</h2>
<p>Each part of a restaurant has a responsibility. A Celery application follows a similar arrangement.</p>
<table>
<thead>
<tr>
<th>In the restaurant</th>
<th>In the application</th>
<th>Responsibility</th>
</tr>
</thead>
<tbody><tr>
<td>Waiter</td>
<td>API / producer</td>
<td>Accepts and submits work</td>
</tr>
<tr>
<td>Order slip</td>
<td>Task message</td>
<td>Describes the task and its inputs</td>
</tr>
<tr>
<td>Order counter</td>
<td>Message broker</td>
<td>Holds and delivers messages</td>
</tr>
<tr>
<td>Pending orders</td>
<td>Queue</td>
<td>Keeps work waiting for execution</td>
</tr>
<tr>
<td>Kitchen team</td>
<td>Celery worker</td>
<td>Executes tasks</td>
</tr>
<tr>
<td>Cooking stations</td>
<td>Execution pool</td>
<td>Provides execution slots</td>
</tr>
<tr>
<td>Status register</td>
<td>Result backend</td>
<td>Stores task states and results</td>
</tr>
</tbody></table>
<p>Suppose a user uploads a PDF. The API submits a processing task and returns a job ID. A worker extracts the text, generates a summary, and saves the result.</p>
<p>The processing still takes time, but the API does not need to keep the original request open until everything finishes.</p>
<h2><strong>2. Where Redis fits</strong></h2>
<p>When every cook is busy, the waiter leaves the order at the counter.</p>
<p>The broker provides this handoff between the application and workers. When Redis is the broker, Celery’s queues live in Redis. There is no additional, separate “Celery queue” that must come before it.</p>
<p>Redis holds task messages. Workers execute the corresponding code, which must already be available to them.</p>
<p>Redis is one option; Celery also supports brokers such as RabbitMQ and Amazon SQS.</p>
<p>The result backend handles a different responsibility:</p>
<table>
<thead>
<tr>
<th>Component</th>
<th>Purpose</th>
<th>Restaurant equivalent</th>
</tr>
</thead>
<tbody><tr>
<td>Broker</td>
<td>Holds and delivers task messages</td>
<td>Counter holding order slips</td>
</tr>
<tr>
<td>Result backend</td>
<td>Stores task states and outputs</td>
<td>Register showing order status</td>
</tr>
</tbody></table>
<p>Redis can serve both roles, but queueing a job and storing its result remain separate operations.</p>
<h2><strong>3. What .delay() and .apply_async() do</strong></h2>
<p>The following examples assume the tasks are already defined and the broker and workers are running.</p>
<pre><code class="language-python"># Execute in the current process
summarize_document(document_id)

# Submit for background execution
job = summarize_document.delay(document_id)
</code></pre>
<p>The second call publishes a task message and returns an AsyncResult handle. <a href="http://job.id">job.id</a> identifies the task.</p>
<p>Publishing takes some time and can fail, but the caller does not wait for the summary to finish.</p>
<p>Despite its name, .delay() does not add an intentional waiting period. An available worker may start promptly.</p>
<p>Use .apply_async() when you need additional options:</p>
<pre><code class="language-python">job = summarize_document.apply_async(
    args=[document_id],
    queue="llm",
    countdown=10,
)
</code></pre>
<p>This selects the llm queue and requests execution no earlier than roughly ten seconds later. Busy workers may start later.</p>
<table>
<thead>
<tr>
<th>Call</th>
<th>What happens</th>
</tr>
</thead>
<tbody><tr>
<td><code>task(inputs)</code></td>
<td>Executes in the current process</td>
</tr>
<tr>
<td><code>task.delay(inputs)</code></td>
<td>Submits a task through the broker</td>
</tr>
<tr>
<td><code>task.apply_async(...)</code></td>
<td>Submits with options such as queue, timing, and task ID</td>
</tr>
</tbody></table>
<p>Think of .delay() as handing over a standard order slip. .apply_async() lets you add instructions about where and when it should be handled.</p>
<h2><strong>4. Background work and parallel work are different</strong></h2>
<p>Even with one cook, a restaurant can let the waiter return to customers. Hiring additional cooks is a separate decision.</p>
<table>
<thead>
<tr>
<th>Concept</th>
<th>Restaurant example</th>
<th>Software meaning</th>
</tr>
</thead>
<tbody><tr>
<td>Async I/O</td>
<td>Prepare salad while rice boils</td>
<td>Progress other work while waiting</td>
</tr>
<tr>
<td>Threads</td>
<td>Share one kitchen</td>
<td>Multiple execution threads share process memory</td>
</tr>
<tr>
<td>Multiprocessing</td>
<td>Use separate cooking workspaces</td>
<td>Separate processes can use multiple CPU cores</td>
</tr>
<tr>
<td>Celery</td>
<td>Hand orders to kitchen teams</td>
<td>Distribute work to background workers</td>
</tr>
</tbody></table>
<p>Celery workers can use processes or threads internally.</p>
<p>Waiting for an external LLM response is usually I/O-bound. Heavy local calculations may be CPU-bound. Running a model locally can also depend on GPU capacity.</p>
<p>“AI task” describes the purpose of the work, but not necessarily the resource it needs most.</p>
<h2>5. Workers, pools, and concurrency</h2>
<p>Consider this command:</p>
<pre><code class="language-python">celery -A app worker --pool=prefork --concurrency=4 -l info
</code></pre>
<p>It starts a worker with four child processes for execution, alongside its main worker process.</p>
<p>Each child can handle a task. When one finishes, it can take the next task immediately. It does not wait for all four to finish, and the processes are normally reused.</p>
<table>
<thead>
<tr>
<th>Pool</th>
<th>How tasks execute</th>
<th>Typical use</th>
</tr>
</thead>
<tbody><tr>
<td>Prefork</td>
<td>Separate child processes</td>
<td>A common starting point; useful for CPU work</td>
</tr>
<tr>
<td>Solo</td>
<td>One task at a time in the main thread</td>
<td>Simple execution and debugging</td>
</tr>
<tr>
<td>Threads</td>
<td>Multiple threads in one process</td>
<td>Some I/O-heavy workloads</td>
</tr>
<tr>
<td>Gevent / Eventlet</td>
<td>Cooperative greenlets</td>
<td>Compatible I/O workloads with high concurrency</td>
</tr>
</tbody></table>
<p>Pool capabilities differ, including support for certain timeout features.</p>
<p>Under ordinary GIL-enabled CPython, threads do not automatically speed up pure Python calculations across CPU cores. Four execution slots also do not guarantee four times the speed—hardware and external service limits still matter.</p>
<h2>6. Different counters for different jobs</h2>
<p>A glass of water should not wait behind a large catering order.</p>
<p>An application can route work to different queues:</p>
<table>
<thead>
<tr>
<th>Queue</th>
<th>Example tasks</th>
</tr>
</thead>
<tbody><tr>
<td><code>llm</code></td>
<td>Summaries and scripts</td>
</tr>
<tr>
<td><code>images</code></td>
<td>Image generation</td>
</tr>
<tr>
<td><code>videos</code></td>
<td>Video rendering</td>
</tr>
<tr>
<td><code>notifications</code></td>
<td>Completion messages</td>
</tr>
</tbody></table>
<p>Assign dedicated worker capacity where isolation matters. This lets quick notifications proceed even when rendering workers are busy.</p>
<p>As demand grows, capacity can change in several ways:</p>
<ul>
<li><strong>Horizontal scaling</strong>: Add worker instances or servers, like opening another kitchen.</li>
<li><strong>Vertical scaling:</strong> Give an existing server more resources, like enlarging a kitchen.</li>
<li><strong>Pool autoscaling:</strong> Adjust a worker’s pool size within configured limits. This does not create servers.</li>
</ul>
<p>A few settings help keep the kitchen manageable:</p>
<table>
<thead>
<tr>
<th>Concept</th>
<th>Meaning</th>
</tr>
</thead>
<tbody><tr>
<td>Prefetch</td>
<td>Work a worker reserves before executing it</td>
</tr>
<tr>
<td>Rate limit</td>
<td>How frequently a task can execute</td>
</tr>
<tr>
<td>Backpressure</td>
<td>Controlling incoming work when capacity is full</td>
</tr>
</tbody></table>
<p>Reserved order slips are not active cooking stations: prefetch and concurrency are different.</p>
<p>Celery’s task rate limits are normally per worker. A provider quota shared across workers may need coordinated limiting.</p>
<h2><strong>7. Celery Beat handles the timetable</strong></h2>
<p>A restaurant prepares ingredients every morning, even before customers order.</p>
<p><strong>Celery Beat submits recurring jobs. Workers execute them.</strong></p>
<p>An AI application might generate a daily report or regularly process newly uploaded documents. Interval schedules express repetition; crontab schedules express calendar-based timing.</p>
<p>Keep one active scheduler for the same schedule set unless you have explicit coordination. Otherwise, duplicate jobs can be submitted.</p>
<p>Also decide what happens if the previous run is still active. A schedule alone does not prevent overlapping executions.</p>
<p>Two deadlines are worth distinguishing:</p>
<ul>
<li><strong>Task expiration:</strong> The job is too old to execute.</li>
<li><strong>Result expiration:</strong> The stored output can be cleaned up.</li>
</ul>
<h2>8. Connecting tasks into workflows</h2>
<p>Rice and curry can cook independently, but both must be ready before packing the meal.</p>
<p>Celery’s Canvas primitives describe these relationships.</p>
<table>
<thead>
<tr>
<th>Concept</th>
<th>Restaurant example</th>
<th>AI application example</th>
</tr>
</thead>
<tbody><tr>
<td>Signature <code>.s()</code></td>
<td>Prepared order description</td>
<td>A task with its inputs and options</td>
</tr>
<tr>
<td>Immutable signature <code>.si()</code></td>
<td>Fixed order independent of earlier output</td>
<td>Ignore automatically passed upstream results</td>
</tr>
<tr>
<td>Chain</td>
<td>Prepare, cook, pack</td>
<td>Extract text, summarize, generate audio</td>
</tr>
<tr>
<td>Group</td>
<td>Prepare independent dishes</td>
<td>Process several documents independently</td>
</tr>
<tr>
<td>Chord</td>
<td>Pack after all dishes succeed</td>
<td>Combine several summaries into a report</td>
</tr>
</tbody></table>
<p>Here is a chain:</p>
<pre><code class="language-python">from celery import chain
chain( 
    extract_text.s(document_id), 
    summarize.s(), 
    generate_audio.s(),
).apply_async()
</code></pre>
<p>Each task’s result becomes input to the next step.</p>
<p>A group submits independent work, but actual parallel execution depends on available capacity. A chord passes a list of results to its callback and requires a compatible result backend with results retained.</p>
<p>For video generation, a workflow might create a script, generate narration and visuals independently, then render after both succeed.</p>
<p>Pass file IDs or storage URLs between tasks instead of large media files.</p>
<p>Conditions such as “if validation fails, request a rewrite” remain application logic. Celery does not decide those rules for you.</p>
<h2>9. Bound tasks and retries</h2>
<p>A cook sometimes needs the order number and attempt history. A bound task gets access to its task instance through self.</p>
<p>This example assumes request_summary is an application helper:</p>
<pre><code class="language-python">@app.task(bind=True, max_retries=3)
def summarize_document(self, document_id):
    try:
        return request_summary(document_id)
    except TimeoutError as exc:
        raise self.retry(exc=exc, countdown=10)
</code></pre>
<p>Celery supplies self; the caller passes only document_id.</p>
<p>The task can inspect:</p>
<ul>
<li><a href="http://self.request.id"><strong>self.request.id</strong></a> for its task ID.</li>
<li><strong>self.request.retries</strong> for its retry count.</li>
</ul>
<p><code>bind=True</code> <strong>does not automatically enable retries.</strong> The retry call does that here. In a real integration, handle the exception types your SDK raises.</p>
<p>Use backoff to increase delays after repeated failures and jitter to spread retries out. Temporary service failures may recover; an invalid API key usually needs correction.</p>
<h2>10. What happens when something goes wrong?</h2>
<p>Suppose the kitchen sends an order but crashes before recording delivery. Repeating the work could send a second meal.</p>
<p>Applications face similar risks with emails, publishing, and paid AI requests.</p>
<p>Idempotency means repeated execution avoids unintended duplicate effects. Store operation IDs and use provider-supported idempotency keys where available. A Celery task ID alone does not prevent duplicates.</p>
<p>An <strong>ACK</strong> acknowledges a broker delivery:</p>
<table>
<thead>
<tr>
<th>Setting</th>
<th>When acknowledgment happens</th>
</tr>
</thead>
<tbody><tr>
<td>Early ACK</td>
<td>Before task execution</td>
</tr>
<tr>
<td>Late ACK</td>
<td>After execution, subject to configured behavior</td>
</tr>
</tbody></table>
<p>Redelivery depends on configuration and the failure involved. Late ACK does not guarantee exactly-once execution.</p>
<p>Set network timeouts so calls cannot wait forever. Supported pools also offer soft and hard execution limits.</p>
<p>A failed workflow does not automatically undo successful earlier steps. Revoking a task does not normally stop one already running, and canceling a provider job requires that provider’s cancellation mechanism.</p>
<h2>11. Why polling is sometimes needed</h2>
<p>Imagine the kitchen orders a cake from another bakery. The bakery replies with a collection number instead of a finished cake.</p>
<p>An external AI service can behave the same way:</p>
<table>
<thead>
<tr>
<th>Provider response</th>
<th>What the application does</th>
</tr>
</thead>
<tbody><tr>
<td>Final result</td>
<td>Save it; no external polling needed</td>
</tr>
<tr>
<td>Job ID or status URL</td>
<td>Check later, or receive a completion webhook</td>
</tr>
</tbody></table>
<p>Celery runs your background work. It cannot automatically know when another service finishes its own job.</p>
<p>There are also two different kinds of status checks:</p>
<ul>
<li><strong>Frontend status check:</strong> Your frontend asks your backend about your job.</li>
<li><strong>External polling</strong>: Your backend asks the provider about its job. For long waits, check once, save the next check time, and release the worker.</li>
</ul>
<p>Human approval follows the same pattern: save a waiting state and submit the next step after approval.</p>
<h2>12. Track the work, not just the final result</h2>
<p>Keep workflow run IDs, individual step IDs, and Celery task IDs connected. This makes failures easier to trace.</p>
<p>Celery exposes states such as <strong>SUCCESS, FAILURE, and RETRY</strong>. <strong>PENDING</strong> can mean the result backend has no information; it does not prove the job is in a queue. Started tracking needs enabling, and progress updates need application code.</p>
<p>Use logs, worker events, and tools such as Flower to watch queue delays, task duration, and failures. Track AI costs and output quality separately from execution success.</p>
<p>Test worker crashes and duplicate delivery, protect broker access, and keep secrets out of messages and logs.</p>
<p>Saving a database job and publishing its message are separate operations. A durable dispatch record can help recover work lost between them.</p>
<p>Celery gives the kitchen a way to receive and execute orders. Your application still owns the menu, permissions, approval rules, and the customer’s view of progress. With those responsibilities clear, background processing becomes much easier to build and explain.</p>
]]></content:encoded></item><item><title><![CDATA[From Your Words to Its Answer: Inside an LLM]]></title><description><![CDATA[Follow one sentence through tokens, attention, and the maths that produces a reply. Ask an AI these two questions: Explain gravity to a 10-year-old.Explain…]]></description><link>https://ainotebook.hashnode.dev/from-your-words-to-its-answer-inside-an-llm</link><guid isPermaLink="true">https://ainotebook.hashnode.dev/from-your-words-to-its-answer-inside-an-llm</guid><dc:creator><![CDATA[Varshitha]]></dc:creator><pubDate>Sat, 31 Oct 2026 15:54:18 GMT</pubDate><enclosure url="https://cdn.hashnode.com/uploads/covers/6acd07f78e040e2b3fb223fd/bec53575-21ef-4370-9980-6acc0c18683e.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<h1>What Happens Between Clicking Send and Seeing an AI Answer?</h1>
<p><em>Follow one sentence through tokens, attention, and the maths that produces a reply.</em></p>
<p>Ask an AI these two questions:</p>
<blockquote>
<p>Explain gravity to a <strong>10-year-old</strong>.<br />Explain gravity to a <strong>physicist</strong>.</p>
</blockquote>
<p>Same topic. Very different answers.</p>
<p>Somewhere inside the model, a small change in your sentence changes the vocabulary, detail, and direction of the response.</p>
<p>How does that happen? Let’s follow the first question inside.</p>
<h3>Your sentence becomes a sequence of numbers</h3>
<p>After you click Send, the application assembles the input. This can include system instructions, earlier messages, and your question.</p>
<p>A tokenizer breaks that text into <strong>tokens</strong>—pieces that may be words, word fragments, punctuation, or spaces combined with text.</p>
<p>An illustrative split:</p>
<p><code>[Explain] [ gravity] [ to] [ a] [ 10] [-] [year] [-] [old] [.]</code></p>
<p>Each piece maps to an integer ID. But an ID alone doesn’t carry useful meaning: token 500 isn’t “more meaningful” than token 250.</p>
<p>Instead, the ID selects a row from a learned <strong>embedding table</strong>. That row is a vector—a list of numbers.</p>
<table>
<thead>
<tr>
<th>Stage</th>
<th>Illustrative representation</th>
</tr>
</thead>
<tbody><tr>
<td>Text fragment</td>
<td><code>gravity</code></td>
</tr>
<tr>
<td>Token ID</td>
<td><code>8472</code></td>
</tr>
<tr>
<td>Embedding</td>
<td><code>[0.21, −0.64, 0.08, …]</code></td>
</tr>
</tbody></table>
<p><em>These values and token boundaries are invented for illustration.</em></p>
<p>Think of the ID as a seat number and the embedding as the person sitting there. The number locates something; the thing it locates is what matters.</p>
<p>Position information is incorporated too, because “Earth pulls objects” and “objects pull Earth” arrange the same words differently.</p>
<p>Now the model has numerical representations it can work with.</p>
<h3>Attention changes a token’s representation using its surroundings</h3>
<p>An embedding is a starting point. The layers update it using context.</p>
<p>Consider the word “bank”:</p>
<blockquote>
<p>She sat beside the river bank.<br />She deposited money at the bank.</p>
</blockquote>
<p>The initial representation of the same token can be identical. Its later representations can differ because the surrounding tokens differ.</p>
<p><strong>Attention is one mechanism that makes this possible.</strong></p>
<p>Inside an attention head, each position’s current vector is multiplied by three learned matrices:</p>
<p>q=xWQ,k=xWK,v=xWV</p>
<p>That looks more intimidating than it is:</p>
<ul>
<li><strong>Query:</strong> a vector used to look for relevant information.</li>
<li><strong>Key:</strong> a vector against which a query is compared.</li>
<li><strong>Value:</strong> information that can be passed onward.</li>
</ul>
<p>These aren’t written questions or labels. They are numerical transformations learned during training.</p>
<h3>Let’s actually calculate a small piece of attention</h3>
<p>For one position, the model compares its query with the keys at positions it is allowed to attend to.</p>
<p>The comparison is a <strong>dot product</strong>: multiply corresponding numbers, then add them.</p>
<p>For example:</p>
<p>[1, 2] · [3, 1] = (1 × 3) + (2 × 1) = 5</p>
<p>The scores are scaled by the square root of the key-vector size. Softmax then converts them into weights.</p>
<p>Suppose three positions have scaled scores of <strong>2, 1, and 0</strong>:</p>
<table>
<thead>
<tr>
<th>Position</th>
<th align="right">Scaled score</th>
<th align="right">Attention weight</th>
</tr>
</thead>
<tbody><tr>
<td>A</td>
<td align="right">2</td>
<td align="right">0.665</td>
</tr>
<tr>
<td>B</td>
<td align="right">1</td>
<td align="right">0.245</td>
</tr>
<tr>
<td>C</td>
<td align="right">0</td>
<td align="right">0.090</td>
</tr>
</tbody></table>
<p>Softmax produces positive weights that sum to one.</p>
<p>Now suppose their value vectors are:</p>
<ul>
<li>A: <code>[2, 0]</code></li>
<li>B: <code>[0, 2]</code></li>
<li>C: <code>[1, 1]</code></li>
</ul>
<p>The attention output is:</p>
<p>[ 0.665[2,0]+0.245[0,2]+0.090[1,1]=[1.42,0.58] ]</p>
<p><strong>That new vector contains a weighted mixture of information from the three positions.</strong></p>
<p>This is what “paying attention” means here. The model calculates how much information to mix from each available position.</p>
<p>In our gravity prompt, information about the topic and the audience can influence later representations. There isn’t a hand-written rule saying “if child, use simple words.”</p>
<p>Multiple heads calculate different mixtures, which are combined and projected back into the model’s working representation. A causal mask blocks access to future positions.</p>
<h3>Attention isn’t the whole layer</h3>
<p>The mixed information still needs processing.</p>
<p>A <strong>feed-forward network</strong>, often called an MLP, transforms each position’s vector. In a basic version, it projects the vector into a larger space, applies a nonlinear function, and projects it back.</p>
<p>The nonlinearity matters: without it, stacking those linear transformations would collapse into a single linear transformation.</p>
<p>Two other components help:</p>
<table>
<thead>
<tr>
<th>Component</th>
<th>Role</th>
</tr>
</thead>
<tbody><tr>
<td>Residual connection</td>
<td>Adds an update to the existing representation, instead of replacing it completely.</td>
</tr>
<tr>
<td>Normalization</td>
<td>Regulates numerical scales as information passes through the network.</td>
</tr>
</tbody></table>
<p>Many transformer layers repeat these operations, though the exact architecture varies.</p>
<p>There is no guaranteed “grammar layer” followed by a “reasoning layer.” Each layer updates representations; useful behaviour emerges across the network.</p>
<p>Changing “10-year-old” to “physicist” changes the input vectors and subsequent calculations. Those changes eventually affect which output tokens receive higher scores.</p>
<h3>How do those vectors turn into a word?</h3>
<p>At the final layer, the representation at the last input position is used to predict the next token.</p>
<p>An output projection calculates a score for every token in the vocabulary:</p>
<p>[ z=Wh+b ]</p>
<p>Here, is the final representation, \(W\) contains learned weights, and \(b\) is an optional bias. The resulting scores are called <strong>logits</strong>.</p>
<p>For a toy vocabulary:</p>
<table>
<thead>
<tr>
<th>Candidate token</th>
<th align="right">Logit</th>
<th align="right">Probability</th>
</tr>
</thead>
<tbody><tr>
<td><code>Gravity</code></td>
<td align="right">3</td>
<td align="right">66.5%</td>
</tr>
<tr>
<td><code>Imagine</code></td>
<td align="right">2</td>
<td align="right">24.5%</td>
</tr>
<tr>
<td><code>Hello</code></td>
<td align="right">1</td>
<td align="right">9.0%</td>
</tr>
</tbody></table>
<p>The probability calculation is:</p>
<p>$$P_i = \frac{e^{z_i}}{\sum_j e^{z_j}}$$</p>
<p>For <code>Gravity</code>, that is , approximately 66.5%.</p>
<p>Real models score their entire vocabulary, not just these three choices.</p>
<p>Notice that softmax has appeared twice. <strong>Inside attention, it weights positions. At the output, it assigns probabilities to candidate tokens.</strong></p>
<h3>The answer grows through repeated choices</h3>
<p>A decoding rule selects the next token.</p>
<p>Greedy decoding picks the highest-scoring option. Sampling draws from a distribution; temperature can make that distribution sharper or flatter. Top-k and top-p can narrow the candidate pool.</p>
<p>Suppose the first choice is <code>Gravity</code>.</p>
<p>The model now predicts again, using the input plus that token:</p>
<table>
<thead>
<tr>
<th>Context includes</th>
<th>Illustrative next token</th>
</tr>
</thead>
<tbody><tr>
<td>The original prompt</td>
<td><code>Gravity</code></td>
</tr>
<tr>
<td>Prompt + <code>Gravity</code></td>
<td><code>is</code></td>
</tr>
<tr>
<td>Prompt + <code>Gravity is</code></td>
<td><code>a</code></td>
</tr>
<tr>
<td>Prompt + <code>Gravity is a</code></td>
<td><code>force</code></td>
</tr>
</tbody></table>
<p>The full answer doesn’t need to exist before the first token is produced.</p>
<p>Generation continues until an end-of-turn token or another stopping condition is reached.</p>
<h3>Why doesn’t it recompute everything?</h3>
<p>The initial prompt is processed during <strong>prefill</strong>. At each layer, prompt positions can be processed in parallel while respecting the causal mask.</p>
<p>Afterward, generation proceeds step by step. A <strong>KV cache</strong> retains earlier keys and values, allowing new tokens to use them without recalculating those earlier keys and values.</p>
<p>Generated token IDs are decoded into text. With streaming enabled, the application sends text chunks to your screen as generation progresses.</p>
<pre><code class="language-mermaid">flowchart TD
    A["Prompt becomes token vectors"] --&gt; B["Layers mix and transform information"]
    B --&gt; C["Score vocabulary and select a token"]
    C --&gt; D{"End of turn?"}
    D --&gt;|No| E["Decode text and extend context"]
    E --&gt; F["Process new token using cached keys and values"]
    F --&gt; B
    D --&gt;|Yes| G["Finish"]
</code></pre>
<h3>But who taught it all these calculations?</h3>
<p>Training adjusted the embeddings, attention matrices, feed-forward weights, and output projection to improve predictions. Later training can improve instruction-following and other behaviours.</p>
<p>During ordinary generation, those learned parameters generally stay fixed. What changes is the context and the numerical activity produced from it.</p>
<p>And none of this automatically verifies an answer. A high next-token probability means a continuation is likely under the model—not that its claim has been checked.</p>
<p>So when “10-year-old” produces a simpler explanation than “physicist,” there is no tiny editor rewriting the answer inside the machine.</p>
<p>There are different input vectors, different information mixtures, different output scores—and a sequence of token choices that makes the difference visible.</p>
]]></content:encoded></item></channel></rss>