AI Systems

RAG — Retrieval-Augmented Generation

RAG fetches relevant data from outside a model's parameters and injects it into the context window, grounding generation in retrievable evidence instead of whatever the model happened to memorize during training. It's also far cheaper to keep a retrieval index current than to retrain a model — which is what makes RAG the default answer for incorporating recent or proprietary knowledge, and the single most frequently designed AI system in practice.

Naive single-pass RAG

Embed the incoming query with the same model family used to embed the corpus, run approximate nearest-neighbor search (HNSW or IVF-PQ indexes, in stores like pgvector, Pinecone, Weaviate or Qdrant), retrieve the top-k chunks by cosine similarity, and concatenate them into the prompt. Chunk size is the first real design decision — 256 to 512 tokens with 10 to 20 percent overlap is a reasonable default for prose, while structured documents (contracts, tables, datasheets) often need semantic or layout-aware chunking instead of fixed-size windows. Naive RAG degrades on ambiguous queries, multi-hop questions that need synthesizing several documents, and corpora with poor structural boundaries.

Hybrid retrieval RAG

Combine dense vector retrieval with sparse lexical search — typically BM25 — and fuse the two ranked lists with reciprocal rank fusion or a learned cross-encoder reranker. This directly addresses a well-known weakness of dense embeddings: they generalize away exact identifiers like part numbers, register names, or rare proper nouns that a keyword index matches trivially. A two-stage pipeline — cheap bi-encoder retrieval fetching 50 to 100 candidate chunks, then a cross-encoder reranker narrowing to the final top 5 to 10 — reliably outperforms either method alone.

Agentic and iterative RAG

The model judges whether retrieved context is actually sufficient and, if not, reformulates the query and retrieves again — sometimes decomposing a complex question into sub-questions first. Self-RAG trains the model to emit special retrieval/critique tokens that trigger on-demand retrieval and self-assessment; Corrective RAG (CRAG) adds a lightweight relevance grader on retrieved documents and falls back to a web search when the internal knowledge base has thin coverage. Both add one to three extra round trips of latency and cost for materially better answers on multi-hop and ambiguous queries — reserve them for query types where naive RAG measurably underperforms, not by default.

Productionizing this pattern

Real-world applications

• Internal knowledge search — letting employees ask questions over policies, procedures, and contracts instead of grepping a shared drive.
• Customer support copilots — grounding responses in a product's actual documentation and past ticket resolutions instead of the model's general training.
• Code assistants — retrieving relevant functions/files from a specific codebase rather than hallucinating a plausible-looking API.
• Legal/compliance review — answers grounded in the exact regulation or contract clause, with citation, rather than a paraphrase from memory.

Embedded applications

• Local-first diagnostic assistants — a gateway device or field-service tool can retrieve from a locally cached corpus of datasheets, error-code tables, and past repair logs stored on-device (SQLite + a lightweight vector index) — the same "the corpus is embedded offline" step in the diagram above, just running on flash instead of a cloud vector DB, so a technician gets grounded answers even with no connectivity.
• Hybrid edge/cloud RAG — a resource-constrained device does a cheap local retrieval pass (small embedding model, small local index) to answer common questions instantly, and only escalates to a cloud RAG pipeline for genuinely novel queries — directly analogous to the exact-match-first, semantic-cache-second layering covered in Caching.
• Fleet log retrieval — an LLM assistant answering "has this fault code appeared on other units?" is doing RAG over a corpus of device telemetry and logs — the same retrieval-quality-vs-generation-quality separation applies, and it's the same kind of structured, high-cardinality data that benefits from hybrid (BM25 + dense) retrieval over dense-only, since fault codes and part numbers behave like the exact-identifier problem hybrid retrieval solves.
• Device-tree and datasheet lookup — a documentation assistant for board bring-up work grounded in a corpus of a project's actual device tree sources and vendor datasheets, rather than a general model's often-outdated recollection of a specific SoC's register map.

Gotchas