Section 7 — RAG in Spring AI

Begin with the problem

Letting an AI answer from your documents

A model may not know your newest handbook or private product data. RAG first retrieves useful passages and then places them beside the question before generation.

question → retrieve evidence → add context → model → grounded answer

What you will learn

• Follow a complete RAG request.
• Distinguish retrieval quality from generation quality.
• Use Spring AI RAG advisors and components.
• Evaluate citations, relevance, and faithfulness.

Current official reference: Spring AI documentation for this topic. The examples below primarily preserve the stated 1.1.x course target. Where Spring AI 2.0 differs, the text must treat that behavior as version-specific rather than universal.

(Continues from Section 6. Target: Spring AI 1.1.x / Spring Boot 3.5.x.)

7.1 Why Spring AI’s RAG Story Is “Modular Advisor Pipeline,” Not “One RAG Class”

Beginner primer: if RAG (Retrieval-Augmented Generation) is new, read the glossary entry first. In one sentence: before asking the LLM to answer, you first retrieve relevant documents (using the embeddings + vector search from Sections 5–6), then stuff them into the prompt as context, so the model answers using your real data instead of purely from what it memorized during training — directly reducing hallucination and giving the model access to information it was never trained on (your company’s internal docs, for instance).

What Spring AI does architecturally is decompose the RAG pipeline into small, independently swappable components — QueryTransformer, DocumentRetriever, QueryAugmenter — composed by one Advisor (RetrievalAugmentationAdvisor) rather than one monolithic “RAG service” class. This mirrors the Advisor-chain pattern from Section 3: RAG isn’t a separate subsystem, it’s a specific, well-supported composition of the same Advisor SPI everything else uses.

Real-world analogy — Hospital Triage → Specialist Routing → Diagnosis: Query transformation is triage (is this question actually answerable, does it need rewriting/splitting into sub-questions). Retrieval is pulling the patient’s relevant chart history. Augmentation is the doctor synthesizing chart + symptoms into a diagnosis. Each stage is a distinct professional with a distinct interface — you don’t want one “do everything” clinician object.

Analogy: The Hospital Triage, Chart Retrieval, and Specialist Diagnosis Think of a RAG query pipeline as a structured workflow inside a major hospital:

• Triage (QueryTransformer): The patient walks in and says: “I feel dizzy. What about yesterday?” The triage nurse translates this into a clear, standalone medical description: “Patient reports dizziness following yesterday’s treatment.” (Compression and Rewrite Query transformation).
• Chart Room (DocumentRetriever): The medical archivist takes the patient description, scans the library shelves, and pulls out the 5 most relevant historical health folders (Similarity Search).
• Reranking (DocumentPostProcessor): The head physician reads the 5 folders and discards 2 of them as outdated or less relevant, ordering the remaining 3 by critical importance.
• Specialist (QueryAugmenter & ChatModel): The final diagnosing specialist receives the selected 3 folders, compiles them, and makes a precise diagnosis recommendation strictly based on that evidence.

📊 Visual Flowchart: The RetrievalAugmentationAdvisor Execution Cycle

Here is how queries are transformed, documents retrieved, and templates augmented inside the ChatClient pipeline:

graph TD
    UserQuery["User Input Query:<br>'what about return policy?'"] --> Compress["1. CompressionQueryTransformer<br>(Rewrite query with history context)"]
    Compress --> StandaloneQuery["Standalone Query:<br>'What is return policy?'"]

    subgraph RetrievalStage ["Document Retrieval & Post-Processing"]
        StandaloneQuery --> Retrieve["2. VectorStoreDocumentRetriever<br>(similaritySearch topK & threshold)"]
        Retrieve --> Chunks["Raw Chunks (JSON list)"]
        Chunks --> Rerank["3. Cross-Encoder Reranker<br>(Sort by relevance score)"]
        Rerank --> CleanChunks["Sorted Chunks"]
    end

    CleanChunks --> Augment["4. ContextualQueryAugmenter<br>(Format Prompt: inject context + user query)"]

    Augment --> AugmentedPrompt["5. Augmented Prompt Template"]
    AugmentedPrompt --> ChatModel["6. ChatModel.call()"]

---

7.2 The Pipeline, Precisely

User Query
    │
    ▼
┌────────────────────────────────────────────────┐
│ QueryTransformer (0..N, chained)                  │
│ - CompressionQueryTransformer: condenses          │
│   conversational history into a standalone query  │
│   (critical for multi-turn RAG — a follow-up      │
│   "what about the second one?" is meaningless as  │
│   a standalone vector query)                       │
│ - RewriteQueryTransformer: LLM-rewrites for        │
│   better retrieval recall                          │
│ - TranslationQueryTransformer: normalizes query    │
│   language if your corpus is monolingual           │
└─────────────────────┬────────────────────────────┘
                       ▼
┌────────────────────────────────────────────────┐
│ QueryExpander (optional)                           │
│ - MultiQueryExpander: generates N variant queries  │
│   from one input to widen recall, results          │
│   merged/deduplicated after retrieval               │
└─────────────────────┬────────────────────────────┘
                       ▼
┌────────────────────────────────────────────────┐
│ DocumentRetriever                                  │
│ - VectorStoreDocumentRetriever: wraps a            │
│   VectorStore.similaritySearch() call, applies     │
│   SearchRequest defaults (topK, threshold,         │
│   filterExpression)                                │
└─────────────────────┬────────────────────────────┘
                       ▼
┌────────────────────────────────────────────────┐
│ DocumentPostProcessor (0..N, e.g. reranker)        │
└─────────────────────┬────────────────────────────┘
                       ▼
┌────────────────────────────────────────────────┐
│ QueryAugmenter                                     │
│ - ContextualQueryAugmenter: injects retrieved      │
│   Documents into the Prompt as formatted context,  │
│   with an explicit "if answer isn't in context,    │
│   say so" instruction baked in by default — this   │
│   default instruction is itself a hallucination-   │
│   reduction lever                                   │
└─────────────────────┬────────────────────────────┘
                       ▼
              Augmented Prompt → ChatModel

---

7.3 RetrievalAugmentationAdvisor — Wiring It Together

@Bean
public ChatClient ragChatClient(ChatClient.Builder builder, VectorStore vectorStore) {

    var retriever = VectorStoreDocumentRetriever.builder()
            .vectorStore(vectorStore)
            .similarityThreshold(0.73)
            .topK(8)
            .filterExpression(filterCtx -> new FilterExpressionBuilder()
                    .eq("tenantId", filterCtx.get("tenantId")).build())
            .build();

    var queryTransformer = CompressionQueryTransformer.builder()
            .chatClientBuilder(builder.clone())
            .build();

    var augmenter = ContextualQueryAugmenter.builder()
            .allowEmptyContext(false)   // refuse to answer rather than hallucinate
                                        // when retrieval returns nothing
            .build();

    var ragAdvisor = RetrievalAugmentationAdvisor.builder()
            .queryTransformers(queryTransformer)
            .documentRetriever(retriever)
            .queryAugmenter(augmenter)
            .build();

    return builder
            .defaultAdvisors(ragAdvisor)
            .build();
}

At call time:

ragChatClient.prompt()
        .user(userQuestion)
        .advisors(a -> a.param("tenantId", currentTenantId))
        .call()
        .content();

allowEmptyContext(false) is a small flag with outsized production importance: without it, an empty retrieval result still lets the model attempt an answer from parametric knowledge alone (i.e., whatever it happened to learn during training) — silently defeating the entire point of RAG grounding and reintroducing hallucination risk exactly when your knowledge base has a gap.

---

7.4 Document Readers and the Ingestion ETL Pipeline

Raw Source	DocumentReader	Produces
PDF file	PagePdfDocumentReader (or ParagraphPdfDocumentReader for paragraph-level granularity)	One Document per page
Word/PPT/etc.	TikaDocumentReader	Apache Tika-backed, broad format support
Markdown	MarkdownDocumentReader	Section-aware splitting respecting heading structure
JSON	JsonReader	Configurable JSON-pointer-based extraction

List<Document> pages = new PagePdfDocumentReader(pdfResource,
        PdfDocumentReaderConfig.builder()
                .withPageExtractedTextFormatter(ExtractedTextFormatter.builder()
                        .withNumberOfTopTextLinesToDelete(1)    // strip repeated headers
                        .withNumberOfBottomTextLinesToDelete(1) // strip page footers
                        .build())
                .withPagesPerDocument(1)
                .build())
        .get();

Stripping headers/footers matters more than it sounds: repeated boilerplate text (“Confidential — Acme Corp — Page X of Y”) on every page pollutes every chunk’s embedding with identical noise, measurably degrading retrieval discrimination between really different pages.

---

7.5 Chunking

Beginner note: “chunking” means splitting a large document into smaller pieces before embedding each piece separately. This is necessary because (a) embedding models have their own input size limits, and (b) retrieval works better with focused, topically-coherent chunks than with one giant embedding representing an entire document’s mixed content.

TextSplitter splitter = new TokenTextSplitter(
        800,   // target chunk size in tokens
        350,   // min chunk size to keep (avoids tiny orphan trailing chunks)
        10,    // min chunk length in characters
        5000,  // max number of chunks (safety cap)
        true   // keep separator
);

List<Document> chunks = splitter.apply(pages);

TokenTextSplitter is Spring AI’s default, splitting on token count (using a tokenizer estimate, not exact provider tokenization — see §7.6) rather than character count, because retrieval quality and downstream context-window budgeting both operate in token space, not character space (recall the glossary’s token discussion). Chunk-size tuning is a real production trade-off:

Chunk size	Retrieval precision	Context completeness	Typical use
Small (200–400 tokens)	Higher — less noise per chunk, sharper similarity match	Lower — may cut mid-thought, missing needed context	FAQ-style, fact-lookup corpora
Medium (600–1000 tokens)	Balanced	Balanced	General-purpose document RAG — the common default
Large (1500+ tokens)	Lower — more noise dilutes the embedding signal	Higher — full sections retained	Legal/technical documents where fragmenting loses critical qualifying context

Overlap (not shown above but commonly configured via a custom splitter or withOverlap where the API version supports it) — typically 10-20% of chunk size — reduces the risk of a key fact landing exactly on a chunk boundary and being retrievable from neither adjacent chunk cleanly.

---

7.6 Tokenization Inside Spring AI

Spring AI uses TokenCountEstimator (JTokkit-backed for OpenAI-family tokenization) to approximate token counts for chunking and context-budget decisions without making a network call. This is an estimate, not the provider’s exact tokenizer output for every model family — Anthropic and Google use different tokenizers than OpenAI’s cl100k_base/o200k_base. For chunking purposes this approximation is fine (you’re choosing a reasonable chunk boundary, not billing); for precise cost/context-window budgeting against a non-OpenAI model, treat the estimate as directional, not exact, and build in headroom margin.

TokenCountEstimator estimator = new JTokkitTokenCountEstimator();
int tokenCount = estimator.estimate(documentContent);

---

7.7 Ranking — Why Retrieval Rank Alone Isn’t Enough (Recap + Implementation)

Section 6 established that raw vector similarity is a weak final-relevance signal. The production remedy is a DocumentPostProcessor implementing reranking:

public class CrossEncoderRerankPostProcessor implements DocumentPostProcessor {

    private final RerankModel rerankModel; // e.g., Cohere rerank, or an
                                            // LLM-as-reranker call via ChatModel

    @Override
    public List<Document> process(Query query, List<Document> documents) {
        List<RerankResult> reranked = rerankModel.rerank(
                query.text(), documents.stream().map(Document::getContent).toList());

        return reranked.stream()
                .sorted(Comparator.comparingDouble(RerankResult::score).reversed())
                .limit(5)
                .map(r -> documents.get(r.originalIndex()))
                .toList();
    }
}

Beginner note: a reranker is a second, more accurate (but slower/more expensive) model that re-scores a small candidate set for true relevance to the query — different from, and typically more accurate than, plain vector similarity, because it can directly compare the query against each candidate document rather than relying purely on how close their pre-computed embeddings happen to sit in vector space. This is why it’s applied after an initial cheap similarity search narrows thousands of documents down to a manageable candidate set (§7.7 below), not instead of it.

Wire it into the RetrievalAugmentationAdvisor builder as a .documentPostProcessors(...) stage. The pattern: over-retrieve (topK=20-30) at the vector-similarity stage, then rerank down to the 3-5 you actually inject into context. This two-stage retrieve-then-rerank pipeline consistently outperforms single-stage retrieval at equivalent final context size, because vector similarity and true task-relevance are correlated but not identical signals — reranking is where that gap gets closed.

---

7.8 Context Building

ContextualQueryAugmenter’s default template roughly follows this shape (customize via .promptTemplate(...) on the builder):

{original user query}

Context information is below.
---------------------
{formatted retrieved documents, typically with source metadata}
---------------------
Given the context information and no prior knowledge,
answer the query. If the answer is not in the context,
state that you don't have enough information.

Production customization worth making explicitly: include source citations in the formatted context (document ID, source URL, page number from metadata) so the model can reference them in its answer — this is what powers citation support in one of this series’ worked applications (App 2 — RAG Assistant), and it’s a template-formatting decision, not a separate API.

---

7.9 Production Pipeline — End to End

Ingestion (offline/batch, Section 5+6 techniques apply directly):
  DocumentReader → header/footer cleanup → TextSplitter (chunking)
  → batch EmbeddingModel calls → VectorStore.add() with tenant/source metadata

Query time (online, per-request):
  User query → QueryTransformer (compress conversational history)
  → DocumentRetriever (VectorStore similarity search, tenant-filtered,
    over-fetch topK=20)
  → DocumentPostProcessor (rerank down to 5)
  → QueryAugmenter (build final Prompt with citations)
  → ChatClient.call() → response with source attribution

---

7.10 Evaluation

Spring AI ships evaluator abstractions for automated RAG quality checks, usable in integration tests (also covered in Section 14):

RelevancyEvaluator relevancyEvaluator = new RelevancyEvaluator(chatClientBuilder);

EvaluationRequest request = new EvaluationRequest(
        userQuestion, retrievedDocuments, generatedResponse);
EvaluationResponse evalResponse = relevancyEvaluator.evaluate(request);

assertThat(evalResponse.isPass()).isTrue();

RelevancyEvaluator uses an LLM-as-judge pattern internally (a separate model call scoring whether the response is actually grounded in/relevant to the retrieved context) — this is the same LLM-as-judge technique used broadly in RAG evaluation across the industry, exposed as a first-class Spring AI type so it plugs directly into JUnit assertions rather than requiring a bespoke evaluation harness.

---

7.11 Hallucination Reduction — The Concrete Levers

1. allowEmptyContext(false) — refuse rather than guess when retrieval is empty (§7.3).
2. Explicit “answer only from context” instruction in the augmentation template (default behavior, but verify if you’ve customized the template).
3. similarityThreshold floor at the retrieval stage (Section 6) — don’t inject noise as if it were relevant context.
4. Reranking (§7.7) — ensures the highest-relevance documents, not just highest-raw-similarity ones, make it into context.
5. Citation requirements in the prompt template — forcing the model to attribute claims to specific retrieved sources makes ungrounded claims more visually/structurally detectable in review, and models tend to hedge more honestly when asked to cite.
6. RelevancyEvaluator in CI (§7.10) — catch grounding regressions before they reach production, not after a user reports a wrong answer.

---

7.12 Common Mistakes

1. Skipping query transformation in multi-turn RAG — a bare follow-up question sent directly to vector search retrieves garbage because it lacks the referent from prior turns.
2. Not setting allowEmptyContext(false) — silently reverts to ungrounded generation exactly when the knowledge base has a gap.
3. Single-stage retrieval with no reranking — leaves retrieval quality on the table for a relatively small additional latency cost.
4. Ignoring header/footer noise during ingestion — measurably degrades chunk embedding quality at scale.
5. No RAG-specific evaluation in CI — regressions in chunking, prompt template changes, or embedding model upgrades go undetected until a user notices a bad answer.
6. Fixed, untested chunk size chosen without considering document type — legal/technical corpora and FAQ corpora have really different optimal chunking, not just a nice-to-tune parameter.

---

7.13 Interview Questions

1. Why does Spring AI decompose RAG into QueryTransformer/DocumentRetriever/QueryAugmenter instead of one RAG service class?
2. What problem does CompressionQueryTransformer solve in multi-turn conversational RAG specifically?
3. What does allowEmptyContext(false) actually change about model behavior, and why is it a hallucination-reduction lever?
4. Why does Spring AI’s TokenTextSplitter split on token count rather than character count?
5. What’s the trade-off between small and large chunk sizes, and how would you choose for a legal-document corpus versus an FAQ corpus?
6. Why is a two-stage retrieve-then-rerank pipeline generally superior to single-stage top-K retrieval at equivalent final context size?
7. What is TokenCountEstimator actually estimating, and why is it not exact for every model family?
8. How would you wire a custom reranker into RetrievalAugmentationAdvisor?
9. What does RelevancyEvaluator use internally to score a RAG response, and how would you integrate it into a CI pipeline?
10. Why does stripping page headers/footers during PDF ingestion measurably affect retrieval quality?
11. Describe the full production pipeline from raw document upload to a cited, grounded chat response.
12. What’s the risk of sending a bare follow-up question directly to vector similarity search without query transformation?
13. How would you implement citation support in a RAG response, and where does that logic live architecturally?
14. What’s the purpose of chunk overlap, and what failure mode does it prevent?
15. Why is MultiQueryExpander useful, and how are results from multiple query variants merged?
16. What’s the relationship between DocumentPostProcessor and the Advisor pattern established in Section 3?
17. How would you test that a prompt-template change to ContextualQueryAugmenter didn’t regress groundedness?
18. What’s the architectural reason RAG in Spring AI is “just” a specific Advisor composition rather than a separate subsystem?
19. Why might you over-fetch topK=20 documents but only inject 5 into the final context?
20. What’s the difference between PagePdfDocumentReader and TikaDocumentReader, and when would you choose one over the other?

---

7.14 Best Practices Checklist

• Always apply query transformation/compression for multi-turn conversational RAG.
• Set allowEmptyContext(false) — never let empty retrieval silently fall back to ungrounded generation.
• Over-retrieve and rerank; don’t rely on raw similarity rank as final relevance.
• Strip boilerplate headers/footers during ingestion.
• Include source citations in the augmentation template for auditability and reduced hallucination surface.
• Run RelevancyEvaluator-based checks in CI against a fixed regression test set of question/expected-grounding pairs.
• Choose chunk size deliberately per corpus type, not as a global default copied from a tutorial.

---

7.15 Key Takeaways

• RAG in Spring AI is a composition of small, swappable Advisor-pipeline components, not a monolithic subsystem.
• Query transformation is not optional for multi-turn RAG — it’s the difference between working and broken follow-up questions.
• Retrieve-then-rerank is the production-grade pattern; single-stage top-K retrieval leaves quality on the table.
• Hallucination reduction is a set of concrete, composable levers (empty-context refusal, threshold floors, reranking, citations, CI evaluation) — not a single switch.
• Ingestion quality (header/footer stripping, chunk sizing) has outsized downstream impact on retrieval quality that’s easy to underweight relative to query-time tuning.

---

End of Section 7. Next: Section 8 — Memory (Chat Memory, Conversation Memory, Persistent Memory, Redis Memory, JDBC Memory, Memory Strategies, Memory Window, Summarization, Production Design).