Section 5 — Embeddings

Begin with the problem

Turning meaning into numbers

Computers compare numbers more easily than sentences. An embedding model turns text into a vector so meaning can be compared mathematically.

text → EmbeddingModel → vector of numbers

What you will learn

• Explain an embedding in plain language.
• Create embeddings through Spring AI.
• Understand dimensions, batching, and model consistency.
• Know when embeddings help and when keyword search is better.

Current official reference: Spring AI documentation for this topic. The examples below primarily preserve the stated 1.1.x course target. Where Spring AI 2.0 differs, the text must treat that behavior as version-specific rather than universal.

(Continues from Section 4. Target: Spring AI 1.1.x / Spring Boot 3.5.x.)

5.1 Why EmbeddingModel Is a Separate Abstraction From ChatModel

Beginner primer: if embeddings are new to you, read the glossary’s embeddings section first. Short version: an embedding turns a piece of text into a list of numbers (a vector) such that texts with similar meaning produce numerically close vectors. This is what powers semantic search — finding relevant documents by meaning, not exact keyword match — which is the foundation Section 6 and Section 7 (RAG) build directly on top of.

What matters at the architecture level here is that Spring AI deliberately gave embeddings their own model interface rather than folding them into ChatModel, because they’re a fundamentally different shape of operation: stateless, high-throughput, batchable, numeric-output — versus chat’s stateful, low-throughput-per-call, text-output nature. Conflating them into one interface would force awkward generics; splitting them lets each interface’s contract match its actual usage pattern.

Real-world analogy — Warehouse Barcode Scanning: A chat call is like a customer service phone call — one conversation, back-and-forth, takes minutes. An embedding call is like a warehouse worker scanning a pallet of barcodes — thousands of items, sub-second per scan, no back-and-forth, pure input→fixed-shape-output. You wouldn’t design the same interface for a phone operator and a barcode scanner; Spring AI doesn’t either.

Analogy: The Barcode Scanner vs. The Customer Service Operator Imagine structuring jobs inside a retail operations office:

• The Customer Operator (ChatModel): Takes calls, handles stateful, unpredictable user requests, and talks back-and-forth for minutes to produce customized text paragraphs.
• The Barcode Scanner (EmbeddingModel): Exists to scan crates of books entering the warehouse. The scanner takes a physical book (text chunk), zips a laser over it, and instantly prints out a fixed-digit numerical UPC label (a float vector coordinate).
• There is no conversation, no memory, and no customization. You feed in the book; it outputs the exact same numbers every time, in milliseconds.
• Because scanning is high-speed and deterministic, you group books onto a conveyer belt and scan them in massive batches (Batching) to avoid stopping the conveyor for every single book.

📊 Visual Flowchart: Parallelized Ingestion Pipeline (Batching)

Here is how bulk document imports slice text into batches and embed them concurrently within safety limits:

graph TD
    Docs["Raw Document Collection (e.g. 1000 Pages)"] --> Split["1. TextSplitter: Slice into 1000 Chunks"]
    Split --> Partition["2. Partition: Group into 10 Batches of 100"]

    subgraph ParallelWorkers ["ThreadPoolExecutor (Size: 4 Concurrency Limit)"]
        Partition --> Batch1["Batch 1 (100 Chunks)"]
        Partition --> Batch2["Batch 2 (100 Chunks)"]
        Partition --> Batch3["Batch 3 (100 Chunks)"]
        Partition --> Batch4["Batch 4 (100 Chunks)"]
    end

    Batch1 -->|Single HTTP Call| ApiCall1["EmbeddingModel API Call"]
    Batch2 -->|Single HTTP Call| ApiCall2["EmbeddingModel API Call"]
    Batch3 -->|Single HTTP Call| ApiCall3["EmbeddingModel API Call"]
    Batch4 -->|Single HTTP Call| ApiCall4["EmbeddingModel API Call"]

    ApiCall1 --> DB["VectorStore.add()"]
    ApiCall2 --> DB
    ApiCall3 --> DB
    ApiCall4 --> DB

---

5.2 The Interface

public interface EmbeddingModel extends Model<EmbeddingRequest, EmbeddingResponse> {

    EmbeddingResponse call(EmbeddingRequest request);

    // convenience overloads built on top of call():
    float[] embed(String text);
    float[] embed(Document document);
    List<float[]> embed(List<String> texts);
    EmbeddingResponse embedForResponse(List<String> texts);

    int dimensions();
}

EmbeddingRequest wraps List<String> instructions + EmbeddingOptions (model name, dimensions override for models that support variable output size like text-embedding-3-*). EmbeddingResponse wraps List<Embedding> (each holding the float[] vector + index) plus EmbeddingResponseMetadata (token usage).

@Service
public class EmbeddingService {

    private final EmbeddingModel embeddingModel;

    public EmbeddingService(EmbeddingModel embeddingModel) {
        this.embeddingModel = embeddingModel;
    }

    public List<float[]> embedChunks(List<String> chunks) {
        // single batched HTTP call for all chunks, NOT one call per chunk —
        // this distinction is the single biggest embedding-pipeline
        // performance lever, covered in §5.4
        EmbeddingResponse response = embeddingModel.embedForResponse(chunks);
        return response.getResults().stream()
                .map(Embedding::getOutput)
                .toList();
    }
}

---

5.3 Auto-Configuration and Provider Notes

spring:
  ai:
    openai:
      embedding:
        options:
          model: text-embedding-3-small
          dimensions: 1536 # text-embedding-3-* supports dimension reduction

Dimension reduction matters operationally: text-embedding-3-large natively outputs 3072 dimensions but supports truncation down to smaller sizes (e.g., 1024, 256) via the dimensions parameter without a proportional quality loss for many use cases — this is a real cost/storage lever (vector storage and similarity search cost scale with dimensionality) that Spring AI exposes directly through OpenAiEmbeddingOptions.builder().dimensions(...). Not every embedding model supports this (older text-embedding-ada-002 does not) — check provider capability before assuming the option is honored; silently ignoring an unsupported option or throwing depends on the specific provider module’s validation.

---

5.4 Batch Embeddings — The Performance-Critical Pattern

This is the single highest-leverage fact in this section. Given a list of 5,000 document chunks to embed for a RAG ingestion pipeline:

// ❌ WRONG — naive loop, 5,000 individual HTTP round trips:
for (String chunk : chunks) {
    float[] vector = embeddingModel.embed(chunk);              // one HTTP call each
    store(vector);
}

// ✅ RIGHT — batched calls respecting the provider's batch-size limit:
int batchSize = 100; // OpenAI's embedding endpoint accepts up to ~2048
                      // inputs per call depending on total token count;
                      // 100-500 is a safe practical batch size balancing
                      // request size against retry-blast-radius on failure
for (List<String> batch : Lists.partition(chunks, batchSize)) {
    EmbeddingResponse response = embeddingModel.embedForResponse(batch);
    List<float[]> vectors = response.getResults().stream()
            .map(Embedding::getOutput).toList();
    storeBatch(batch, vectors);
}

The naive loop isn’t just slower — at 5,000 sequential HTTP round-trips with even 100ms latency each, that’s 8+ minutes of wall-clock time and 5,000x the connection-overhead cost, plus you’ll hit rate limits far sooner (request-count limits, not just token limits, apply on most providers). Batching to ~100-500 items per call turns this into ~10-50 calls.

5.4.1 Parallelizing Batches (Production Ingestion Pipeline)

@Service
public class BulkEmbeddingIngestionService {

    private final EmbeddingModel embeddingModel;
    private final VectorStore vectorStore;
    private final ExecutorService executor =
            Executors.newFixedThreadPool(4); // bounded — respect provider concurrency limits

    public void ingest(List<Document> documents) {
        List<List<Document>> batches = Lists.partition(documents, 100);

        List<CompletableFuture<Void>> futures = batches.stream()
                .map(batch -> CompletableFuture.runAsync(() -> {
                    // VectorStore.add() internally calls the EmbeddingModel
                    // for you (see Section 6) — but for direct control,
                    // embed explicitly here if you need custom metadata
                    // enrichment before storage
                    vectorStore.add(batch);
                }, executor))
                .toList();

        CompletableFuture.allOf(futures.toArray(new CompletableFuture[0])).join();
    }
}

Bound the thread pool deliberately — most embedding providers enforce per-account concurrent-request limits; an unbounded parallel stream here will trigger 429s that then trigger retries, which can paradoxically make ingestion slower than a well-tuned bounded pool.

---

5.5 Caching

Embeddings for the same input text are deterministic (given the same model + version) — recomputing them is pure waste. A cache-aside pattern in front of EmbeddingModel:

@Component
public class CachingEmbeddingModel implements EmbeddingModel {

    private final EmbeddingModel delegate;
    private final Cache cache; // e.g., Spring Cache abstraction backed by Redis

    public CachingEmbeddingModel(EmbeddingModel delegate, CacheManager cacheManager) {
        this.delegate = delegate;
        this.cache = cacheManager.getCache("embeddings");
    }

    @Override
    public EmbeddingResponse call(EmbeddingRequest request) {
        List<String> texts = request.getInstructions();
        List<float[]> results = new ArrayList<>(Collections.nCopies(texts.size(), null));
        List<Integer> missIndexes = new ArrayList<>();
        List<String> missTexts = new ArrayList<>();

        for (int i = 0; i < texts.size(); i++) {
            String key = cacheKey(texts.get(i));
            float[] cached = cache.get(key, float[].class);
            if (cached != null) {
                results.set(i, cached);
            } else {
                missIndexes.add(i);
                missTexts.add(texts.get(i));
            }
        }

        if (!missTexts.isEmpty()) {
            EmbeddingResponse missResponse = delegate.call(
                    new EmbeddingRequest(missTexts, request.getOptions()));
            for (int j = 0; j < missIndexes.size(); j++) {
                float[] vector = missResponse.getResults().get(j).getOutput();
                results.set(missIndexes.get(j), vector);
                cache.put(cacheKey(missTexts.get(j)), vector);
            }
        }

        List<Embedding> embeddings = IntStream.range(0, results.size())
                .mapToObj(i -> new Embedding(results.get(i), i))
                .toList();
        return new EmbeddingResponse(embeddings);
    }

    private String cacheKey(String text) {
        // hash the text + model identifier — cache keys must be
        // model-scoped since different models produce different vectors
        // for identical input text
        return delegate.getClass().getSimpleName() + ":" +
                DigestUtils.sha256Hex(text);
    }

    @Override
    public float[] embed(String text) {
        return call(new EmbeddingRequest(List.of(text), EmbeddingOptions.EMPTY))
                .getResults().get(0).getOutput();
    }

    // ... other interface methods delegate similarly, omitted here only
    // because they're mechanical one-line delegations, not omitted content
}

Where this pays off most: RAG pipelines where the same FAQ/document set gets re-ingested repeatedly during development iteration, or query-time caching when many users ask semantically-repeated questions (“what’s your return policy” phrased 50 different ways won’t cache-hit on exact text match, but exact-duplicate queries — very common in high-traffic support bots — will).

---

5.6 Similarity — What Spring AI Does and Doesn’t Do For You

EmbeddingModel produces vectors. It does not compute similarity — that’s VectorStore’s job (Section 6), or you compute it yourself for one-off comparisons:

public double cosineSimilarity(float[] a, float[] b) {
    double dotProduct = 0.0, normA = 0.0, normB = 0.0;
    for (int i = 0; i < a.length; i++) {
        dotProduct += a[i] * b[i];
        normA += a[i] * a[i];
        normB += b[i] * b[i];
    }
    return dotProduct / (Math.sqrt(normA) * Math.sqrt(normB));
}

Beginner note: this is computing cosine similarity — see the glossary — the standard way to measure how close two embedding vectors are in meaning. The result ranges from -1 (opposite meaning) to 1 (identical meaning); values above roughly 0.7–0.8 are typically treated as “similar enough” for retrieval purposes, though the right threshold is always domain- and model-dependent and should be tuned empirically.

Spring AI does ship a small math helper (SimilarityUtils or equivalent in spring-ai-commons, exact class name check per version) for exactly this — reach for it before hand-rolling, but understand it’s simple vector math, not a service call. This distinction — embedding generation is a network call, similarity computation is local CPU math — is where a lot of engineers new to the space misallocate optimization effort, trying to “cache similarity scores” when what they should cache is the embeddings themselves (§5.5).

---

5.7 Optimization Checklist for Production Embedding Pipelines

1. Batch, always. Never loop-call embed() per item (§5.4).
2. Cache by content hash, scoped per model identifier (§5.5) — model upgrades must invalidate the cache namespace.
3. Use dimension reduction where supported to cut vector storage and similarity-search cost, after validating recall doesn’t regress unacceptably for your use case.
4. Bound concurrency to stay under provider rate limits rather than trigger retry storms.
5. Pre-filter/deduplicate near-identical chunks before embedding — embedding two near-duplicate document chunks wastes both API cost and vector-store storage/query cost downstream.
6. Track token usage via EmbeddingResponseMetadata and feed it into the cost observability pipeline (Section 13) — embedding costs are easy to ignore relative to chat costs per-call, but at ingestion-time bulk volume they’re often the larger line item for a RAG-heavy system.

---

5.8 Common Mistakes

1. Looping embed() one item at a time in ingestion pipelines — the single most common performance bug in Spring AI RAG code.
2. Caching without scoping the key to the model identifier — silently serves stale/wrong vectors after a model upgrade, causing subtle retrieval quality degradation that’s hard to diagnose because nothing errors.
3. Assuming dimension reduction is universally supported — check provider/model capability rather than assuming OpenAiEmbeddingOptions.dimensions(...) works identically everywhere.
4. Confusing similarity computation with embedding generation cost — optimizing the wrong operation.
5. Unbounded parallel embedding calls — triggers rate limiting that makes bulk ingestion slower, not faster.
6. Re-embedding unchanged documents on every ingestion run instead of content-hash-based change detection — wasteful for large, mostly-static document sets refreshed periodically.

---

5.9 Debugging

logging:
  level:
    org.springframework.ai.embedding: DEBUG

For ingestion pipelines, log batch sizes and per-batch latency explicitly — this surfaces both rate-limit throttling (latency spikes with 429 retries in logs) and pathological batch sizing (too-small batches = too many round trips; too-large batches = risk of request-size/token limits and larger retry blast radius on transient failure) far faster than staring at aggregate pipeline duration.

---

5.10 Interview Questions

1. Why does Spring AI define EmbeddingModel as a separate interface from ChatModel rather than a shared Model supertype with generic output?
2. What is the single biggest performance lever in an embedding ingestion pipeline, and why does the naive per-item loop underperform so dramatically?
3. How does dimension reduction on text-embedding-3-* models affect downstream vector-store cost, and what’s the trade-off against retrieval quality?
4. Why must embedding cache keys be scoped to a model identifier, and what failure mode results if they aren’t?
5. Does Spring AI compute cosine similarity for you inside EmbeddingModel? Where does similarity computation actually live?
6. What’s the risk of unbounded parallel embedding calls during bulk ingestion, and how would you bound concurrency correctly?
7. How would you detect and skip re-embedding unchanged documents in a periodic re-ingestion job?
8. What metadata does EmbeddingResponseMetadata expose, and how would you feed it into cost observability?
9. Explain the object shape difference between EmbeddingRequest/EmbeddingResponse and Prompt/ChatResponse.
10. Why might production teams pre-deduplicate near-identical chunks before embedding, beyond simple cost savings?
11. What determines a safe batch size for a provider’s embedding endpoint, and what happens if you exceed it?
12. How would you design a cache-aside EmbeddingModel decorator that handles partial cache hits within a single batch request?
13. What’s the practical difference between embed(String), embed(Document), and embedForResponse(List<String>) on the EmbeddingModel interface?
14. Why is embedding cost often the dominant cost line item for RAG-heavy systems despite being cheaper per-call than chat completions?
15. What happens if you request dimensions truncation on a model that doesn’t support it?
16. How would you structure an ingestion pipeline to be resumable/idempotent if it fails partway through 5,000 documents?
17. What’s the relationship between token usage and embedding cost, and how does batch size affect the token-count-per-request limit you might hit?
18. Why is embedding generation described as a network-bound operation while similarity computation is CPU-bound, and how should that shape your optimization priorities?
19. How would you test a CachingEmbeddingModel decorator’s partial-cache-hit logic without hitting a real embedding API?
20. What observability signals would tell you your embedding pipeline is being rate-limited versus simply under-parallelized?

---

5.11 Best Practices Checklist

• Always batch embedding calls; never loop one-item-at-a-time in ingestion code.
• Scope embedding caches by model identifier; invalidate the cache namespace on model upgrade.
• Use dimension reduction where supported after validating retrieval quality impact for your domain.
• Bound ingestion concurrency to stay under provider rate limits.
• Content-hash documents to skip redundant re-embedding on periodic refresh jobs.
• Track and alert on embedding token usage as a distinct cost line item, not lumped in with chat costs.

---

5.12 Key Takeaways

• EmbeddingModel is a deliberately separate, batch-oriented interface — its usage pattern (high-throughput, stateless) is fundamentally different from ChatModel.
• Batching is the single biggest lever in embedding pipeline performance and cost — this is worth internalizing more than any other fact in this section.
• Similarity computation is not part of EmbeddingModel’s job; it’s local math or VectorStore’s responsibility.
• Caching is high-value because embeddings are deterministic per model version — but only if scoped correctly.
• Dimension reduction is a real production cost lever on supporting models, not a gimmick.

---

End of Section 5. Next: Section 6 — Vector Store (every supported vector store, internal architecture, metadata filtering, hybrid search, performance, production scaling).