Section 16 — Performance

Begin with the problem

Finding where AI time and money go

In an AI endpoint, the model call and token volume often dominate. Performance work begins by measuring queue time, time to first token, total latency, retries, and tokens—not by guessing.

request → local work → provider queue/inference → stream → response

What you will learn

Break latency into measurable stages.
Reduce unnecessary context and model calls.
Use caching, batching, streaming, and concurrency carefully.
Balance speed, quality, cost, and reliability.

Current official reference: Spring AI documentation for this topic. The examples below primarily preserve the stated 1.1.x course target. Where Spring AI 2.0 differs, the text must treat that behavior as version-specific rather than universal.

(Continues from Section 15. Target: Spring AI 1.1.x / Spring Boot 3.5.x.)

16.1 The Performance Model — Where Time and Money Actually Go

Unlike a typical CRUD service where database query time usually dominates, an AI-backed request’s latency and cost breakdown looks fundamentally different: model inference time dominates latency (frequently 1-30+ seconds, vastly larger than typical DB/network overhead), and token count dominates cost, not request count. This reframes where optimization effort actually pays off — shaving milliseconds off your Advisor chain’s local processing is usually not where the real gains are; reducing token usage and avoiding unnecessary model calls almost always is.

Real-world analogy — Optimizing a Restaurant’s Cooking Time vs. Its Seating Process: Most of this section’s guidance is about reducing how much you ask the kitchen to cook (fewer, better-targeted requests to the model — token optimization) rather than optimizing how fast the host seats people (local Java/Spring processing overhead, which is real but comparatively small next to model latency/cost).

Analogy: The Sports Car Fuel Economy Tuning Think of managing performance and resource usage in your Spring AI application as tuning a high-performance sports car:

The Bottleneck (The Engine Fuel Burn): Model inference time (the actual execution in the LLM’s remote engine) is like your fuel rate. It burns 95% of your total budget (latency and API fees). Shaving 2 milliseconds off your local Java code is like wiping dust off the car’s mirror — it looks nice, but it doesn’t change your fuel efficiency.

The Cache (Stoplight Idle Cutoff): Turning off the engine at red lights (caching embedding vectors for duplicate query texts) avoids burning fuel when you’re not moving.

The Small Model (City Commuter Car): You don’t use a heavy truck (GPT-4o) to fetch a single envelope from down the block (simple intent classification). You drive a tiny commuter car (GPT-4o-mini / Llama 3B) to save 90% of your fuel.

📊 Visual Flowchart: Spring AI Performance Optimization Pipeline

Here is how cache lookups, token trimming, and routing strategies combine to optimize request speeds and reduce API costs:

graph TD
    UserReq["User Prompt Request"] --> CacheCheck{"1. Local Cache Lookup<br>(Redis MD5 hash search)"}

    CacheCheck -->|Cache Hit| ReturnCache["Return cached float[] / String response<br>(Latency: ~5ms, Cost: $0.00)"]
    CacheCheck -->|Cache Miss| TrimContext["2. Trim Context Budget<br>(Trim old history, strip headers/footers)"]

    TrimContext --> Router{"3. Task Router Selector"}

    Router -->|Simple / Fast| CheapModel["4. Low-Cost Fast Model<br>(gpt-4o-mini / Ollama local)<br>Latency: ~300ms, Cost: Cheap"]
    Router -->|Complex / Vision| FrontierModel["4. Frontier Reasoning Model<br>(gpt-4o / Claude Sonnet)<br>Latency: ~3s, Cost: Standard"]

    CheapModel --> Return["Return final response"]
    FrontierModel --> Return

16.2 Token Optimization — The Highest-Leverage Lever

16.2.1 System Prompt Efficiency

A verbose, redundant system prompt is paid for on every single call in that conversation — this compounds significantly over a long conversation history if the system prompt isn’t cached (see §16.2.3) or if memory strategy re-sends it every turn:

// WASTEFUL — verbose, repetitive instructions
"You are a helpful, friendly, and knowledgeable customer support
assistant. You should always be helpful. You should always be
friendly. You should always provide knowledgeable answers. Please
make sure to be helpful in every response you give..."

// EFFICIENT — same intent, ~70% fewer tokens
"You are a friendly, knowledgeable customer support assistant.
Be concise and accurate."

16.2.2 Context Window Discipline

Section 7’s over-retrieve-then-rerank pattern exists partly for quality, but the final injected context size is also a direct cost lever — injecting topK=20 raw retrieved documents instead of reranking down to 5 multiplies your per-request input token cost by 4x for often-marginal quality gain beyond what reranking-and-trimming achieves. Similarly, Section 8’s memory window size is a direct, linear cost lever — maxMessages=50 costs meaningfully more per turn than maxMessages=15 once a conversation is long, for often-diminishing UX return.

16.2.3 Prompt Caching (Provider-Level)

Several providers (notably Anthropic, and OpenAI for sufficiently large/stable prompt prefixes) support prompt caching — a stable prefix (a long system prompt, a large set of few-shot examples, a large RAG-injected document) can be cached provider-side across requests, billed at a steep discount on cache hits versus full-price on cache misses:

AnthropicChatOptions options = AnthropicChatOptions.builder()
        .model("claude-sonnet-4-6")
        // structure the request so the STABLE portion (system prompt,
        // few-shot examples) comes first and is marked cacheable,
        // with the variable portion (the actual user question) last —
        // cache hit rate depends directly on this ordering discipline;
        // check the current API surface for exact cache-control
        // configuration as this is an actively evolving capability
        .build();

Production impact: for a support-bot system prompt of, say, 2,000 tokens sent on every one of thousands of daily requests, prompt caching alone can meaningfully reduce that portion of cost — this is one of the highest-leverage, lowest-effort optimizations available when your provider supports it and your prompt structure has a really stable, reusable prefix.

16.2.4 Model Selection Per Task (Recap from Section 4)

The routing pattern from Section 4 is fundamentally a token-cost optimization at the model-selection level: a gpt-4o-mini-class model can cost an order of magnitude less per token than a frontier model — routing simple classification/extraction tasks there while reserving frontier models for tasks that really need the reasoning capability is often the single largest cost lever available, larger than any prompt-trimming technique.

16.3 Caching Beyond Embeddings (Recap + Extension of Section 5)

Section 5 covered embedding caching. The same principle extends to full chat responses for really repeated queries:

@Component
public class CachingChatService {

    private final ChatClient chatClient;
    private final Cache responseCache;

    public String answer(String question) {
        String cacheKey = DigestUtils.sha256Hex(question.trim().toLowerCase());
        String cached = responseCache.get(cacheKey, String.class);
        if (cached != null) {
            return cached;
        }
        String response = chatClient.prompt().user(question).call().content();
        responseCache.put(cacheKey, response);
        return response;
    }
}

Where this really helps: exact-duplicate FAQ-style queries at high volume (very common in high-traffic public-facing support bots — “what is your return policy” phrased identically by many users). Where it doesn’t help: conversational, personalized, or really-varied queries, where exact-text-match caching will rarely hit. Semantic caching (cache-hit on similar, not identical, queries via an embedding-similarity lookup) is a more sophisticated variant worth considering for higher hit rates, at the cost of implementation complexity and the risk of serving a subtly-wrong cached answer for a similar-but-not-identical question — evaluate that risk carefully against your domain’s tolerance for imprecise cache hits.

16.4 Connection Pooling

@Bean
public RestClient.Builder restClientBuilder() {
    ConnectionConfig connectionConfig = ConnectionConfig.custom()
            .setConnectTimeout(Timeout.ofSeconds(5))
            .setSocketTimeout(Timeout.ofSeconds(60))
            .build();

    PoolingHttpClientConnectionManager connectionManager =
            PoolingHttpClientConnectionManagerBuilder.create()
                    .setMaxConnTotal(200)
                    .setMaxConnPerRoute(50)   // per-provider-host connection cap
                    .setDefaultConnectionConfig(connectionConfig)
                    .build();

    CloseableHttpClient httpClient = HttpClients.custom()
            .setConnectionManager(connectionManager)
            .build();

    return RestClient.builder()
            .requestFactory(new HttpComponentsClientHttpRequestFactory(httpClient));
}

Default HTTP client connection pool settings are frequently too conservative for a production AI service making many concurrent outbound calls to the same provider host — explicit pool sizing (as shown) avoids connection-establishment overhead (TCP + TLS handshake) becoming a meaningful fraction of overall latency under concurrent load, distinct from and additive to the model-inference-time cost that dominates single-request latency.

16.5 Threading

For blocking .call() usage in an MVC application under load, thread pool sizing needs to account for the fact that AI calls hold a thread for seconds, not milliseconds — a default Tomcat thread pool sized for typical fast CRUD endpoints will exhaust rapidly under AI-backed endpoint load:

server:
  tomcat:
    threads:
      max:
        400 # sized meaningfully higher than default (200) given
        # multi-second hold times per AI-backed request —
        # calculate based on target concurrent AI requests
        # × expected hold-time, not copied from a generic
        # web-service sizing guideline
      min-spare: 50

For really high-concurrency AI workloads, migrating the AI-backed endpoints specifically to WebFlux (Section 12) is usually a more scalable architectural answer than simply growing the Tomcat thread pool indefinitely — thread-per-request models fundamentally don’t scale as well to long-held-connection workloads as an event-loop model does, though the migration cost is real and shouldn’t be undertaken reflexively for moderate load.

16.6 Batching (Recap + Broader Application)

Beyond Section 5’s embedding batching, the same principle applies to any bulk LLM-processing workload — document summarization for 1,000 documents, classification of a bulk dataset:

public List<ClassificationResult> classifyBulk(List<String> texts) {
    // batch into a SINGLE prompt asking the model to classify N items
    // at once, rather than N separate ChatClient calls — reduces
    // per-request overhead (fixed system-prompt cost amortized across
    // N items) at the cost of requiring the model to produce N
    // structured results reliably in one response, which has its own
    // failure-mode considerations from Section 11 (a partial parse
    // failure now affects the whole batch, not just one item)
    String batchPrompt = buildBatchClassificationPrompt(texts);
    return chatClient.prompt().user(batchPrompt).call()
            .entity(new ParameterizedTypeReference<List<ClassificationResult>>() {});
}

The trade-off to weigh explicitly: batching N items into one call amortizes fixed prompt overhead and reduces total request count/latency, but couples their success/failure together (Section 11’s structured-output parsing risk now applies to the whole batch) and risks hitting context-window/output-token limits for large N. A practical middle ground — batch in groups of 10-50 rather than either one-at-a-time or all 1,000 at once — is the common production pattern, mirroring the same batch-size reasoning from Section 5’s embedding guidance.

16.7 Parallel Calls

For independent sub-tasks (e.g., extracting three different fact categories from the same document, or querying multiple data sources to answer one composite question), parallel model calls reduce wall-clock latency at the cost of proportionally higher concurrent API usage:

public CompositeAnswer answerComposite(String question) {
    CompletableFuture<String> factualAnswer = CompletableFuture.supplyAsync(
            () -> factClient.prompt().user(question).call().content(), executor);
    CompletableFuture<String> sentimentAnalysis = CompletableFuture.supplyAsync(
            () -> sentimentClient.prompt().user(question).call().content(), executor);

    return new CompositeAnswer(factualAnswer.join(), sentimentAnalysis.join());
}

Bound the executor’s thread pool deliberately (Section 4/5’s concurrency-limit guidance applies identically here) — parallel calls that individually respect provider rate limits can collectively exceed them if unbounded, triggering the exact retry-storm degradation pattern already covered in Sections 4 and 5.

16.8 Latency Reduction Checklist

Time-to-first-token via streaming (Section 12) — even when total generation time is unchanged, perceived latency drops substantially when tokens appear incrementally rather than all at once.
Smaller/faster models for latency-sensitive paths (Section 4’s Groq-for-speed example) — really different hardware/model trade-offs exist specifically optimized for low-latency inference.
Reduce context size (§16.2.2) — less input to process is directly faster, not just cheaper, since input processing time scales with input length too, not just output generation.
Parallelize independent sub-calls (§16.7) rather than sequential chaining where task structure allows it.
Provider/region selection — network latency to the provider’s nearest region endpoint is a real, sometimes-overlooked component of total latency, particularly for teams whose infrastructure isn’t co-located with common provider regions.

16.9 Common Mistakes

Optimizing local Java/Spring processing time first when model inference time dominates overall latency by orders of magnitude — really wasted effort relative to token/prompt-structure optimization.
Ignoring prompt caching where the provider supports it and the prompt structure has a stable, reusable prefix — a high-leverage, low-effort optimization left on the table.
Fixed generic thread pool sizing copied from typical web-service guidance, not recalculated for multi-second AI-call hold times.
Batching too aggressively (all 1,000 items in one call), coupling failure modes and risking context/output limits, versus a sensible middle batch size.
Unbounded parallel call fan-out, individually rate-limit-respecting calls collectively exceeding provider limits.
Exact-text-match caching applied to really varied, personalized queries where it will essentially never hit, wasting implementation effort for no real benefit.

16.10 Debugging Performance Issues

Break down request latency explicitly into: connection/TLS establishment, time-to-first-token, generation duration, and local Advisor/processing overhead — Section 13’s observability instrumentation (custom timers per phase) is what makes this breakdown possible rather than guessing. A request that’s “slow” for local-processing reasons versus one that’s “slow” because the model itself is really taking a long time to generate a long response require completely different fixes, and conflating them wastes debugging effort in the wrong direction.

16.11 Interview Questions

Why does token count, not request count, dominate AI service cost, and how does that reframe optimization priorities versus a typical CRUD service?
Explain prompt caching at the provider level — what structural discipline does it require in how you build your prompts?
Why might routing to a smaller/cheaper model for simple tasks be the single largest cost lever available, larger than prompt-trimming techniques?
What’s the trade-off between batching many items into one LLM call versus one-at-a-time calls, specifically regarding failure-mode coupling?
Why does default Tomcat thread pool sizing frequently under-provision for AI-backed endpoints, and how would you calculate appropriate sizing?
When does migrating to WebFlux become architecturally justified over simply growing a blocking thread pool for AI workloads?
What’s the difference between exact-text-match response caching and semantic caching, and what risk does semantic caching introduce?
Why does reducing context size improve latency, not just cost?
How would you bound a parallel-call fan-out to avoid collectively exceeding provider rate limits even when each individual call respects them?
What’s the practical benefit of streaming for perceived latency even when total generation time is unchanged?
Why is connection pool sizing a real latency factor distinct from model inference time, and how would you size it for a production AI service?
Describe how you’d break down a slow AI-backed request’s latency into distinct diagnosable phases.
What risk does aggressive batching (e.g., 1,000 items in one call) introduce regarding structured-output parsing (recap from Section 11)?
Why might provider/region selection be an overlooked latency factor, and how would you diagnose it?
What’s the argument for calculating thread pool sizing based on concurrent-request-count × hold-time rather than copying generic web-service guidance?
How would you decide whether a “slow” AI-backed request is a local-processing problem or a genuine model-latency problem?
Why does memory window size (Section 8) function as a direct cost lever, not just a UX/context-completeness trade-off?
What’s the relationship between Section 7’s retrieve-then-rerank pattern and cost optimization, beyond its quality benefits?
How would you validate that a semantic cache isn’t serving a subtly-wrong answer for a similar-but-meaningfully-different cached query?
Why is optimizing local Advisor-chain processing time usually low-leverage compared to token/prompt-structure optimization?

16.12 Best Practices Checklist

Prioritize token/prompt-structure optimization over local Java processing optimization — inference time dominates.
Use provider-level prompt caching wherever a stable, reusable prompt prefix exists.
Route tasks to the cheapest model capable of handling them, reserving frontier models for really complex reasoning.
Size HTTP connection pools and thread pools explicitly for multi-second AI-call hold times, not generic web-service defaults.
Batch bulk-processing workloads in moderate group sizes (not one-at-a-time, not all-at-once).
Bound parallel-call fan-out to stay under provider rate limits collectively, not just per-call.
Instrument distinct latency phases (connection, time-to-first-token, generation, local processing) to diagnose slowness accurately.

16.13 Key Takeaways

The performance model for AI-backed services fundamentally differs from typical CRUD services: model inference time and token count dominate latency and cost respectively, reframing where optimization effort pays off.
Token/prompt-structure optimization (caching, context trimming, model routing) is almost always higher-leverage than local processing optimization.
Thread pool and connection pool sizing need explicit recalculation for multi-second AI-call hold times, not generic defaults.
Batching and parallelization both trade increased efficiency for coupled failure modes or collective rate-limit risk — moderate, deliberate sizing beats either extreme.
Accurate performance debugging requires phase-level latency breakdown to distinguish genuine model latency from local processing overhead.

End of Section 16. Next: Section 17 — Enterprise Architecture (Spring AI inside Microservices, API Gateway, Kafka, Event Driven, Async Processing, Distributed Systems, Scalability, Fault Tolerance).

Author: TechByteByByte Editorial Team
Reviewed by: TechByteByByte Admin
Published: August 31, 2026
Last reviewed: August 31, 2026