Section 4 — ChatModel: Multi-Provider Integration

Begin with the problem

One interface, different model providers

OpenAI, Anthropic, Google, Ollama, and other providers do not have identical APIs or capabilities. ChatModel gives your application a common starting point without pretending every provider is identical.

Prompt → ChatModel interface → chosen provider → ChatResponse

What you will learn

Configure a provider-backed ChatModel.
Separate portable features from provider-specific features.
Compare providers using measured requirements.
Plan routing and fallback without assuming identical behavior.

Current official reference: Spring AI documentation for this topic. The examples below primarily preserve the stated 1.1.x course target. Where Spring AI 2.0 differs, the text must treat that behavior as version-specific rather than universal.

(Continues from Section 3. Target: Spring AI 1.1.x / Spring Boot 3.5.x.)

4.1 Why This Section Isn’t “How to Call 8 Different APIs”

Every provider’s raw HTTP shape differs — different auth headers, different JSON field names, different rules about what’s required. What actually matters at the architecture level is: every ChatModel implementation is a translator between Spring AI’s generic Prompt/ChatResponse and one provider’s wire DTOs, and understanding where the seams are is what lets you build real production patterns — fallback, routing, cost-based selection — on top without fighting the abstraction.

Real-world analogy — Freight Forwarding: A freight forwarder (Spring AI) doesn’t own ships, trucks, or planes. It has standard contracts with carriers (OpenAI, Anthropic, Ollama…) and a common manifest format (Prompt) that gets translated into each carrier’s specific paperwork. If FedEx (OpenAI) has a customs delay, the forwarder reroutes through DHL (Anthropic) — that’s provider fallback — without the shipper (your service code) needing to know or care which carrier actually moved the freight.

Analogy: The Airline Booking Desk & Code-Share Routing Imagine booking a flight from San Francisco to Paris:

The Interface (Delta Booking Desk): You book a single ticket through Delta Airlines (Spring AI ChatModel). You don’t have to know details about local airport gates, specific baggage scales, or pilot credentials for partner flights.

The Partners (Code-Share): Under the hood, Delta doesn’t fly that route themselves. They route your actual seat to Air France (Anthropic), KLM (OpenAI), or a small domestic hop (local Ollama server).

Fallback & Recovery: If the KLM (OpenAI) flight gets delayed by a storm (transient rate-limit API exception), the desk agent at the counter automatically prints a new ticket routing you on the Air France (Anthropic) flight instead. You don’t have to leave the airport or buy a new ticket; your Delta ticket remains valid.

📊 Visual Flowchart: Multi-Provider Fallback and Routing Architecture

Here is how routing decisions and fallback catch triggers operate inside Spring AI:

graph TD
    classDef fail stroke:#e74c3c,stroke-width:2px;
    
    UserPrompt["Prompt Request"] --> Router{"1. Router: check TaskType"}
    
    Router -->|SIMPLE_CLASSIFICATION| CheapModel["2. gpt-4o-mini ChatModel<br>(Fast & cost-efficient)"]
    Router -->|COMPLEX_REASONING| FrontierModel["2. Primary: gpt-4o ChatModel<br>( Frontier Model)"]
    
    FrontierModel -->|API success| Return["Return ChatResponse"]
    
    FrontierModel -->|Transient Exception<br>e.g. Rate Limit / Timeout| Catch{"3. Fallback Exception Catch"}:::fail
    
    Catch -->|Reroute| SecondaryModel["4. Secondary: Claude Sonnet Model<br>(Anthropic fallback)"]
    
    SecondaryModel -->|Success| Return
    SecondaryModel -->|Fail| FailOut["Throw AllProvidersUnavailableException"]:::fail

4.2 The Provider Matrix — What’s Actually Different

Provider	Module	Auth	Streaming	Tool Calling	Vision	Notable Spring AI-specific behavior
OpenAI	`spring-ai-openai`	API key header	SSE	Native, parallel calls supported	Yes	Also backs Azure OpenAI since the Azure module was folded in (1.1.5+) — same `OpenAiChatModel`, different `baseUrl`/auth
Anthropic	`spring-ai-anthropic`	`x-api-key` header	SSE	Native (check current model support for parallel vs. single-call-at-a-time)	Yes (Claude 3+)	Rate-limit metadata exposed via `ChatResponseMetadata` as of recent releases
Google Gemini (Vertex AI)	`spring-ai-vertex-ai-gemini`	GCP service account / ADC	SSE (gRPC-backed under the hood in some paths)	Native	Yes	Requires GCP project/location config, distinct from a simple API-key model
Ollama	`spring-ai-ollama`	None (local daemon)	Yes, over local HTTP	Model-dependent (depends on the local model’s tool-calling support)	Model-dependent	No network egress — critical for on-prem/air-gapped deployments
DeepSeek	Community/OpenAI-compatible endpoint	API key	Yes	Model-dependent	No (text-focused models primarily)	Typically integrated via `spring-ai-openai` pointed at DeepSeek’s OpenAI-compatible endpoint, not a dedicated module
Mistral AI	`spring-ai-mistral-ai`	API key	Yes	Native	Limited	Distinct `MistralAiApi` DTOs; a recent fix improved Jackson mapping for message content (per 2.0.0-M8 notes)
Groq	OpenAI-compatible endpoint	API key	Yes	Model-dependent	No	Also typically wired via `spring-ai-openai` with a custom `baseUrl` — Groq’s value is inference speed (LPU hardware), not a different API shape
Together AI	OpenAI-compatible endpoint	API key	Yes	Model-dependent	Model-dependent	Same OpenAI-compatible pattern as DeepSeek/Groq

Key architectural fact: several “providers” in this list aren’t separate Spring AI modules at all — DeepSeek, Groq, and Together AI expose OpenAI-compatible REST APIs, so the idiomatic Spring AI integration is spring-ai-openai with a custom baseUrl and API key, not a dedicated provider module. Don’t go looking for spring-ai-deepseek — it doesn’t need to exist.

@Bean
public OpenAiApi deepSeekApi(@Value("${deepseek.api-key}") String apiKey) {
    return OpenAiApi.builder()
            .baseUrl("https://api.deepseek.com")
            .apiKey(apiKey)
            .build();
}

@Bean
public OpenAiChatModel deepSeekChatModel(OpenAiApi deepSeekApi) {
    return OpenAiChatModel.builder()
            .openAiApi(deepSeekApi)
            .defaultOptions(OpenAiChatOptions.builder()
                    .model("deepseek-chat")
                    .build())
            .build();
}

4.3 Configuration Per Provider

spring:
  ai:
    openai:
      api-key: ${OPENAI_API_KEY}
      chat:
        options:
          model: gpt-4o
          temperature: 0.7
    anthropic:
      api-key: ${ANTHROPIC_API_KEY}
      chat:
        options:
          model: claude-sonnet-4-6
          max-tokens: 4096
    vertex:
      ai:
        gemini:
          project-id: ${GCP_PROJECT_ID}
          location: us-central1
          chat:
            options:
              model: gemini-2.0-flash
    ollama:
      base-url: http://localhost:11434
      chat:
        options:
          model: llama3.1
    mistralai:
      api-key: ${MISTRAL_API_KEY}
      chat:
        options:
          model: mistral-large-latest

Each block independently triggers its own auto-configuration class — the moment you have two or more provider starters on the classpath simultaneously, you’ll hit the NoUniqueBeanDefinitionException from Section 1 unless you qualify.

4.4 Switching Providers — The Portable Path

If your service code depends only on ChatModel (interface) and generic ChatOptions, switching is a config + dependency change:

@Service
public class SummarizationService {

    private final ChatClient chatClient;

    // constructor injection of a ChatClient built from ANY provider's ChatModel —
    // this class has zero compile-time knowledge of which provider is behind it
    public SummarizationService(ChatClient.Builder builder) {
        this.chatClient = builder
                .defaultSystem("Summarize concisely.")
                .build();
    }

    public String summarize(String text) {
        return chatClient.prompt().user(text).call().content();
    }
}

The moment you need OpenAiChatOptions.builder().logprobs(true) or AnthropicChatOptions.builder().thinking(...) for a specific feature, that call site becomes provider-coupled — which is fine and expected; the goal isn’t zero coupling everywhere, it’s isolating the coupling to the call sites that actually need provider-specific features, while the rest of your codebase stays portable.

4.5 Multi-Provider Strategy — Real Production Patterns

4.5.1 Fallback Chain

@Service
public class ResilientChatService {

    private final ChatModel primary;     // e.g., OpenAI
    private final ChatModel fallback;    // e.g., Anthropic

    public ResilientChatService(
            @Qualifier("openAiChatModel") ChatModel primary,
            @Qualifier("anthropicChatModel") ChatModel fallback) {
        this.primary = primary;
        this.fallback = fallback;
    }

    public ChatResponse call(Prompt prompt) {
        try {
            return primary.call(prompt);
        } catch (TransientAiException | NonTransientAiException e) {
            log.warn("Primary provider failed, falling back", e);
            try {
                return fallback.call(prompt);
            } catch (Exception fallbackEx) {
                fallbackEx.addSuppressed(e);
                throw new AllProvidersUnavailableException(fallbackEx);
            }
        }
    }
}

Production nuance: TransientAiException (rate limits, 5xx, timeouts) is worth falling back on; NonTransientAiException (400 bad request, content-policy rejection) usually is not — the same malformed request will fail identically on the fallback provider, and you’ve just doubled your latency for a guaranteed-failure path. Distinguish these in your fallback logic rather than catching Exception broadly.

4.5.2 Routing by Task Type (Cost/Capability-Based)

@Service
public class RoutingChatService {

    private final Map<TaskType, ChatModel> modelsByTask;

    public RoutingChatService(
            @Qualifier("gpt4oChatModel") ChatModel highCapability,
            @Qualifier("gpt4oMiniChatModel") ChatModel cheapFast,
            @Qualifier("claudeChatModel") ChatModel longContext) {
        this.modelsByTask = Map.of(
                TaskType.COMPLEX_REASONING, highCapability,
                TaskType.SIMPLE_CLASSIFICATION, cheapFast,
                TaskType.LONG_DOCUMENT_ANALYSIS, longContext
        );
    }

    public ChatResponse route(TaskType taskType, Prompt prompt) {
        return modelsByTask.getOrDefault(taskType, modelsByTask.get(TaskType.COMPLEX_REASONING))
                .call(prompt);
    }
}

This is the production reality for cost control: routing cheap classification/extraction tasks to a small fast model (gpt-4o-mini, Groq-hosted Llama for latency-critical paths) while reserving expensive frontier models for tasks that actually need the reasoning capability — Section 16 — Performance covers token-cost optimization in depth, but the routing mechanism itself is architecturally simple: a Map<Enum, ChatModel> keyed by task classification, nothing more exotic.

4.5.3 Load Balancing Across Multiple API Keys/Regions

For high-throughput scenarios hitting provider rate limits on a single key:

@Component
public class RoundRobinChatModel implements ChatModel {

    private final List<ChatModel> pool;
    private final AtomicInteger counter = new AtomicInteger(0);

    public RoundRobinChatModel(List<ChatModel> pool) {
        this.pool = pool;
    }

    @Override
    public ChatResponse call(Prompt prompt) {
        int index = counter.getAndIncrement() % pool.size();
        return pool.get(index).call(prompt);
    }

    @Override
    public ChatOptions getDefaultOptions() {
        return pool.get(0).getDefaultOptions();
    }
}

This wraps N ChatModel instances (same provider, different API keys/regional endpoints, each with independent rate-limit buckets) behind one ChatModel-shaped facade — because ChatModel is just an interface, you can implement it yourself to build exactly this kind of infrastructure-level composition. This is the same pattern as a DataSource connection pool: the interface abstraction is what makes composition like this possible without touching a single call site.

4.6 Internal Working — Tracing One Real Call (Anthropic)

AnthropicChatModel.call(Prompt prompt)
        │
        ▼
1. AnthropicChatOptions merged: defaultOptions (bean-level) overridden by
   prompt.getOptions() (request-level) field-by-field
        │
        ▼
2. Prompt.getInstructions() (List<Message>) mapped to Anthropic's
   Messages API shape: SystemMessage extracted separately (Anthropic's
   API takes system as a TOP-LEVEL field, NOT a message in the array —
   this is a real wire-format difference Spring AI's AnthropicChatModel
   handles for you so your Message list stays provider-agnostic)
        │
        ▼
3. AnthropicApi.ChatCompletionRequest DTO built, tools serialized to
   Anthropic's tool-use JSON Schema format (different field names than
   OpenAI's function-calling schema — another translation Spring AI owns)
        │
        ▼
4. RestClient POST https://api.anthropic.com/v1/messages
   with anthropic-version header, x-api-key header
        │
        ▼
5. Response parsed: content blocks (text + tool_use blocks) mapped back
   to a generic AssistantMessage with ToolCall list
        │
        ▼
6. ChatResponseMetadata populated: usage (input/output tokens),
   rate-limit headers surfaced (anthropic-ratelimit-* headers →
   ChatResponseMetadata.getRateLimit(), a recent addition per 2026
   release notes)
        │
        ▼
7. Generic ChatResponse returned — indistinguishable in shape from
   what OpenAiChatModel or VertexAiGeminiChatModel would return

The system-message-as-top-level-field detail in step 2 is a really useful “why” to internalize: it’s a concrete example of exactly what the portability layer buys you — you write new SystemMessage(...) once, and each provider module decides whether that becomes an array element (OpenAI) or a top-level request field (Anthropic).

4.7 Common Mistakes

Assuming ChatOptions fields map 1:1 across providers — maxTokens is required on Anthropic’s API (no default), optional with a provider-side default on OpenAI; omitting it against Anthropic throws, omitting it against OpenAI doesn’t.
Building a fallback chain that retries NonTransientAiException — wastes latency retrying guaranteed failures (content policy violations, malformed schema) against a second provider.
Not qualifying beans in multi-provider setups — see Section 1; this remains the #1 startup failure once you add a second provider starter.
Wiring DeepSeek/Groq/Together AI as if they need bespoke modules — they’re OpenAI-compatible; reusing spring-ai-openai with a custom baseUrl is simpler and avoids maintaining redundant bean configuration.
Ignoring rate-limit metadata — ChatResponseMetadata.getRateLimit() (where the provider exposes it) tells you exactly how close you are to throttling; production routing/backoff logic that ignores this and only reacts to actual 429s is reactive instead of proactive.
Hardcoding model name strings scattered across the codebase — centralize model identifiers as constants or config properties; a provider’s model deprecation (frequent, given the CVE/release cadence you saw in Section 1) shouldn’t require a grep-and-replace across services.

4.8 Debugging

logging:
  level:
    org.springframework.ai.anthropic: DEBUG
    org.springframework.web.client.RestClient: DEBUG   # logs raw HTTP request/response

For provider-specific error diagnosis, always inspect the exception chain — Spring AI wraps provider HTTP errors in NonTransientAiException/TransientAiException, but the cause typically still carries the provider’s raw error body (e.g., OpenAI’s {"error": {"type": "invalid_request_error", ...}}), which is where the actionable detail lives:

catch (NonTransientAiException e) {
    log.error("Provider rejected request: {}", e.getCause() != null
            ? e.getCause().getMessage() : e.getMessage());
}

4.9 Interview Questions

Why do DeepSeek, Groq, and Together AI not have dedicated Spring AI provider modules, and how do you integrate them idiomatically?
Walk through the wire-format difference in how Anthropic and OpenAI handle system prompts, and where Spring AI abstracts that difference away.
What’s the architectural difference between TransientAiException and NonTransientAiException, and why does that distinction matter for fallback logic?
How would you implement a cost-based routing service that sends simple classification tasks to a cheap model and complex reasoning tasks to a frontier model?
Why can you implement ChatModel yourself to build a round-robin load balancer across multiple API keys, and what does that tell you about the interface’s design intent?
What happens if you omit maxTokens when calling Anthropic vs. OpenAI, and why does the difference exist?
Where does rate-limit metadata surface in ChatResponse, and how would you use it for proactive backoff instead of reactive 429 handling?
What’s the correct way to configure Azure OpenAI post-1.1.5, given the dedicated Azure module was removed?
Describe the full translation path for a tool-calling request from generic ToolCallback to Anthropic’s tool-use JSON Schema format.
Why is retrying a NonTransientAiException against a fallback provider usually wasted latency?
How would you configure Ollama for an air-gapped, no-external-egress deployment, and what changes about your fallback strategy in that environment?
What’s the risk of hardcoding model name strings across many services, given the provider release cadence you saw in Section 1’s version history?
Explain how ChatOptions merging works between bean-level defaultOptions and request-level prompt.getOptions().
What class-level abstraction makes a round-robin ChatModel wrapper possible without touching any call site’s code?
How does Spring AI represent Gemini’s Vertex AI authentication differently from OpenAI’s simple API-key model, and what does that imply for local development setup?
What debugging technique reveals the actual provider error body when a NonTransientAiException is thrown with a generic message?
Why might parallel tool calls be supported on OpenAI but require different handling on another provider, and how would you write portable tool-calling code around that difference?
Describe a production incident where provider fallback made latency worse instead of better, and what guardrail would prevent it.
What’s the trade-off between routing to a fast/cheap model for a task versus always using the highest-capability model, beyond raw cost?
How would you structure integration tests that verify your fallback chain actually engages the secondary provider under simulated primary-provider failure?

4.10 Best Practices Checklist

Use spring-ai-openai with a custom baseUrl for any OpenAI-compatible provider (DeepSeek, Groq, Together AI) instead of hunting for a dedicated module.
Distinguish TransientAiException from NonTransientAiException in every fallback/retry path.
Centralize model identifiers as configuration properties, never inline string literals scattered across services.
Explicitly qualify every ChatModel bean the moment a second provider is added to the classpath.
Build routing/fallback logic against the ChatModel interface so it’s testable with mocks and swappable without touching call sites.
Monitor rate-limit metadata proactively rather than reacting only to thrown 429 exceptions.

4.11 Key Takeaways

Several “providers” are really spring-ai-openai pointed at a different baseUrl — don’t over-engineer integration for OpenAI-compatible APIs.
The portability layer’s real value shows up in wire-format differences you’d otherwise hand-roll yourself (system-prompt placement, tool schema shape, required vs. optional fields).
Fallback, routing, and load balancing are all buildable as plain ChatModel implementations/compositions — no special Spring AI multi-provider API is needed because the interface itself is the extension point.
Distinguish transient from non-transient failures before deciding whether a fallback attempt is worth the latency cost.

End of Section 4. Next: Section 5 — Embeddings (EmbeddingModel, batch embeddings, caching, similarity, optimization).

Author: TechByteByByte Editorial Team
Reviewed by: TechByteByByte Admin
Published: August 31, 2026
Last reviewed: August 31, 2026