Section 13 — Observability

Begin with the problem

Seeing what an AI request actually did

When an answer is slow, costly, or wrong, a single application log is not enough. Observability connects model calls, advisors, tools, retrieval, timing, and token usage into one trace.

request → trace spans + metrics + safe logs → dashboard

What you will learn

Identify Spring AI observations and metrics.
Trace calls across advisors and tools.
Measure latency, errors, and token usage.
Avoid leaking prompts, answers, or secrets into telemetry.

Current official reference: Spring AI documentation for this topic. The examples below primarily preserve the stated 1.1.x course target. Where Spring AI 2.0 differs, the text must treat that behavior as version-specific rather than universal.

(Continues from Section 12. Target: Spring AI 1.1.x / Spring Boot 3.5.x.)

13.1 Why Spring AI’s Observability Is “Just” Micrometer, and Why That’s the Right Call

Spring AI’s decision here is deliberate and worth stating plainly: it does not invent a parallel observability system (unlike LangChain’s LangSmith, a separate product). Every ChatModel, EmbeddingModel, and Advisor call is wrapped in a Micrometer Observation, which means it participates automatically in whatever your application already uses for observability — Prometheus, Grafana, Zipkin, Jaeger, Datadog, whatever your OTel exporter targets — with zero new tooling to adopt.

Real-world analogy — Building-Wide Fire Alarm System vs. Standalone Smoke Detector: LangSmith-style separate observability is a standalone smoke detector you install and monitor independently per room. Spring AI’s Micrometer integration is wiring AI calls into the building’s existing fire-alarm system — same control panel, same alert routing, same maintenance staff, no separate vendor relationship to manage. For an enterprise already running a full observability stack, this is a significant operational simplification, not just a technical detail.

Analogy: The Building-Wide Fire Alarm System vs. Standalone Smoke Detector Imagine managing fire safety inside a large enterprise office skyscraper:

The Standalone Smoke Detector (LangSmith approach): You install a separate detector unit in every single office room. It has its own battery, communicates with its own mobile app, and has its own vendor billing contract. You must train your security guard on a completely separate control interface.

The Integrated Alarm System (Micrometer approach): You wire all room alarm sensors directly into the building’s central electrical conduits. Whenever a sensor triggers (observation), the signal travels over the existing wiring to your main security desk console (OpenTelemetry Jaeger/Zipkin collector).

You manage a single alarm panel, use the same maintenance crew, and alert dispatchers using your standard office protocols.

📊 Visual Chart: Observation Instrumentation Tracing Hierarchy

Here is how an incoming user request maps to nested spans and metrics under Micrometer:

graph TD
    HttpReq["Incoming HTTP Request Span (e.g. GET /api/chat)"] --> ChatClientObs["1. ChatClient Observation<br>(spring.ai.chat.client)"]

    subgraph SpringAI [Spring AI Request Pipeline]
        ChatClientObs --> AdvisorBefore["2. Advisor before-hooks Spans<br>(e.g. Memory lookup duration)"]
        AdvisorBefore --> ChatModelObs["3. ChatModel Observation<br>(spring.ai.chat.model)"]

        ChatModelObs --> ToolCallObs["4. Tool Call Reflection Span<br>(spring.ai.tool.call: lookupOrder)"]

        ToolCallObs --> AdvisorAfter["5. Advisor after-hooks Spans"]
    end

    AdvisorAfter --> MetricExport["5. Export Metrics / Traces<br>(OTel Span exporter & Prometheus meter)"]

13.2 What Gets Instrumented Automatically

ChatClient.prompt()...call()
        │
        ▼
ObservationRegistry creates an Observation named "spring.ai.chat.client"
   ├── low-cardinality tags: gen_ai.operation.name, gen_ai.system (provider),
   │   gen_ai.request.model
   ├── high-cardinality tags (opt-in, content can be logged separately):
   │   prompt content, response content — DISABLED BY DEFAULT for good
   │   reason (Section 15 PII/security implications), enable deliberately
   └── nested: "spring.ai.chat.model" observation for the actual model call
        │
        ▼
Micrometer's default handlers convert this Observation into:
   - A Timer metric (spring.ai.chat.client / spring.ai.chat.model)
     exported to Prometheus/whatever MeterRegistry is configured
   - A distributed tracing Span (if micrometer-tracing + an OTel/Zipkin
     bridge is on the classpath) — correlated automatically with the
     surrounding HTTP request span if this call happened inside one

management:
  endpoints:
    web:
      exposure:
        include: health, prometheus, metrics
  tracing:
    sampling:
      probability:
        1.0 # 100% in dev; lower meaningfully in production
        # for high-QPS AI-heavy services given trace
        # storage/processing cost at volume
  observations:
    key-values:
      application: my-ai-service

13.3 Semantic Conventions — `gen_ai.*` Attributes

Spring AI follows the OpenTelemetry GenAI semantic conventions (a real, evolving standard specifically for LLM observability, not a Spring-AI-invented tag scheme) — this matters because it means your traces are interoperable with any OTel-compliant tooling that understands these conventions, not locked to Spring-AI-specific dashboards:

Attribute	Meaning
`gen_ai.system`	Provider identifier (`openai`, `anthropic`, etc.)
`gen_ai.request.model`	Requested model name
`gen_ai.response.model`	Actual model that served the response (can differ from request, e.g., provider-side routing/aliasing)
`gen_ai.usage.input_tokens`	Prompt token count
`gen_ai.usage.output_tokens`	Completion token count
`gen_ai.request.temperature`, `gen_ai.request.max_tokens`	Request parameters

13.4 Enabling Prompt/Response Content Logging (Deliberately, Not by Default)

@Bean
public ChatClientObservationConvention chatClientObservationConvention() {
    return new DefaultChatClientObservationConvention() {
        @Override
        public KeyValues getHighCardinalityKeyValues(ChatClientObservationContext context) {
            // deliberately opt-in, and pair with the redaction/PII
            // handling covered in Section 15 before enabling in any
            // environment handling real user data
            return super.getHighCardinalityKeyValues(context)
                    .and("gen_ai.prompt", context.getRequest().prompt().getContents());
        }
    };
}

spring:
  ai:
    chat:
      client:
        observations:
          include-input:
            true # off by default; PII/compliance implications —
            # this is a genuine security-relevant config
            # flag, treat it with the same care as
            # enabling verbose SQL logging with bind
            # values in a database layer

This is worth flagging with real weight: enabling full prompt/response logging in a production environment handling real user data means every conversation is now flowing into your metrics/tracing backend — apply the same data-classification and access-control rigor you’d apply to logging raw request bodies containing PII anywhere else in your stack, and coordinate with Section 15’s guidance before flipping this on outside of development/staging with synthetic data.

13.5 Custom Metrics — Cost and Business-Level Observability

Micrometer’s automatic instrumentation covers latency/count/error-rate — token-cost tracking (a first-class production concern given Section 5’s point about embedding costs and general LLM API cost sensitivity) requires a small amount of custom instrumentation on top:

@Component
public class CostTrackingAdvisor implements CallAroundAdvisor {

    private final MeterRegistry meterRegistry;
    private final Map<String, Double> costPerThousandTokens = Map.of(
            "gpt-4o", 5.00,           // illustrative — keep pricing config
            "gpt-4o-mini", 0.15,      // externalized, not hardcoded, given
            "claude-sonnet-4-6", 3.00 // how frequently provider pricing changes
    );

    @Override
    public ChatClientResponse adviseCall(ChatClientRequest request, CallAroundAdvisorChain chain) {
        ChatClientResponse response = chain.nextCall(request);

        Usage usage = response.chatResponse().getMetadata().getUsage();
        String model = response.chatResponse().getMetadata().getModel();
        double pricePerK = costPerThousandTokens.getOrDefault(model, 0.0);
        double estimatedCost = (usage.getTotalTokens() / 1000.0) * pricePerK;

        meterRegistry.counter("gen_ai.usage.cost.estimated",
                "model", model, "tenant", TenantContext.getCurrentTenantId())
                .increment(estimatedCost);

        return response;
    }

    @Override
    public int getOrder() { return Ordered.LOWEST_PRECEDENCE; }
}

This is exactly the same Advisor pattern from Section 3 — cost tracking isn’t a special Spring AI feature, it’s a custom Advisor reading ChatResponseMetadata.getUsage() and emitting a Micrometer counter, tagged by whatever business dimensions matter (tenant, feature, model) for your cost-attribution needs.

13.6 Correlation IDs Across Advisor Chains and Async Boundaries

Standard Spring/MDC correlation-ID practices apply directly, with one Spring-AI-specific wrinkle: the tool-execution loop and streaming pipelines cross thread boundaries (reactive schedulers, potentially async tool execution), where MDC context doesn’t propagate automatically unless explicitly bridged:

@Bean
public CallAroundAdvisor correlationIdAdvisor() {
    return new CallAroundAdvisor() {
        @Override
        public ChatClientResponse adviseCall(ChatClientRequest request, CallAroundAdvisorChain chain) {
            String correlationId = MDC.get("correlationId");
            try {
                // ensure correlationId survives into the tool-execution
                // loop and any nested advisor logging by putting it into
                // the advisor CONTEXT (available across the whole
                // request lifecycle) not just thread-local MDC, which
                // is NOT guaranteed to survive a reactive scheduler hop
                ChatClientRequest enriched = request.mutate()
                        .context(ctx -> ctx.put("correlationId", correlationId))
                        .build();
                return chain.nextCall(enriched);
            } finally {
                // MDC cleanup as usual
            }
        }
    };
}

For .stream() paths specifically, use Reactor’s Context propagation (contextWrite) rather than assuming MDC (thread-local) survives across the reactive pipeline’s scheduler hops — this is a standard reactive-programming correlation-ID pitfall that applies to Spring AI’s streaming path exactly as it would to any other WebFlux reactive chain in your application.

13.7 Dashboards — What Actually Matters for an AI Service

Beyond generic latency/error-rate panels (standard for any service), an AI-specific Grafana dashboard should include:

Time-to-first-token vs. total duration (Section 12) as separate panels — conflating them hides real UX degradation.
Token usage rate (input/output, per model, per tenant) — the leading indicator for cost, and for approaching context-window limits under load.
Tool-call frequency and failure rate per tool — surfaces both usage patterns and tools that are silently failing/being avoided by the model due to poor descriptions.
Fallback/routing engagement rate (Section 4) — if your fallback chain engages frequently, that’s a primary-provider health signal worth alerting on independently, not just a resilience feature quietly absorbing failures.
Structured-output parse failure rate (Section 11) — a rising trend here often precedes a broader quality regression (model version change upstream, prompt template drift) before it’s visible in generic error metrics.
Estimated cost per tenant/feature (§13.5) — business-level cost attribution, not just infrastructure cost.

13.8 Common Mistakes

Enabling include-input: true (full prompt/response logging) in production without a deliberate PII/compliance review — the highest-severity mistake in this section.
100% trace sampling in high-QPS production without considering trace-storage/processing cost at volume — appropriate for dev/staging, often not for production at scale.
No cost tracking at all, discovering budget overruns only when the provider invoice arrives rather than via a live dashboard.
Assuming MDC correlation IDs survive reactive scheduler hops in streaming paths without explicit Reactor Context propagation.
Treating fallback engagement as invisible resilience instead of an alertable primary-provider health signal.
Generic latency/error dashboards only, missing AI-specific signals (time-to-first-token, token usage rate, structured-output failure rate) that catch quality regressions before generic metrics do.

13.9 Debugging

logging:
  level:
    io.micrometer: DEBUG
    org.springframework.ai: DEBUG
management:
  observations:
    annotations:
      enabled: true

If metrics aren’t appearing in Prometheus, verify (in order): micrometer-registry- prometheus is on the classpath, /actuator/prometheus is exposed and accessible, and the ObservationRegistry bean auto-configured by Spring AI is actually the same instance Micrometer’s MeterRegistry is wired to (a manually-constructed ChatModel bean that doesn’t receive the ObservationRegistry via constructor injection — an easy mistake when overriding auto-configuration per Section 1 — silently produces zero AI-specific metrics despite the rest of the application’s metrics working fine).

13.10 Interview Questions

Why does Spring AI integrate with Micrometer rather than building a parallel observability product, and what operational benefit does that give an enterprise already running an observability stack?
What OpenTelemetry standard do Spring AI’s gen_ai.* attributes follow, and why does that matter for tooling interoperability?
Why is prompt/response content logging disabled by default, and what governance process should precede enabling it in production?
How would you implement custom cost-tracking instrumentation using the Advisor pattern, and what metadata does ChatResponseMetadata.getUsage() provide?
Why doesn’t thread-local MDC correlation-ID propagation work reliably across Spring AI’s streaming/reactive pipeline, and what’s the correct fix?
What AI-specific dashboard panels would you add beyond generic latency/error-rate metrics, and what does each catch that generic metrics miss?
Why should fallback/routing engagement rate be treated as an alertable signal rather than invisible resilience?
What’s the risk of 100% trace sampling in a high-QPS production AI service, and how would you tune it?
Explain how gen_ai.response.model can differ from gen_ai.request.model, and why that distinction is worth capturing separately.
How would you debug a scenario where the rest of your application’s metrics work but Spring AI-specific metrics are entirely absent from Prometheus?
What’s the difference between low-cardinality and high-cardinality Observation key-values, and why does Spring AI treat prompt/response content as high-cardinality, opt-in data?
Describe how you’d structure a structured-output parse-failure-rate metric and why a rising trend there can precede visible errors elsewhere.
Why is time-to-first-token tracked as a distinct metric from total request duration specifically in an observability context (recap/apply from Section 12)?
What governance/compliance concern is specific to enabling include-input: true, beyond general logging best practices?
How does Spring AI’s Observation for a ChatClient call nest with the Observation for the underlying ChatModel call, and what does that nesting give you in a trace view?
What’s the correct Reactor mechanism for propagating a correlation ID across scheduler hops in a .stream() pipeline?
How would you attribute AI API cost per-tenant in a multi-tenant SaaS application using Micrometer?
What’s the operational argument for keeping token-pricing configuration externalized rather than hardcoded in a cost-tracking advisor?
Why is a tool-call failure-rate-per-tool dashboard panel valuable beyond simple error-rate monitoring?
What’s the practical difference between metrics correlation and distributed tracing correlation for debugging a slow AI-backed request end-to-end?

13.11 Best Practices Checklist

Never enable full prompt/response content logging in production without an explicit PII/compliance review.
Tune trace sampling rate deliberately for production AI-heavy services, not left at 100%.
Implement cost-tracking instrumentation as a custom Advisor tagged by business dimensions (tenant, feature, model).
Use Reactor Context propagation, not thread-local MDC alone, for correlation IDs across streaming pipelines.
Alert on fallback/routing engagement rate as a primary-provider health signal.
Build AI-specific dashboard panels (time-to-first-token, token usage rate, tool-call failure rate, structured-output parse failure rate) beyond generic latency/error metrics.
Verify ObservationRegistry wiring explicitly whenever overriding auto-configured ChatModel beans.

13.12 Key Takeaways

Spring AI’s observability is Micrometer-native by design — this is a genuine architectural advantage for enterprises with existing observability investment, not an incidental integration.
gen_ai.* semantic conventions follow an actual OTel standard, making traces interoperable beyond Spring-AI-specific tooling.
Prompt/response content logging is opt-in and carries real PII/compliance weight — treat it with the same rigor as any sensitive-data logging decision.
Cost tracking, correlation IDs, and AI-specific dashboards all build on the same Advisor and Observation primitives covered earlier in this series — there’s no separate “observability subsystem” to learn.
Fallback engagement and structured-output failure rate are leading indicators worth dedicated alerting, not just resilience features to take for granted.

End of Section 13. Next: Section 14 — Testing (Mock Models, Integration Tests, Contract Tests, Load Testing, Evaluation, Regression Testing).

Author: TechByteByByte Editorial Team
Reviewed by: TechByteByByte Admin
Published: August 31, 2026
Last reviewed: August 31, 2026