Section 8 — Memory

Begin with the problem

Why a chatbot seems to remember

Most model calls are stateless. The application creates the feeling of memory by storing earlier messages and sending selected history with the next request.

new message + stored history → prompt → model → save new turn

What you will learn

• Distinguish ChatMemory from ChatMemoryRepository.
• Explain windowing and storage choices.
• Keep conversations isolated by ID.
• Recognize privacy, cost, and scaling limits.

Current official reference: Spring AI documentation for this topic. The examples below primarily preserve the stated 1.1.x course target. Where Spring AI 2.0 differs, the text must treat that behavior as version-specific rather than universal.

(Continues from Section 7. Target: Spring AI 1.1.x / Spring Boot 3.5.x.)

8.1 Why Memory Is a Storage-Pluggable SPI, Not a Framework Feature

Beginner primer: recall the glossary’s statelessness section — every LLM API call is independent; the provider remembers nothing about your previous calls. “Memory,” from the model’s point of view, doesn’t exist at all. Everything in this section is about your application storing prior conversation turns and re-sending them as part of the prompt on every new call, so the conversation feels continuous to the user even though, mechanically, it’s a series of unrelated requests each carrying the growing history along with it.

Spring AI’s contribution is standardizing this as a clean SPI (ChatMemory + ChatMemoryRepository) so the strategy (window, summarization) is decoupled from the storage (in-memory, JDBC, Redis) — exactly the Repository pattern from Spring Data, applied to conversation state.

Real-world analogy — Company Meeting Minutes: ChatMemory is the policy for what gets recorded and how much history a new meeting attendee is briefed on (last 5 meetings verbatim, or a summarized brief). ChatMemoryRepository is where those minutes physically live — a shared drive folder (JDBC), a fast local cache (in-memory, single-instance only), or a company-wide searchable archive (Redis, for multi-instance deployments). You can change where minutes are stored without changing the briefing policy, and vice versa.

Analogy: The Corporate Board Meeting Minutes Secretary Imagine structuring the record-keeping policy for a corporate board:

• The Secretary (ChatMemory): Decides the policy for what gets recorded during meetings and how much history a new attendee is briefed on. The secretary has a strict instruction card: “Keep only the last 10 messages verbatim so the brief is cheap to read, but always retain the corporate system directive (SystemMessage) at the very top of the folder.”
• The Archive Cabinet (ChatMemoryRepository): Decides where the minutes physically live:
  • An in-memory notepad on the secretary’s desk (lost if the secretary leaves the room — InMemoryChatMemoryRepository).
  • A shared SQL relational filing cabinet (JDBC — secure, shared across all offices).
  • A fast Redis server memory cache (ephemeral, with automatic document shredding/expiry after 30 days).
• You can swap the filing cabinet for a Redis server without changing the secretary’s briefing rules.

📊 Visual Flowchart: MessageWindowChatMemory Eviction Pipeline

Here is how conversation turns accumulate, evict older turns, and preserve the system instructions at the top:

graph TD
    classDef sys fill:#3498db,stroke:#333,stroke-width:1px,color:#fff;
    classDef user fill:#2ecc71,stroke:#333,stroke-width:1px,color:#fff;
    classDef evict fill:#e74c3c,stroke:#333,stroke-width:1px,color:#fff;

    subgraph RepoState ["Repository Store (Full History)"]
        MsgSys["System Message:<br>'You are a support bot'"]:::sys
        Msg1["Turn 1 User:<br>'Hi'"]:::evict
        Msg2["Turn 1 Assistant:<br>'Hello'"]:::evict
        Msg3["Turn 2 User:<br>'Order status'"]:::user
        Msg4["Turn 2 Assistant:<br>'Shipped'"]:::user
    end

    RepoState --> EvictionRule{"Memory Window check:<br>maxMessages = 2"}

    EvictionRule -->|1. Keep SystemMessage| OutputList["Trimmed Prompt Messages"]
    EvictionRule -->|2. Evict Turn 1 (FIFO)| OutputList
    EvictionRule -->|3. Keep Turn 2| OutputList

    subgraph OutputList ["Trimmed Prompt Context (Sent to LLM)"]
        PromptSys["System Message:<br>'You are a support bot'"]:::sys
        Prompt3["Turn 2 User:<br>'Order status'"]:::user
        Prompt4["Turn 2 Assistant:<br>'Shipped'"]:::user
    end

---

8.2 The Interfaces

public interface ChatMemory {
    void add(String conversationId, List<Message> messages);
    List<Message> get(String conversationId);
    void clear(String conversationId);
}

public interface ChatMemoryRepository {
    List<String> findConversationIds();
    List<Message> findByConversationId(String conversationId);
    void saveAll(String conversationId, List<Message> messages);
    void deleteByConversationId(String conversationId);
}

ChatMemory is the strategy layer (windowing, eventually summarization); ChatMemoryRepository is pure storage. MessageWindowChatMemory is the built-in strategy implementation, composing any ChatMemoryRepository:

@Bean
public ChatMemory chatMemory(ChatMemoryRepository repository) {
    return MessageWindowChatMemory.builder()
            .chatMemoryRepository(repository)
            .maxMessages(20)
            .build();
}

---

8.3 Storage Implementations

Implementation	Module	Persistence	Multi-instance safe	Fit
InMemoryChatMemoryRepository	spring-ai-model-chat-memory (core)	None — lost on restart	No — each instance has its own memory, breaks with a load balancer routing a user to a different instance mid-conversation	Local dev, single-instance demos only
JdbcChatMemoryRepository	spring-ai-model-chat-memory + a JDBC driver	Full, relational	Yes	Teams already running Postgres/MySQL, wanting queryable conversation history, straightforward backup/compliance story
RedisChatMemoryRepository	spring-ai-redis-store (or dedicated memory module depending on version)	Configurable TTL-based	Yes	Low-latency chat UIs, ephemeral conversations with natural expiry (e.g., support sessions)
CassandraChatMemoryRepository	Community/vendor-maintained	Full, distributed	Yes	Very high write-throughput, globally distributed deployments

The single most important production fact in this section: InMemoryChatMemoryRepository breaks the moment you run more than one application instance behind a load balancer, because conversation state lives in that specific JVM’s heap. A user’s second message, routed to a different instance by the load balancer, finds no memory of the first. This is an extremely common “works in dev, broken in staging” bug — dev runs one instance, staging/prod runs three.

// Production JDBC setup
@Bean
public ChatMemoryRepository chatMemoryRepository(JdbcTemplate jdbcTemplate) {
    return JdbcChatMemoryRepository.builder()
            .jdbcTemplate(jdbcTemplate)
            .build();
}

-- Schema JdbcChatMemoryRepository expects (managed via Flyway/Liquibase in
-- production, not auto-created — same principle as Section 6's VectorStore
-- schema guidance)
CREATE TABLE ai_chat_memory (
    conversation_id VARCHAR(256) NOT NULL,
    content         TEXT NOT NULL,
    type            VARCHAR(32) NOT NULL,    -- USER / ASSISTANT / SYSTEM / TOOL
    timestamp       TIMESTAMP NOT NULL,
    PRIMARY KEY (conversation_id, timestamp)
);

---

8.4 Memory Window Strategy — Internals

MessageWindowChatMemory.add():

add(conversationId, newMessages)
        │
        ▼
1. repository.saveAll(conversationId, newMessages) — always persists everything
        │
        ▼
2. repository.findByConversationId(conversationId) — read back full history
        │
        ▼
3. If total message count > maxMessages:
   evict oldest messages (FIFO), but PRESERVE any leading SystemMessage
   (window eviction skips system messages so persona/instructions
   don't get silently dropped as conversation length grows — a detail
   worth verifying against your exact version's eviction logic, since
   this system-message-preservation behavior is the kind of nuance
   that differs across releases)
        │
        ▼
4. Trimmed list becomes what get() returns on the next call

Important nuance: the repository stores the full history (nothing is deleted from storage by the window strategy itself — clear() is a separate explicit operation); the window only limits what gets returned and re-injected into the prompt. This means switching maxMessages doesn’t lose historical data, and you can build a separate “full conversation export” feature reading directly from ChatMemoryRepository even while the live chat only sees the windowed view.

---

8.5 Summarization-Based Memory (Production Pattern)

Spring AI’s built-in strategy is window-based, not automatically summarizing. For long-running conversations where a 20-message window still exceeds a useful context budget, or where you want unbounded conversation length without unbounded token cost, implement summarization as a custom ChatMemory decorator:

public class SummarizingChatMemory implements ChatMemory {

    private final ChatMemoryRepository repository;
    private final ChatClient summarizerClient;
    private final int summarizeAfterMessages = 30;
    private final int keepRecentVerbatim = 10;

    @Override
    public void add(String conversationId, List<Message> messages) {
        repository.saveAll(conversationId, messages);

        List<Message> all = repository.findByConversationId(conversationId);
        if (all.size() > summarizeAfterMessages) {
            List<Message> toSummarize = all.subList(0, all.size() - keepRecentVerbatim);
            List<Message> toKeep = all.subList(all.size() - keepRecentVerbatim, all.size());

            String summary = summarizerClient.prompt()
                    .system("Summarize this conversation history concisely, "
                            + "preserving names, decisions, and open questions.")
                    .user(formatMessagesForSummary(toSummarize))
                    .call()
                    .content();

            repository.deleteByConversationId(conversationId);
            List<Message> compacted = new ArrayList<>();
            compacted.add(new SystemMessage("Prior conversation summary: " + summary));
            compacted.addAll(toKeep);
            repository.saveAll(conversationId, compacted);
        }
    }

    @Override
    public List<Message> get(String conversationId) {
        return repository.findByConversationId(conversationId);
    }

    @Override
    public void clear(String conversationId) {
        repository.deleteByConversationId(conversationId);
    }

    private String formatMessagesForSummary(List<Message> messages) {
        return messages.stream()
                .map(m -> m.getMessageType() + ": " + m.getText())
                .collect(Collectors.joining("\n"));
    }
}

Production trade-off to weigh explicitly: summarization costs an extra LLM call (latency + cost) every time the threshold is crossed, and compresses information lossily — a user referencing a very specific detail from 40 messages ago may find it summarized away. Window-only memory is simpler and lossless-within-window but bounds effective conversation “attention span” hard at the window edge. Many production systems use a hybrid: window-based for typical sessions, with summarization triggered only for really long-running conversations (support escalations, long research sessions) rather than universally.

---

8.6 Memory Strategy Comparison

Strategy	Token cost per turn	Information loss	Latency overhead	Best for
Full history (no limit)	Grows unbounded — dangerous	None	None extra	Never appropriate in production beyond very short-lived sessions
Sliding window	Bounded, predictable	Old messages dropped entirely	None extra	Most chat applications — the sane default
Summarization	Bounded, predictable, slightly higher baseline (summary text)	Lossy compression of old content	Extra LLM call on threshold crossing	Long-running sessions where distant context still matters somewhat
Hybrid (window + periodic summarization)	Bounded	Partial — summarized old content, verbatim recent	Occasional extra call	Production systems wanting both cost control and reasonable long-range coherence

---

8.7 Production Design Considerations

1. Conversation ID strategy: derive from an authenticated session/user identity, never from client-supplied unvalidated input — a client-controlled conversationId is a direct cross-user memory access vector if your storage layer doesn’t independently verify ownership.
2. Retention/compliance: conversation memory often contains PII; define a retention policy (TTL on Redis, scheduled deletion job for JDBC) rather than storing indefinitely by default — this intersects directly with Section 15’s Security/PII coverage.
3. Multi-instance consistency: always use JdbcChatMemoryRepository, RedisChatMemoryRepository, or equivalent persistent/shared storage in any deployment with more than one instance — this is not optional, per §8.3.
4. Read/write cost at scale: MessageWindowChatMemory.add() as shown reads back the entire stored history on every write to compute the window — at high message-volume conversations this becomes an avoidable read amplification; production-hardened implementations often cap the read query itself (findRecentByConversationId(id, limit)) rather than reading unbounded history just to discard most of it.

---

8.8 Common Mistakes

1. Using InMemoryChatMemoryRepository in a multi-instance production deployment — the single most common memory-related production bug.
2. Trusting a client-supplied conversationId without validating it belongs to the authenticated caller.
3. No retention policy — conversation memory silently accumulates PII indefinitely.
4. Assuming window eviction deletes from storage — it doesn’t; clear() is the explicit deletion operation, and forgetting this leads to unexpectedly large storage tables in JDBC-backed deployments.
5. Summarizing on every single turn instead of threshold-triggered — unnecessary added latency/cost for conversations that never get long enough to need it.
6. Not indexing conversation_id in the JDBC schema — degrades read performance as conversation-history tables grow.

---

8.9 Debugging

logging:
  level:
    org.springframework.ai.chat.memory: DEBUG

For “the bot forgot what we discussed” reports, check three things in order: (1) is the same conversationId being sent on every request from the client (session/cookie issue is extremely common), (2) is the deployment multi-instance with InMemoryChatMemoryRepository still configured (the #1 cause per §8.8), (3) has the window/summarization threshold actually evicted the relevant message.

---

8.10 Interview Questions

1. Why does Spring AI separate ChatMemory (strategy) from ChatMemoryRepository (storage) as distinct interfaces?
2. What specifically breaks when InMemoryChatMemoryRepository is used behind a load balancer with multiple instances?
3. Does MessageWindowChatMemory delete evicted messages from storage? Explain the distinction between window eviction and clear().
4. Design a summarization-based memory strategy and explain the cost/latency trade-off versus pure windowing.
5. Why must conversationId be derived from authenticated identity rather than trusted from client input?
6. What compliance/retention concern is specific to chat memory that doesn’t apply to, say, embedding caches?
7. How would you detect, from a bug report of “the bot forgot our conversation,” whether the root cause is session/cookie handling, multi-instance memory isolation, or window eviction?
8. What’s the read-amplification issue with a naive MessageWindowChatMemory.add() implementation at high message volume, and how would you address it?
9. Why does window eviction typically preserve a leading SystemMessage rather than evicting it along with old user/assistant turns?
10. Compare JdbcChatMemoryRepository and RedisChatMemoryRepository in terms of natural fit for conversation TTL/expiry use cases.
11. What schema would you design for JdbcChatMemoryRepository in production, and why manage it via Flyway/Liquibase rather than auto-creation?
12. How would you implement a hybrid memory strategy that uses windowing for short conversations and summarization only for long-running ones?
13. What’s the architectural parallel between ChatMemoryRepository and Spring Data’s repository pattern?
14. Why might a client-supplied conversationId used without server-side ownership validation constitute a security vulnerability, not just a bug?
15. What token-cost trade-off exists between sliding-window and summarization-based memory strategies?
16. How would you test that your ChatMemory implementation correctly isolates conversations across concurrent users?
17. What’s lost when summarizing 30 messages down to a compact summary, and how would you mitigate that loss for critical details?
18. Why is InMemoryChatMemoryRepository acceptable for local development but never for staging/production beyond single-instance demos?
19. Describe how you’d implement an “export my full conversation history” feature given that the live chat only ever sees a windowed view.
20. What indexing strategy would you apply to a conversation_id column at scale, and why does its absence degrade over time rather than immediately?

---

8.11 Best Practices Checklist

• Never deploy InMemoryChatMemoryRepository in any multi-instance environment.
• Derive conversationId from authenticated session identity, never raw client input.
• Define and enforce a retention/TTL policy for conversation memory given its PII exposure.
• Manage JDBC memory schema via migration tooling, not auto-creation, in production.
• Index conversation_id (and timestamp for range queries) explicitly.
• Choose summarization only when window-only memory demonstrably degrades UX for your actual conversation lengths — don’t add the complexity preemptively.

---

8.12 Key Takeaways

• ChatMemory/ChatMemoryRepository is a clean strategy/storage split, mirroring Spring Data’s repository pattern applied to conversation state.
• InMemoryChatMemoryRepository is a dev-only convenience — the most common production memory bug is running it multi-instance.
• Window eviction limits what’s returned, not what’s stored — clear() is the only real deletion operation.
• Summarization is a deliberate cost/latency trade-off for long-running conversations, not a universal upgrade over windowing.
• Conversation memory is a PII/compliance surface as much as a UX feature — treat retention policy as a first-class design decision, not an afterthought.

---

End of Section 8. Next: Section 9 — Tool Calling (Function Calling, Java Methods, Bean Discovery, Arguments, JSON Schema, Error Handling, Retry, Tool Execution, Security).