AI & LLM Concepts Glossary — Read This First

Begin with the problem

AI concepts before Spring AI

An AI application can look magical until we separate the model, prompt, tokens, memory, tools, and retrieval. This glossary gives each piece a simple name before the framework combines them.

User request → prompt → model → response

What you will learn

• Explain the basic parts of an AI request.
• Distinguish a model from the Java application around it.
• Recognize tokens, embeddings, memory, tools, and RAG.
• Use the glossary while reading later modules.

Current official reference: Spring AI documentation for this topic. The examples below primarily preserve the stated 1.1.x course target. Where Spring AI 2.0 differs, the text must treat that behavior as version-specific rather than universal.

Why this file exists: The original notes (Sections 1–17) are written for a reader who already knows AI/LLM fundamentals — they repeatedly say things like “you already know embeddings conceptually” or “you already understand tool calling conceptually.” If you’re a Spring Boot developer who has not worked with LLMs before, that assumption will leave real gaps. This file fills them. Read it once, then treat it as a reference — every later section links back to the relevant term here the first time it uses it.

If you already know these terms, skim the table of contents and skip straight to Section 1.

Analogy: The Restaurant Dining Loop & The Amnesiac Chef Imagine sitting down at a fine dining restaurant with a very peculiar setup:

• The Chef (The LLM): The kitchen chef is a master artist who cooks magnificent dishes, but has complete, clinical amnesia. The second a dish leaves the kitchen window, the chef forgets who you are, what you ordered, and that they ever cooked a meal.
• The Waiter (Spring AI Application Layer):
  • Turn 1: You ask: “Can I get the steak?” The waiter writes this down (Prompt) and gives it to the chef. The chef cooks a steak (Completion).
  • Turn 2: You say: “Make it medium rare.” If the waiter only tells the chef “Make it medium rare”, the chef has no idea what “it” is. They don’t know you ordered a steak 5 minutes ago.
  • The Memory Fix: The waiter must carry a pad recording your entire conversation history. To get the medium-rare steak, the waiter goes back to the kitchen and tells the chef: “Customer ordered steak. Now they say make it medium rare.” Only by re-submitting the entire transcript does the chef know how to proceed.
• Every call to an LLM is a fresh start in the kitchen. If you don’t re-send the history, the chef is blind.

📊 Visual Flowchart: Concept Flow of an LLM Client Pipeline

Here is how prompts, tokens, and context windows constrain stateless API calls:

graph TD
    UserQuery["User Query:<br>'Make it medium rare'"] --> MemoryAdvisor["1. Memory Advisor<br>(Append order history: 'Customer ordered steak')"]
    MemoryAdvisor --> Tokenizer["2. Tokenizer<br>(Convert text to token ID budget)"]

    Tokenizer --> ContextCheck{"3. Context Window Check<br>(Tokens < maxTokens?)"}

    ContextCheck -->|Exceeds Budget| Trim["Trim oldest turns"]
    ContextCheck -->|Fits| ApiCall["4. HTTP REST API Client<br>(Stateless provider request)"]

    ApiCall --> LLM["5. Remote LLM Provider<br>(Predict next token completion)"]
    LLM --> Response["6. Return Response String"]

---

What is an LLM, actually?

A Large Language Model (LLM) — GPT-4, Claude, Gemini, Llama, Mistral, and similar — is a program that has been trained on enormous amounts of text and learned to predict, one small chunk at a time, what text is likely to come next given everything before it. You send it text (a prompt); it sends back text (a completion or response), generated one chunk at a time until it decides it’s done, or you tell it to stop.

That’s really the whole mechanical idea. Everything else in this glossary — chat, tools, RAG, agents — is application-layer machinery built on top of that one core capability: “given some text, predict more text.” Keeping this in mind demystifies a lot of what otherwise looks like magic. An LLM doesn’t “look things up” or “know facts” the way a database does — it generates plausible-sounding continuations based on patterns learned during training. This is exactly why hallucination (below) is a real, structural risk, not a rare bug.

Spring AI does not run the model itself (no weights, no GPU inference inside your JVM). Every ChatModel implementation is an HTTP client wrapper that calls a model provider’s API (OpenAI, Anthropic, a local Ollama daemon, etc.) and shapes the request/response.

Tokens

LLMs don’t process text character-by-character or word-by-word — they process tokens, which are sub-word chunks. “Unbelievable” might tokenize as Un + believ + able, for instance. Roughly: 1 token ≈ 4 characters of English text, or about 0.75 words.

Why you should care as an engineer, not a linguist: every provider bills by token count (input tokens + output tokens), and every model has a maximum context window (below) measured in tokens, not characters. When these notes mention maxTokens in ChatOptions, or discuss cost/performance trade-offs, tokens are the actual unit being counted.

Context window

The context window is the maximum number of tokens a model can “see” at once — the system prompt, the conversation history, any retrieved documents, and the model’s own response all have to fit inside this one budget. A model with a 128K-token context window forgets nothing within that window, but has zero awareness of anything outside it. This is precisely why Memory (Section 8) and RAG (Section 7) exist: they’re both, at their core, strategies for deciding what to fit into a limited context window on every single call, since the model itself remembers nothing between calls (see statelessness below).

Prompt, system/user/assistant roles

A prompt is the input you send the model. Modern chat-style LLM APIs structure a prompt as a list of messages, each tagged with a role:

• System message — instructions about how the model should behave overall (“You are a helpful customer support assistant. Only answer questions about our products.”). Usually set once, not per-turn.
• User message — what the actual human (or your application, on their behalf) is asking.
• Assistant message — the model’s own prior responses, included in the message list so a multi-turn conversation has continuity.

This SystemMessage / UserMessage / AssistantMessage split is exactly what Spring AI’s Message interface hierarchy models directly (Section 2).

Statelessness — the single most important fact about how LLM APIs work

Every single call to an LLM API is independent. The model has no memory of a previous call unless you re-send the entire relevant conversation history as part of the new prompt, every single time. There is no session on the provider’s side. This single fact is the reason:

• Memory (Section 8) exists — an application-layer concern for storing and re-sending prior turns.
• Cost and latency both grow with conversation length, since you’re re-sending everything, every turn.
• A “long conversation” isn’t actually one ongoing exchange under the hood — it’s many independent calls, each one re-fed the growing history.

Temperature and other sampling parameters

Temperature controls how “random” versus “predictable” the model’s next-token choices are. Low temperature (near 0) makes the model consistently pick its most likely next token — deterministic, focused, repeatable output, good for factual/structured tasks. Higher temperature (approaching 1 or above, depending on provider) makes the model more willing to pick less-likely tokens — more varied, creative, sometimes less reliable output. This is one of the settings exposed through Spring AI’s ChatOptions.

Hallucination

Hallucination is when a model generates text that is fluent and confident-sounding but factually wrong or entirely fabricated — a made-up citation, a nonexistent API method, an incorrect statistic. This isn’t a bug that gets patched out; it’s a structural consequence of how LLMs work (predicting plausible text, not looking up verified facts). It’s the central reason RAG (below) matters for any application where factual accuracy is important: grounding the model’s response in real, retrieved documents substantially reduces (but does not eliminate) hallucination.

Embeddings and vectors

An embedding is a numeric representation of a piece of text (a word, sentence, or document) as a list of floating-point numbers — a vector — typically hundreds to thousands of numbers long. The property that makes embeddings useful: texts with similar meaning produce vectors that are numerically close together, and texts with different meaning produce vectors that are far apart, in this high-dimensional numeric space.

This is what makes semantic search possible — finding documents by meaning rather than exact keyword match. “How do I get a refund?” and “What’s your return policy?” might share zero words in common, but their embedding vectors would land close together, because they mean similar things. Spring AI’s EmbeddingModel interface (Section 5) produces these vectors; VectorStore (Section 6) stores and searches them.

Cosine similarity is the most common way to measure how “close” two vectors are — mathematically, the cosine of the angle between them, ranging from -1 (opposite meaning) to 1 (identical meaning). When these notes mention a similarityThreshold on a vector search, this is the metric usually being thresholded.

RAG (Retrieval-Augmented Generation)

RAG is a pattern for reducing hallucination and giving a model access to information it wasn’t trained on (your company’s internal documents, for instance): before calling the LLM, you first retrieve relevant documents from a knowledge base (typically via embedding-based similarity search against a vector store), then augment the prompt by inserting those documents as context, then generate the response — so the model is answering using the retrieved text, not purely from what it memorized during training. Section 7 covers Spring AI’s RAG implementation in depth.

Tool calling (a.k.a. function calling)

LLMs can’t directly execute code, query a live database, or call an external API — they can only generate text. Tool calling is a pattern where you describe available functions (name, description, expected parameters) to the model as part of the prompt. If the model decides a function would help answer the request, instead of replying with a normal text answer, it replies with a structured request to call that function with specific arguments. Your application code actually executes the function, and feeds the result back to the model, which then generates its final, informed response. Section 9 covers how Spring AI automates this using plain Java methods and the @Tool annotation.

Streaming

Rather than waiting for a model to generate its entire response before showing anything to the user, streaming sends back partial output as it’s generated — the token-by- token (or small-chunk-by-chunk) typing effect you see in ChatGPT-style interfaces. This matters for perceived responsiveness on longer responses. Section 12 covers Spring AI’s StreamingChatModel and reactive Flux<ChatResponse> support.

Agents

An agent, loosely, is an LLM-driven system that can autonomously decide which tools to call, in what order, potentially looping through multiple tool calls, before producing a final answer — as opposed to a single request/response exchange. The term gets used loosely across the industry; these notes use it precisely where relevant (Sections 9, 10, 17) and are explicit about what Spring AI does and doesn’t provide natively for multi-step agentic orchestration.

MCP (Model Context Protocol)

MCP is an open, provider-agnostic protocol (originated by Anthropic, now widely adopted) for exposing tools, resources, and prompts to LLM applications in a standardized way — think of it as analogous to how JDBC standardized talking to different databases, but for connecting an AI application to external tools and data sources. Section 10 covers Spring AI’s MCP client and server support.

---

Quick-reference map: where each concept is covered in depth

Concept	Deep-dive section
Prompts, messages, roles	Section 2 — Prompt API
Orchestration, request pipeline	Section 3 — ChatClient
Providers (OpenAI, Anthropic, Ollama, etc.)	Section 4 — ChatModel Providers
Embeddings, vectors	Section 5 — Embeddings
Vector storage & similarity search	Section 6 — Vector Store
RAG	Section 7 — RAG
Conversation memory	Section 8 — Memory
Tool/function calling	Section 9 — Tool Calling
MCP	Section 10 — MCP Integration
Structured output (typed responses)	Section 11 — Structured Output
Streaming	Section 12 — Streaming
Observability	Section 13 — Observability
Testing AI applications	Section 14 — Testing
Security (prompt injection, etc.)	Section 15 — Security
Performance tuning	Section 16 — Performance
Enterprise architecture patterns	Section 17 — Enterprise Architecture

Worked applications (full projects applying these concepts): Build a Chatbot with ChatClient · Build a RAG Assistant · Build a Tool-Calling Support Agent