Anatomy of an AI Agent

An agent may look like one smart box, but it is a small team of parts. The model decides; tools act; state records progress; guardrails limit danger; and the environment returns results.

Goal
  ↓
Agent brain ↔ Current state
  ↓              ↑
Tool → Real world result
  ↓
Safety check → Continue, ask a human, or stop

What You Will Learn

• The job of the goal, model, instructions, tools, environment, state, and memory.
• How an observation differs from an action and why both are needed.
• How planning, decision-making, feedback, and termination connect inside the loop.
• Which parts are produced by the language model and which are enforced by application code.
• How to trace one request through the complete agent architecture.

See the Parts in a Real Agent Platform

The Gemini managed-agent documentation exposes many of the same parts: a model, system instructions, tools such as search and code execution, a sandboxed environment, files, and configurable data sources. Your application still supplies the goal and decides which permissions and results are acceptable. (Google, Building Managed Agents)

Vendor names may change, but these architectural jobs remain recognizable. Learning the jobs first makes it easier to understand any agent framework later.

By now you know why agents exist and what the loop looks like at a high level: goal, observe, reason, decide, act, observe result, repeat. That’s the right mental model, but it’s still a silhouette. If someone handed you a whiteboard right now and asked you to design an agent for a real task, “goal, observe, reason, decide, act” isn’t quite enough to start building from. You’d have real questions: where does the goal live in the system? What, precisely, is being “observed”? What decides when the loop is finished?

This module is where we open the silhouette up and name every real part inside it — the same way you’d study anatomy before you’d trust yourself to operate. We’re not going deep on the mechanics of any one part yet; tools, planning, reasoning, and memory each get their own full module later. What this module gives you is the complete map: every component, what it’s for, and how it connects to the others — so that when we do go deep on each piece individually, you already know exactly where it fits.

The full picture

Here’s the complete architecture, all components together:

                    ┌──────────────┐
                    │     Goal     │
                    └──────┬───────┘
                           ↓
                    ┌──────────────┐
                    │    Agent     │
                    │              │
                    │ Reason       │
                    │ Decide       │
                    │ Plan         │
                    └──────┬───────┘
                           ↓
              ┌────────────┴────────────┐
              ↓                         ↓
         Tools / APIs              Knowledge
              ↓                         ↓
              └────────────┬────────────┘
                           ↓
                      Environment
                           ↓
                        Feedback
                           ↓
                         Agent

Every box here is doing real, distinct work, and confusing two of them for each other is one of the most common sources of “why is my agent behaving strangely” confusion once you start building. So let’s take them one at a time, slowly, and we’ll ground every single one in a real, running example: the Customer Support Resolution Agent from Module 1, whose job is to resolve a ticket like “my payment failed twice this week and I still haven’t been refunded” — not just respond to it.

Goal: the thing that defines success

The goal is where everything starts, and it’s worth being precise about what makes a goal different from an ordinary prompt. A prompt to a plain LLM application says “answer this.” A goal says “achieve this outcome” — and critically, it implies a condition under which the agent can say done.

For our support agent, the goal isn’t “respond to the customer’s message.” It’s something closer to “determine why this customer’s payment situation is unresolved, and take whatever action correctly resolves it — a refund, a retry, a card-update notice, or an escalation — then confirm the customer has been informed.” Notice how much more specific that is than “answer the message,” and notice that it implies several different possible correct endings, depending on what gets discovered. That’s the property that makes this a job for an agent rather than a single generation, exactly as we established in Module 1.

A goal that’s vague or ambiguous is a real engineering liability, not just an inconvenience — it directly determines what “termination” (the last component in this list) is even checking against. We’ll come back to that.

The agent core: the model plus its instructions

The box labeled “Agent” in the diagram — reasoning, deciding, planning — is powered by an LLM, but the model alone isn’t the whole story. What makes it this specific agent, doing this specific job, rather than a generic assistant, is a set of instructions — usually a system prompt — that establishes its role, its boundaries, and what tools are available to it.

Think of the instructions as the job description you’d hand a new employee: “you are a support resolution agent. Your job is to determine the cause of payment issues and resolve them using the available tools. You have access to the following actions: [tool list]. If the account shows any sign of fraud or dispute, do not act — escalate to a human immediately.”

This is where the boundaries of the agent’s authority get established, and it’s worth flagging now, even though we’ll go much deeper on it in Module 11, that instructions are a starting point for behavior, not a guarantee of it. A well-written instruction reduces the chance of the agent doing something wrong. It doesn’t structurally prevent it the way a, code-enforced permission boundary does. Hold onto that distinction — it matters a great deal later.

Tools and knowledge: the two ways an agent reaches outward

The diagram splits into two paths beneath the agent core, and they’re different in kind, even though both eventually feed into “the environment.”

Tools are how the agent takes action — get_payment_history(), retry_payment(), send_email(). Each one does something in a real system and returns a real result. Knowledge is what the agent retrieves to inform its reasoning without necessarily changing anything — company refund policy, prior documentation, historical patterns. You’ve already studied this half deeply as RAG, and the connection is direct: retrieval, in an agent context, is simply one more kind of lookup the agent can perform mid-task, the same way it might call check_payment_gateway().

The agent doesn’t fundamentally treat “search the knowledge base” any differently from “call an API” — both are things it can decide to do, both return a result it observes, and both feed into its next decision.

The distinction worth keeping in your head, though: tools generally do something — retry the payment, send the email, update the CRM — while knowledge retrieval generally informs something without changing state anywhere. That distinction becomes important in Module 11, because actions that change real systems are exactly where permission boundaries and approval gates matter most; retrieval is comparatively low-risk. We’ll go properly deep on tools themselves in Module 5 — for now, just register that this is where the agent’s capability to do things lives, distinct from where its knowledge lives.

Environment: the real system the agent is operating in

The environment is everything outside the agent — the CRM, the payment gateway, the customer database, the email system. It’s worth naming explicitly because it’s easy to think of “the agent” as the whole system, when really the agent is a relatively small reasoning core sitting next to a much larger real environment it’s interacting with through tools.

This matters because the environment has properties the agent’s reasoning doesn’t control and can’t assume away. APIs can be slow. A database can return stale data. A payment gateway can have a outage. The environment is where real-world unpredictability lives, and a big part of building a reliable agent — which we’ll cover properly in Module 10 — is designing for the fact that the environment will not always behave the way the agent expects.

Observation and feedback: how the environment talks back

When the agent calls get_payment_history(), whatever comes back — two declined charges, both marked insufficient_funds — is an observation. The feedback loop is the mechanism that takes that observation and routes it back into the agent’s next reasoning step, rather than the interaction simply ending.

This is worth dwelling on, because it’s the literal mechanical difference between an agent and the single-shot tool-calling pattern from Module 2. In tool calling, a result comes back and the model folds it into one final answer — the loop diagram doesn’t have an arrow going back into “reason.” In an agent, that arrow exists and it’s load-bearing: the observation changes what happens next.

In our example, the payment-history observation is what causes the agent to decide to check the payment gateway next, rather than assuming the customer’s stated cause is correct. Without a real feedback path, that specific, correct decision couldn’t happen — the agent would have no way to notice the mismatch between what the customer claimed and what the data showed.

Reasoning: what happens between observation and decision

Reasoning is the model’s own generated analysis of the current situation, produced before it commits to an action — something like “the payment history shows two declines for insufficient funds, which doesn’t match the customer’s claim of a gateway issue; I should verify the gateway’s status before concluding anything.”

This text is useful for two reasons: it often improves the quality of the eventual decision (a model that’s asked to reason through a situation before acting tends to make better decisions than one asked to jump straight to an action), and it gives you, the engineer, real visibility into why the agent did what it did — which becomes essential the moment something goes wrong and you need to debug it.

It’s worth being precise here, the same way Module 1 was precise about prompts not creating agents: this reasoning text is the model generating language, not a window into some separate internal thought process happening elsewhere. It’s useful, and it’s also just another generation, produced the same way any other model output is. We’ll unpack this distinction — and the difference between the model’s raw reasoning capability and the orchestration logic sitting around it — properly in Module 6.

Planning: deciding the shape of the work before doing it

Planning is a related but distinct capability: producing a rough decomposition of the task before diving into individual actions, rather than discovering the task’s structure purely reactively, one step at a time. For our support agent, a plan might look like “first confirm the account is in good standing, then check payment history, then check the gateway if there’s a mismatch, then decide on an action based on what’s found.”

Notice this plan doesn’t fully specify the gateway check — it’s conditional, “if there’s a mismatch” — which is exactly right, because the actual need for that step only becomes clear once the payment history is observed.

Planning isn’t always necessary. For a task simple enough that each step naturally suggests the next one, an explicit upfront plan can be unnecessary overhead. We’ll go deep on exactly when planning earns its keep in Module 6.

State: the agent’s evolving, in-progress understanding

State is what the agent has learned and concluded so far, in this specific run — and it accumulates as the loop proceeds. After checking the customer and payment history, our agent’s state might look something like:

customer_status: active, good standing
payment_history: 2 declines, reason "insufficient_funds"
gateway_status: not yet checked
conclusion: pending

Notice this is different from the agent’s instructions, which stay fixed for the entire run, and different again from its reasoning, which is generated fresh at each step. State is the accumulated, structured record of progress — it’s what lets a later step in the loop “know” what an earlier step already discovered, without needing to re-derive it. We’ll dedicate an entire module (Module 7) to state specifically, because confusing it with memory — the next component — is one of the most common points of confusion in this whole subject.

Memory: what survives beyond this one task

Memory is information retained across separate tasks, not just within this one. If this same customer contacts support again next month, and the agent recalls “this customer previously had a card-related payment issue and prefers email over phone,” that’s memory — deliberately persisted, unlike state, which gets discarded the moment this particular ticket is closed.

The relationship to what you’ve already studied is direct: memory is very often implemented using the same retrieval mechanism as RAG — embedding past interactions or facts, and pulling back the relevant ones when a new, related situation comes up. We’re not going to re-derive vector retrieval here since you already know it; Module 7 will focus specifically on how memory fits into an agent’s decision- making, not on the retrieval mechanics underneath it.

Action: where a decision becomes a real effect

Action is the literal execution of a tool call — the moment retry_payment(customer_id="C-4471", amount=49. 99) runs against a real payment system, rather than remaining a decision the model merely generated. This is worth naming as its own distinct component because there’s real engineering work sitting between “the model decided to call this tool” and “the tool ran” — argument validation, permission checks, error handling — none of which is the model’s job. The model decides.

Your application code is what executes, and it’s responsible for making sure that execution is safe. We’ll go deep on exactly what that responsibility involves in Module 5.

Termination: knowing when the job is done

This is the component most beginners underweight, and it deserves real attention, because it’s one of the highest-leverage places to get an agent wrong.

Termination is what determines the loop should stop — and there are meaningfully different ways this can happen, not just one:

• Goal satisfied. The agent’s own reasoning concludes the goal, as originally defined, has been met. For our example: the cause was found, the appropriate action was taken, and the customer was notified.
• stuck. The agent has tried what it reasonably can and cannot proceed further — perhaps a required tool is unavailable, or the situation is ambiguous enough that continuing without more information would be guessing. The correct behavior here is an honest stop with an explanation, not a fabricated success.
• A hard limit is hit. A maximum number of iterations, a time budget, or a cost cap — enforced by your code, not the model’s own judgment — is reached, and the loop is forced to stop regardless of whether the model believes it’s making progress. We’ll cover exactly why this hard limit is non-negotiable in Module 10, but it’s worth planting the idea here: relying purely on the model to decide when it’s “done” is risky, because a model can be confidently, fluently wrong about its own progress. A real engineering system needs a backstop that doesn’t depend on the model agreeing with itself.

Go back to the goal you defined at the very start of this module, and notice that termination is really just checking the current state against that original goal. This is exactly why a vague, poorly-defined goal causes real downstream problems — if “done” was never precisely defined, the system has no reliable way to recognize it.

How real systems structure this

It’s worth knowing that this isn’t an abstraction unique to this course — every serious agent framework and provider API you’ll encounter structures things in this same shape, even when the specific names differ. OpenAI’s Assistants API concept separates “instructions” from “tools” from a persistent “thread” that holds conversation and tool-call history — instructions and tools mapping directly onto what we’ve called instructions and tools here, and the thread doing a job close to what we’ve called state.

Anthropic’s tool use documentation describes essentially the same split: a system prompt establishing the agent’s role and boundaries, a defined set of tool schemas, and a loop your own code runs, feeding each tool result back into the next model call. The specific vocabulary varies by provider and framework — you’ll meet several of these names properly once this course reaches LangChain and LangGraph — but the underlying anatomy, the actual components doing actual work, is consistently this same set of pieces.

Learning them here, provider-agnostic, means you’ll recognize them immediately no matter which specific framework you eventually build with.

Putting the whole map together

Let’s walk the support agent through every component in one pass, so you can see the full anatomy operating as a single system rather than a list of definitions.

The goal is set: resolve the customer’s payment issue. The agent core, guided by its instructions, begins reasoning about what it needs to know first. It decides to call the get_customer tool — an action — which reaches into the environment (the customer database) and returns an observation: account active, good standing. That observation, via feedback, updates the agent’s state and informs the next round of reasoning.

The agent decides to check payment history next, observes two declines for insufficient funds, and — reasoning again — notices this doesn’t match the customer’s stated cause. It optionally checks knowledge (the refund policy) to confirm what response is appropriate here, decides to verify the gateway directly rather than assume, observes zero reported outages, and reaches a grounded conclusion. It takes a final action — sending the card-update notice — and termination triggers: the goal, as originally defined, has been satisfied.

If this customer contacts support again next month, whatever’s worth remembering from this interaction can be written to memory, available for a future task — while everything in this task’s state is simply discarded, its job now finished.

That’s the whole anatomy, working together, on one real example.

When to Add Each Part

Add only the components the task needs. A read-only research agent may need search tools and temporary state but no long-term user memory. A refund agent needs authentication, policy checks, approval boundaries, idempotent actions, and an audit trail.

Common Misconception

Incorrect idea: The LLM is the complete agent.

Why it is incorrect: The LLM proposes language and decisions. Application code owns tool execution, state storage, permissions, validation, monitoring, and enforced stopping conditions.

Key Takeaways

• An agent’s architecture has distinct components — goal, instructions, tools, knowledge, environment, observation, feedback, reasoning, planning, state, memory, action, and termination — and confusing two of them for each other is a common, real source of confusing agent behavior once you start building.
• A goal defines not just what the agent should do, but implicitly what “done” means — and termination logic depends directly on that definition being precise enough to check against.
• Instructions establish an agent’s role and boundaries, but they’re a starting point for behavior, not a structural guarantee of it — real permission enforcement has to live in code, not just in the prompt.
• Tools take real action and change real systems; knowledge retrieval informs reasoning without necessarily changing anything — a distinction that matters directly for where security and approval boundaries need to be strictest.
• State is this task’s accumulated, in-progress understanding, and gets discarded once the task ends. Memory is what’s deliberately retained across separate tasks. These are different components with different lifetimes.
• Termination isn’t a single event — an agent can stop because the goal was satisfied, because it’s honestly stuck and says so, or because a hard, code-enforced limit was reached regardless of what the model itself believes about its own progress.
• This exact anatomy — under varying names — shows up consistently across real provider APIs and agent frameworks, which means learning the components provider-agnostically now will transfer directly once you reach specific tools later in this learning path.

Think Like an AI Engineer

• Write out a precise goal statement for an agent that manages a team’s meeting scheduling. What would “termination” check for in that case? Is there more than one legitimate way the task could end successfully?

• Go back through the support agent walkthrough at the end of this module and identify one point where the agent’s state was used in a decision. Now identify a plausible point in a longer-running version of this same agent where memory — not state — would change its behavior on a future, separate ticket.

• Suppose an agent’s instructions say “never issue a refund over $500 without approval,” but there’s no code-level check enforcing that limit — it exists purely as a sentence in the system prompt. What could realistically go wrong? What would you need to build to make that boundary reliable rather than merely requested?

• Think of a real, multi-step task from your own work. Try to name its goal, its available tools, and what its state would look like halfway through. If you struggle to name any one of these three clearly, what does that tell you about whether the task is well-suited to an agent yet?

Module 4 takes this anatomy and puts it in motion. We’ll walk through the agent loop in full, real depth — a complete run of the support agent from start to finish, plus what happens when things go wrong along the way: a failed API call, a wrong diagnosis, missing information, a permission denied — and exactly how the agent decides whether to retry, change strategy, ask for help, escalate, or stop.