What is AI agent architecture?

AI agent architecture is the arrangement of components that lets a language model work on a task by itself: the model, its instructions, the tools it can call, its memory, the loop that runs it, and the controls around it. It also fixes the order in which a request moves through those parts, which tells you where context gets added and where a person can stop an action.

The vendor guides start smaller. OpenAI’s guide says “In its most fundamental form, an agent consists of three core components”: a model, tools, and instructions, the last described as “Explicit guidelines and guardrails defining how the agent behaves” openai.com. Anthropic’s version is that “The basic building block of agentic systems is an LLM enhanced with augmentations such as retrieval, tools, and memory” anthropic.com.

Anthropic separates two kinds of system. “Workflows are systems where LLMs and tools are orchestrated through predefined code paths”, while agents “dynamically direct their own processes and tool usage”. Both use the same components. The difference is who decides the next step: your code or the model.

What are the components of an AI agent?

The common core is a model, instructions and tools. Agents that run on real business work add memory, an orchestration loop, guardrails, a source of current business rules, and tracing. The table is our synthesis of the papers and guides cited on this page; the “what fails” column is the failure each source describes or implies when that part is missing.

AI agent architecture as a layered table: each component, its job, example implementations and what fails without it (our synthesis, checked 20 October 2026)
ComponentJobExample implementationsWhat fails without it
Model (reasoning core) Reads the assembled context and chooses the next action or the final answerAny model with tool calling, such as Claude or GPT modelsNothing decides. A weaker model picks the wrong tool or passes malformed arguments
Instructions States the role, the steps, the limits and when to hand offA system prompt; the instructions field in the OpenAI Agents SDKThe agent improvises the process, and two runs on the same case can end differently
Tools Reads data from other systems and takes actions in themFunction calling, MCP servers, REST APIs, other agents used as toolsThe agent can only produce text. It cannot look anything up or change a record
Working memory Holds the goal, recent steps and tool results for the current runThe context window and the run state kept by the orchestratorThe agent loses track mid-task, repeats tool calls or drops a constraint the user gave
Long-term memory Keeps facts, past episodes and learned procedures across runsVector or graph stores, logs of earlier runs, saved proceduresEach session starts from nothing and the same mistake comes back
Orchestration loop Runs the reason, act, observe cycle; routes, hands off and stopsLangGraph, OpenAI Agents SDK, Claude Agent SDK, or a plain loop in your codeNo exit condition, so runs loop, overspend or stop halfway with no retry
Guardrails and approvals Screens inputs and outputs and pauses risky actions for a personRelevance and safety classifiers, regex rules, a moderation API, MCP confirmation promptsInjected instructions get followed and irreversible actions run unreviewed
Business rules and context Supplies the current policies and exceptions that govern this case, and who approved themPolicy documents in retrieval, a rule engine, a governed context layerThe agent applies a generic or lapsed policy, and nobody can say which rule it used
Tracing and evaluation Records each model call, tool call and decision, and tests changes before releaseOpenTelemetry GenAI agent spans, trace viewers, evaluation setsA bad decision cannot be reproduced and a fix cannot be shown to work
Trail (one option for the rules row) Returns the approved rules that govern a case, each cited to document, page and lineNative MCP server, LangChain and LlamaIndex retrievers, REST and GraphQLNot a model or an agent runtime; the agent queries Trail, and approvals and writes stay in your own systemstrailhq.com

Facts in this table were checked against the sources listed at the end of this page, between and .

Tools come in three kinds in OpenAI’s guide: data tools that “retrieve context and information”, action tools that change records or send messages, and orchestration, where “Agents themselves can serve as tools for other agents”. The Model Context Protocol is a protocol for plugging tools in. It “follows a client-host-server architecture where each host can run multiple client instances” modelcontextprotocol.io. The 2026-07-28 revision removed the initialize handshake modelcontextprotocol.io, and since then “MCP is a stateless protocol” modelcontextprotocol.io. The protocol details stay in the trailhq.com explainer on what an MCP server is.

Memory has its own vocabulary. The CoALA paper splits it into working memory, which holds “active and readily available information as symbolic variables for the current decision cycle”, and three long-term stores: episodic, semantic and procedural arxiv.org. Our guide to what agent memory is covers the four types in depth.

The orchestration loop decides when the agent stops. OpenAI’s guide says every approach “needs the concept of a ‘run’”, a loop that runs “until an exit condition is reached”, and lists “tool calls, a certain structured output, errors, or reaching a maximum number of turns” as common exits. LangGraph describes itself as “a low-level orchestration framework for building, managing, and deploying long-running, stateful agents” github.com; our LangGraph explainer covers when that is worth it.

Tracing has a standard in progress. OpenTelemetry’s GenAI conventions define spans such as invoke_agent and execute_tool, so one trace can show which agent ran and which tools it called. The conventions are still marked Development github.com.

How does a request flow through an AI agent?

A request enters, gets screened, picks up context, then loops between the model and its tools until an exit condition fires. Risky actions pause for a person, and everything is logged on the way out. The flow below is the text form of the usual architecture diagram, using an accounts payable agent that receives a supplier invoice as the example. The order is ours, assembled from the sources on this page.

One request through an AI agent, step by step

Request flow
  1. A trigger starts a runAn invoice email arrives, or a user asks a question. The orchestrator opens a run with a turn limit and a budget.
  2. Input guardrails screen itClassifiers and simple rules check relevance and look for injected instructions. Text inside the invoice is handled as data, and the agent's permissions assume it may carry a hidden instruction.
  3. Context is assembledInstructions, the run state and anything retrieved from long-term memory are combined with the business rules that govern this vendor, amount and entity.
  4. The model chooses an actionIt reads the context and either answers or emits a tool call with arguments, for example a purchase order lookup with the PO number from the invoice.
  5. The host runs the toolThe host checks the agent may call that tool, sends the call through the MCP client or API, and adds the result to working memory as an observation.
  6. The loop repeats until an exit conditionSteps 4 and 5 run again until there is a final output, an error, or the turn limit is reached.
  7. Risky actions wait for a personPosting the invoice or releasing a payment pauses for approval. Reading a PO does not.
  8. Output is checked and writtenOutput guardrails check the result, and the agent writes to the target system with the rules it applied attached.
  9. The run is traced and corrections are queuedEvery model and tool call is recorded. A reviewer correction becomes a proposed change to memory or rules, reviewed before any agent reads it.

Steps 2, 5 and 7 are the security boundaries. Step 3 is where a model can reason correctly and still get the case wrong, because it was handed a generic or lapsed rule.

Which papers and guides define the standard agent diagram?

Three research papers and two vendor guides supply most of what a typical diagram shows. The papers explain why the loop looks the way it does; the guides explain how teams build it.

ReAct (Yao and colleagues, first posted October 2022) proposed letting a model “generate both reasoning traces and task-specific actions in an interleaved manner” arxiv.org. The think, act, observe loop in the vendor guides below follows the same pattern.

Toolformer (Schick and colleagues, February 2023) trained a model “to decide which APIs to call, when to call them, what arguments to pass”, and how to use the results in what it writes next arxiv.org. A tool-calling model makes the same four decisions each time it might call a tool.

CoALA (Sumers, Yao, Narasimhan and Griffiths, September 2023) is a framework for describing language agents. In the paper’s words, CoALA “organizes agents along three key dimensions”. The first is information storage, “divided into working and long-term memories”. The second is the action space, “divided into internal and external actions”. The third is the decision-making procedure, “structured as an interactive loop with planning and execution” arxiv.org. If you need one academic diagram to work from, use CoALA’s.

Anthropic’s “Building effective agents” (December 2024) names five workflow patterns (prompt chaining, routing, parallelization, orchestrator-workers and evaluator-optimizer) and describes agents as “typically just LLMs using tools based on environmental feedback in a loop”. The page now carries a note that much of the tooling it mentions has changed since then; the five patterns do not depend on any one tool. OpenAI’s practical guide adds the run loop, the tool types, layered guardrails and human intervention.

When should an agent architecture use more than one agent?

After a single agent has hit a limit you can name. OpenAI’s guide recommends that you “maximize a single agent’s capabilities first”, and Anthropic recommends “finding the simplest solution possible, and only increasing complexity when needed”. We turned both into a decision rule: move down the list only when the step above fails a test you can name.

Which agent pattern to start with (our decision rule from the OpenAI and Anthropic guides)

Checklist
  • One model call with retrievalUse it when the task is a single answer from known documents. Anthropic notes this is usually enough for many applications.
  • A fixed workflowUse prompt chaining, routing or parallel calls when you can write the steps down in advance. You get predictable cost and easier testing.
  • One agent in a loop with toolsUse it when the number of steps depends on what the agent finds, such as chasing a mismatch across three systems.
  • A manager agent with specialist agents as toolsUse it when the prompt fills with if-then branches, or when similar tools still get mixed up after you rename and describe them better. OpenAI notes some implementations manage more than 15 distinct tools while others struggle with fewer than 10 overlapping ones.
  • Peer agents that hand offUse it when separate teams own separate domains, such as sales and support, and a case should move fully from one agent to the other.

Each step down adds latency and cost. Anthropic notes that agentic systems “often trade latency and cost for better task performance”. A multi-agent system also multiplies the places where context can go stale, which is a reason to keep business rules in one shared source rather than copied into each agent’s prompt.

Where do security and approval controls sit in the architecture?

At the boundaries: where untrusted data enters the context, where a tool can change something, and where long-term memory or rules get written. A model-level safety filter does not cover all three.

The first boundary exists because of how agents are built. NIST’s Center for AI Standards and Innovation writes that “The architecture of current LLM-based agents generally requires combining trusted developer instructions with other task-relevant data into a unified input”. Attackers exploit this, the post says, with an email, file or website carrying hidden instructions nist.gov. Tool results and retrieved documents therefore belong in the context as data, and the agent’s permissions should assume some of that data is hostile.

The second boundary is the tool call. The MCP specification puts permission decisions in the host, not the server: the host “Enforces security policies and consent requirements”, and “Servers should not be able to read the whole conversation” modelcontextprotocol.io. Its tools section adds that “there SHOULD always be a human in the loop with the ability to deny tool invocations” modelcontextprotocol.io. OpenAI’s guide names the actions that warrant it: those that are “sensitive, irreversible, or have high stakes”, such as canceling orders, authorizing large refunds, or making payments. It also says to “Think of guardrails as a layered defense mechanism”, combining model-based checks with rules such as regex and input limits.

The third boundary is the write path for memory and rules. CoALA warns that letting an agent write to its own procedural memory “is significantly riskier than writing to episodic or semantic memory, as it can easily introduce bugs or allow an agent to subvert its designers’ intentions” arxiv.org. Business rules are procedural knowledge in that sense. An architecture that lets an agent rewrite the rules it follows, without a person reviewing the change, has moved a security decision into the model. Our guide on governing a memory layer for LLM agents covers write permissions in detail.

How Trail plugs into the rules layer of an agent

Trail, the memory and context layer for enterprise AI agents, mainly fills an agent’s business rules layer: the approved, cited rules it reads before acting. Its Agent Builder also drafts an agent’s role, phases, tools and approval steps from an SOP, but the finished agent “runs inside Claude, ChatGPT or your own stack” trailhq.com. Its how-it-works page says “Trail isn’t where the work happens; it’s what the agent consulted” trailhq.com.

In the request flow it acts at step 3. The agent asks over the “native MCP server, a LangChain or LlamaIndex retriever, or plain REST and GraphQL”, and what comes back is “a short, ordered set of rules”, each cited and scored, rather than a page of retrieved text. Reads are role-scoped, and your team sets a confidence threshold “below which an agent must stop and ask rather than proceed”, which gives step 7 a second trigger besides the action type.

It also handles the third security boundary. “Nothing becomes available to an agent because a model inferred it” trailhq.com, and an agent that spots a problem can flag it but cannot change a rule: “It can report. It cannot approve.”

What Trail does not do: run the agent. Its finance page says approvals and writes “stay in the systems your team already controls” trailhq.com. You still need a model, an orchestrator, guardrails and tracing. For a prototype with a handful of rules that rarely change, a well-kept system prompt is simpler. The comparison of rule engines for finance and operations agents shows the other ways to fill this row.

How we built this architecture guide

We read the ReAct, Toolformer and CoALA papers on arXiv (abstracts, plus the CoALA full text for its memory definitions). From the vendors, we read Anthropic’s “Building effective agents” and OpenAI’s “A practical guide to building agents” PDF. The other sources were the Model Context Protocol specification dated 2026-07-28, NIST’s CAISI post on agent hijacking, the OpenTelemetry GenAI agent span conventions and the LangGraph README. We read the top-ranking architecture guide to see what it covered, and Trail’s homepage, how-it-works, rule engine, Agent Builder and finance pages for every Trail claim. Every quote was fetched and matched word for word on 20 October 2026. The component table, the request flow and the pattern rule are our synthesis; we did not build or benchmark an agent for this page. Trail publishes this page.

Frequently asked questions

What is AI agent architecture?

AI agent architecture is the arrangement of components that lets a language model act on a task: the model, its instructions, the tools it can call, its memory, the loop that runs it, and the controls around it. It also fixes the order in which a request moves through those parts, so you can see where context is added and where a person can stop an action.

What are the components of an AI agent?

The common core is a model, instructions and tools, which is how OpenAI's agent guide puts it. Production agents add working and long-term memory, an orchestration loop with exit conditions, guardrails and human approval for risky actions, a source of current business rules, and tracing. A simple agent can skip some of these, and the component table on this page lists how each one fails when it is missing.

What should an AI agent architecture diagram include?

Draw the model in the middle, then instructions, tools, memory and the business rules that feed it, and the orchestration loop that runs it. Mark the trust boundaries: where untrusted data enters, which tools can change records, and where a person approves.

Do you need a framework like LangGraph to build an AI agent?

No. Anthropic's guide suggests starting with model APIs directly, because many patterns take a few lines of code, and warns that frameworks can hide the underlying prompts and responses. A framework earns its place when you need durable state, resumable runs or many coordinated agents. If you use one, read what it does under the hood.

What drives the cost of running an AI agent?

Mostly the number of model calls in each run and the size of the model behind each call. Anthropic notes that an agent's autonomy brings higher costs and the risk of compounding errors. OpenAI's guide suggests prototyping with the most capable model to set a baseline, then swapping in smaller models wherever the results stay acceptable. A turn limit on each run caps the worst case.

How should you test an AI agent before it goes live?

Set up evaluations first, since OpenAI's guide uses them to set the performance baseline that any later switch to a smaller model is measured against. Anthropic recommends extensive testing in sandboxed environments with guardrails in place. After that, OpenAI's advice is to start small, validate with real users and add capabilities over time.

Sources checked

Show 18 sources