The definition of an AI agent depends on who you ask. Vendors use it for chatbots and researchers for autonomous systems, while engineering teams often use the term as an umbrella label for automated systems they’ve deployed. Four years into shipping production agents, our definition is:
An AI agent is a system that uses a large language model to decide the control flow of an application.
How agentic the system is depends on how much control flow the model owns. At one end is a simple LLM router, and at the other is a system that picks its own tools and runs over long horizons. The more control you give the model, the more robust your tooling needs to be, especially for observability, evals, memory, permissions, and safe execution.
This definition gives us a practical way to distinguish agents from workflows, describe levels of autonomy, and decide what production infrastructure a system needs.
What is an AI agent?
The common perception is that an AI agent is a system that pursues a goal by iteratively taking actions, evaluating progress, and deciding its own next steps. Anthropic offers a complementary operational definition: agents are systems where LLMs dynamically direct their own processes and tool usage, maintaining control over how they accomplish tasks.
Those definitions describe behavior, but for teams building production systems, the distinction is simpler. An agent starts when the model chooses the next step, rather than only producing text as code carries out a predefined sequence.
Autonomy is a spectrum
Most production systems sit between a workflow and a fully autonomous agent. Their position on that spectrum depends on how much control the model has over outputs, next steps, and available tools.
The spectrum framing lets us stop arguing about which system is "really" an agent and focus on the engineering requirements at different levels of autonomy. Autonomous vehicles got there first. Nobody asks whether a Level 2 driver-assist system is "really" self-driving, because the industry agreed on a numbered scale and engineers talk about which level they are shipping. The AI industry hasn't agreed on its own grading system yet.
The chart below maps six levels of autonomy. For each level, it shows who controls three important decisions.
- Who decides the output of each step?
- Who decides which step comes next?
- Who decides what steps are available?
The chart shows how systems become more autonomous as the LLM takes control of more decisions. You move from hand-written logic to a single LLM call, to a chain, to a Router, to a State Machine, and finally to a fully Autonomous Agent that builds and remembers its own tools. The Voyager paper is the clearest example of that top tier: an agent that learned Minecraft skills across runs and reused them later.
Workflows and AI agents
A workflow orchestrates LLMs and tools through predefined code paths, while an AI agent lets the LLM dynamically direct what happens next. Anthropic files both under "agentic systems," reserving the distinction for a single question: how much of the control flow is fixed in code versus decided by the model at runtime.
In production, the distinction shows up as a design decision. A team that reaches for an agent when a workflow will do introduces non-determinism they do not need. Conversely, a team that reaches for a workflow when the problem needs an agent hard-codes paths that will break the first time the input shifts.
The answer is usually a blend. Use deterministic code for steps with clear requirements, and give the LLM control where the application must interpret unstructured input or choose the next action.
Inside the agent loop
An AI agent is a large language model running in a loop, and that loop is the mechanism by which the model ends up owning control flow. At each turn, it reads the current state, selects an action, invokes a tool, observes the result, updates its memory, and decides whether to continue.
Four components sit inside any agent: a model that decides the next action, tools that give it the ability to read and change the world, memory that persists context across turns, and a reasoning loop that connects them.
| Component | What it does | Examples |
|---|---|---|
| Model | Decides the next action given the current state | A frontier LLM with strong reasoning, tool-calling, and instruction-following |
| Tools | Let the agent retrieve information and take actions in external systems. | APIs, databases, code execution, retrieval, other agents, and MCP servers |
| Memory | Persists context across turns and tasks | Conversation history, long-term state stores, editable instruction files |
| Loop | Sequences perceive, reason, act, observe, update | ReAct, Reflexion, plan-and-execute, and orchestrator-worker patterns |
When you actually need an AI agent
All of that architecture only matters if you actually need an agent. A deterministic workflow, a retrieval system, or a single-call LLM app will outperform an agent on simpler tasks. Reach for an agent only when the control flow has to be decided at runtime.
Our advice:
- Start with a single agent and good prompt engineering. A well-prompted LLM with a handful of tools solves a surprising share of real problems. Many agentic tasks are best handled by a single agent with well-designed tools.
- Add tools before adding agents. If the agent needs a capability, give it a tool. Adding a tool is cheaper and easier to debug than splitting the reasoning across multiple agents.
- Graduate to multi-agent patterns only when you hit clear limits. Context overflow, capability sprawl, and team boundaries are the usual reasons to move to sub-agents, routers, or handoffs. Until one of those limits becomes a real constraint, a single agent is the right answer.
- Consider retrieval before reasoning. A well-built retrieval pipeline outranks a mediocre agent on most knowledge tasks.
- Do not outsource judgment you cannot evaluate. If you wouldn't recognize a correct answer, neither will the agent.
The right level of autonomy depends on the task, the tools it requires, and the team’s ability to evaluate the result.
Improving agents via the Agent Development Lifecycle
LangChain’s Agent Development Lifecycle (ADLC) connects four stages: build, test, deploy, and monitor. Production traces and feedback from monitoring become test cases for the next build, allowing teams to evaluate changes before deploying them.
Online evals and offline evals
Agent evaluation includes online and offline evals, which serve different purposes. Online evals run against a sample of production traces. They are benchmarks rather than ground truths. They tell you whether your helpfulness score dipped overnight, or whether a tool is failing more often than before.
Offline evals run against curated datasets. They are the ground-truth tests that tell you whether a prompt change, a model swap, or a new architecture actually improved the agent.
You need both. Online evals surface problems in production, while offline evals show whether a proposed change improves performance before deployment.
LLM-as-a-judge
LLM-as-a-judge can evaluate production traces at a scale that manual review cannot support. The judge scores an agent’s output against human-defined criteria such as helpfulness, faithfulness, or task completion.
A judge is only as aligned as the criteria you give it, so teams that skip calibration can end up with scores that look rigorous and mean nothing. We calibrate by hand-labeling real traces, running the judge against them, seeing where it disagrees, and iterating on the judge's prompt until the grades line up. Align Evals turns that calibration into a repeatable workflow, which is how we keep the judge accurate as the agent changes.
Enriched traces create a flywheel
Adding corrected traces, negative user feedback, and reviewer edits to the evaluation dataset gives teams a stronger test set for each new agent version. If evals are only run against synthetic data, the agent will only be good at handling imagined problems, which rarely resemble complex real world issues.
Human oversight and agent quality
Human feedback can improve an agent when it is captured in traces and used to strengthen future evaluations. For sensitive or irreversible actions, we recommend human-in-the-loop controls that pause the agent for approval, edits, rejection, or clarification.
Approvals, reviewer edits, and cancellations can become feedback for the next version. When a domain expert reviews a trace in an annotation queue and rewrites the response to match what the agent should have said, that edit becomes a golden dataset row.
Trajectory evals
Human reviewers can use trajectory evals to assess the decisions an agent made before producing its final output. In multi-agent systems, it may be unclear in advance which sub-agent will handle a task, and evaluating only the final output can hide mistakes in sub-agent selection, tool use, the order of steps, or unnecessary calls. A trajectory eval asks whether the agent called the right sub-agent, in the right order, with the right tools.
Running agents reliably in production
Many organizations are experimenting with agents, but few have scaled them. McKinsey’s November 2025 State of AI survey found that 62% of respondents were experimenting with agents, while no more than 10% in any business function had scaled them. Gartner predicts that more than 40% of agentic AI projects will be canceled by the end of 2027 because of escalating costs, unclear business value, or inadequate risk controls.
Moving from a prototype to a production agent requires infrastructure around the model. Teams need observability to inspect runs, evals and datasets to measure changes, sandboxes to isolate code execution, and access controls to limit what the agent can do.
LangSmith Sandboxes run agent-generated code in isolated environments. LangSmith Deployment provides authentication controls for agents that serve multiple users. Together with tracing and evals, these systems help teams diagnose failures, test changes, and deploy agents safely.
If you're starting from scratch, pick the simplest tier on the spectrum that your problem actually needs. Capture traces from day one with LangSmith, our framework-agnostic agent observability platform, which works with LangChain, LangGraph, Deep Agents, any other framework, or custom code, and build eval coverage before your first user touches the agent.