Best AI agent framework in 2026: five built, one task

There is no single best AI agent framework in 2026. For a TypeScript team, the Vercel AI SDK is the smallest step from “call a model” to “agent that waits for approval”, and Mastra adds saved, resumable runs on top. For Python, Pydantic AI is the closest match, LangGraph suits long, branching flows with checkpoints, and CrewAI suits multi-agent “crews” inside a larger flow.

We built the same small refund agent in all five on 2026-10-13 and ran it three ways. Every version worked in 25 to 35 lines. The real differences showed up in where the approval pause lives, what happens when a tool throws an error, and how you turn on tracing.

Who this is for: a developer or technical founder picking a framework for an agent that touches real business systems. Who should look elsewhere: if you’re choosing between OpenAI’s and Anthropic’s own agent SDKs, read our OpenAI Agents SDK vs Claude Agent SDK comparison instead. We didn’t repeat it here.

The test

One task, the same in all five: a customer writes “Order 4471 arrived broken. Please refund it.” The agent has two tools, get_order and issue_refund. The refund must wait for a person to approve it.

We ran each version three times:

1. Approve. The manager says yes. A refund of $129 should happen once.
2. Deny. The manager says no. No refund should happen.
3. Tool failure. get_order throws “503 from orders API” on its first call. Does the run crash, or does the model see the error and try again?

To keep runs repeatable and free, each framework talked to a scripted fake model that makes the same tool calls a real model would, using each framework’s own testing hooks (MockLanguageModelV4 in the AI SDK, a BaseChatModel subclass for LangChain, FunctionModel in Pydantic AI, a BaseLLM subclass in CrewAI). So this tests the framework’s plumbing, not model quality. We did not measure speed or token use.

Versions, all released before 2026-09-28: Vercel AI SDK ai 7.0.118, @mastra/core 1.71.0, langchain 1.5.14 with @langchain/langgraph 1.4.18, pydantic-ai-slim 2.51.0, and crewai 1.15.22. Node 24 and Python 3.14, except CrewAI, which ran on Python 3.12 because its install failed on 3.14 while building an old pyyaml.

Results

Lines are non-blank, non-comment lines in each agent file, including imports. They leave out the shared fake back office and the scripted model, which were the same for all five.

Two results surprised us.

Pydantic AI does not send tool errors back to the model by default. Its docs say so plainly: “Any other exception propagates out of the agent run and is not sent back to the model.” You raise ModelRetry for errors the model should see (Pydantic AI, advanced tools). That is a sound default for money-moving code, because a crash is loud. But if you port an agent from TypeScript, where all three frameworks we tried passed the error to the model, it will behave differently.

Mastra’s approval needs storage, even in a test. Our first run suspended correctly, then failed to resume with “could not find a suspended run”. Mastra’s docs: “Configure a storage provider on your Mastra instance or you’ll see a “snapshot not found” error” (Mastra, agent approval). Two lines fixed it. In production you’d use Postgres or LibSQL, because “The default in-memory store loses snapshots when the process exits.”

How each one handles approval

Approval is the feature that matters most for business agents. It’s also where the five differ most.

• Vercel AI SDK returns a tool-approval-request and stops. You save the messages, add a tool-approval-response later, and call the agent again. Nothing is stored for you. That is simple and easy to put behind a Slack button, but the saving is your job (AI SDK docs).
• Mastra suspends the run and keeps a snapshot in storage. You resume with approveToolCall() or declineToolCall() and the run id. A decline reason “is returned to the model in place of the tool result” (Mastra).
• LangGraph pauses with an interrupt. “You must configure a checkpointer to persist the graph state across interrupts”, and you resume with a Command holding approve, edit, or reject decisions (LangChain docs). Its edit decision lets a person change the refund amount before it runs.
• Pydantic AI ends the run with a DeferredToolRequests output. You pass a DeferredToolResults with the old message history to a new run (Pydantic AI, deferred tools). For runs that must survive restarts, it lists durable execution integrations, including Temporal, DBOS, and Prefect, for “durable agents that can preserve their progress across transient API failures and application errors or restarts” (Pydantic AI, durable execution).
• CrewAI approves a step of a Flow, not a single tool call. “By default, @human_feedback blocks execution waiting for console input.” For Slack or email, a provider raises HumanFeedbackPending, and CrewAI “automatically persists state” until you resume (CrewAI docs). So in our CrewAI version the agent decides the refund, a person approves, and plain Python issues it. The agent never holds the refund tool at all.

For a full walk-through of approval gates outside any one framework, read how to make an AI agent wait for human approval.

Tracing

All five can show you each model call and tool call. They differ on where the traces go.

• Vercel AI SDK: “The AI SDK uses OpenTelemetry to collect telemetry data.” You register @ai-sdk/otel once and send spans to any OpenTelemetry backend (AI SDK telemetry).
• Mastra: an Observability instance with exporters. It “supports external tracing providers like Langfuse, Datadog, and any OpenTelemetry-compatible platform” (Mastra observability).
• LangGraph: set LANGSMITH_TRACING=true and an API key to send traces to LangSmith (LangGraph observability).
• Pydantic AI: “Pydantic AI’s instrumentation uses OpenTelemetry (OTel), which Logfire is based on”, so you can send it to Logfire or another OTel backend (Pydantic AI, Logfire).
• CrewAI: tracing=True on a Crew or Flow sends traces to the CrewAI AMP platform after you log in (CrewAI tracing).

If you already run OpenTelemetry, the AI SDK, Mastra, and Pydantic AI fit in with the least new tooling.

Which one to pick

Our own default for client work is in the AI SDKs we reach for. For simple fixed flows you may not need a framework, or an agent, at all. Our AI agent or Zapier, n8n, Make post covers that line.

The code

Each file below is the full agent, exactly as we ran it. fixtures holds the fake back office and the prompt. The scripted model is left out.

import { ToolLoopAgent, tool, type ModelMessage } from 'ai';
import { z } from 'zod';
import { getOrder, issueRefund, INSTRUCTIONS, PROMPT } from './fixtures.ts';
import { scriptedModel } from './scripted-aisdk.ts';

const agent = new ToolLoopAgent({
  model: scriptedModel(),
  instructions: INSTRUCTIONS,
  tools: {
    get_order: tool({
      description: 'Look up an order by id',
      inputSchema: z.object({ order_id: z.string() }),
      execute: async ({ order_id }) => getOrder(order_id),
    }),
    issue_refund: tool({
      description: 'Refund an order',
      inputSchema: z.object({ order_id: z.string(), amount: z.number() }),
      execute: async ({ order_id, amount }) => issueRefund(order_id, amount),
    }),
  },
  toolApproval: { issue_refund: 'user-approval' },
});

export async function run(approve: boolean) {
  const messages: ModelMessage[] = [{ role: 'user', content: PROMPT }];
  const first = await agent.generate({ messages });
  messages.push(...first.responseMessages);
  const requests = first.content.filter((p) => p.type === 'tool-approval-request');
  // In production you save `messages` here and resume when the manager clicks a button.
  messages.push({
    role: 'tool',
    content: requests.map((r) => ({ type: 'tool-approval-response', approvalId: r.approvalId, approved: approve })),
  });
  const final = await agent.generate({ messages });
  return final.text;
}

import { Mastra } from '@mastra/core';
import { Agent } from '@mastra/core/agent';
import { InMemoryStore } from '@mastra/core/storage';
import { createTool } from '@mastra/core/tools';
import { z } from 'zod';
import { getOrder, issueRefund, INSTRUCTIONS, PROMPT } from './fixtures.ts';
import { scriptedModel } from './scripted-aisdk.ts';

const getOrderTool = createTool({
  id: 'get_order',
  description: 'Look up an order by id',
  inputSchema: z.object({ order_id: z.string() }),
  execute: async ({ order_id }) => getOrder(order_id),
});

const issueRefundTool = createTool({
  id: 'issue_refund',
  description: 'Refund an order',
  inputSchema: z.object({ order_id: z.string(), amount: z.number() }),
  requireApproval: true,
  execute: async ({ order_id, amount }) => issueRefund(order_id, amount),
});

const agent = new Agent({
  id: 'refunds',
  name: 'Refunds',
  instructions: INSTRUCTIONS,
  model: scriptedModel(),
  tools: { get_order: getOrderTool, issue_refund: issueRefundTool },
});

// Suspended runs are saved here. Use a real database (Postgres, LibSQL) in production.
new Mastra({ agents: { agent }, storage: new InMemoryStore() });

export async function run(approve: boolean) {
  const first = await agent.generate(PROMPT);
  if (first.finishReason !== 'suspended') return first.text;
  // In production the run is saved to storage and resumed by runId later.
  const resume = { runId: first.runId!, toolCallId: first.suspendPayload.toolCallId };
  const final = approve ? await agent.approveToolCallGenerate(resume) : await agent.declineToolCallGenerate(resume);
  return final.text;
}

import { createAgent, humanInTheLoopMiddleware, tool } from 'langchain';
import { MemorySaver, Command } from '@langchain/langgraph';
import { z } from 'zod';
import { getOrder, issueRefund, INSTRUCTIONS, PROMPT } from './fixtures.ts';
import { ScriptedChatModel } from './scripted-langchain.ts';

const getOrderTool = tool(async ({ order_id }) => JSON.stringify(await getOrder(order_id)), {
  name: 'get_order',
  description: 'Look up an order by id',
  schema: z.object({ order_id: z.string() }),
});

const issueRefundTool = tool(async ({ order_id, amount }) => JSON.stringify(await issueRefund(order_id, amount)), {
  name: 'issue_refund',
  description: 'Refund an order',
  schema: z.object({ order_id: z.string(), amount: z.number() }),
});

const agent = createAgent({
  model: new ScriptedChatModel({}),
  systemPrompt: INSTRUCTIONS,
  tools: [getOrderTool, issueRefundTool],
  middleware: [humanInTheLoopMiddleware({ interruptOn: { issue_refund: true } })],
  // Saves the paused run. Use PostgresSaver in production.
  checkpointer: new MemorySaver(),
});

export async function run(approve: boolean) {
  const config = { configurable: { thread_id: 'refund-4471' } };
  await agent.invoke({ messages: [{ role: 'user', content: PROMPT }] }, config);
  const decision = approve ? { type: 'approve' } : { type: 'reject', message: 'Refund denied by manager' };
  const final = await agent.invoke(new Command({ resume: { decisions: [decision] } }), config);
  return String(final.messages.at(-1)?.content);
}

from pydantic_ai import Agent, ModelRetry, DeferredToolRequests, DeferredToolResults, ToolDenied
from fixtures import get_order_impl, issue_refund_impl, INSTRUCTIONS, PROMPT, report
from scripted_pydantic import scripted_model

agent = Agent(scripted_model, instructions=INSTRUCTIONS, output_type=[str, DeferredToolRequests])

@agent.tool_plain
def get_order(order_id: str) -> dict:
    """Look up an order by id."""
    try:
        return get_order_impl(order_id)
    except RuntimeError as e:
        raise ModelRetry(f"Orders API error: {e}")  # sends the error back to the model

@agent.tool_plain(requires_approval=True)
def issue_refund(order_id: str, amount: float) -> dict:
    """Refund an order."""
    return issue_refund_impl(order_id, amount)

def run(approve: bool) -> str:
    first = agent.run_sync(PROMPT)
    if not isinstance(first.output, DeferredToolRequests):
        return first.output
    # In production you store first.all_messages() and resume when the manager answers.
    results = DeferredToolResults()
    for call in first.output.approvals:
        results.approvals[call.tool_call_id] = True if approve else ToolDenied("Refund denied by manager")
    final = agent.run_sync(message_history=first.all_messages(), deferred_tool_results=results)
    return final.output

report("pydantic-ai", run)

import json
from crewai import Agent, Crew, Task
from crewai.flow import Flow, start, listen, human_feedback
from crewai.tools import tool
from fixtures import get_order_impl, issue_refund_impl, INSTRUCTIONS, PROMPT, report
from scripted_crewai import ScriptedLLM

@tool("get_order")
def get_order(order_id: str) -> dict:
    """Look up an order by id."""
    return get_order_impl(order_id)

agent = Agent(role="Refunds", goal=INSTRUCTIONS, backstory="Support team member.", tools=[get_order], llm=ScriptedLLM())
task = Task(description=PROMPT + " Reply with JSON: order_id and amount to refund.", expected_output="JSON", agent=agent)

class RefundFlow(Flow):
    @start()
    @human_feedback(message="Approve this refund? Type yes or no.")
    def decide(self):
        return Crew(agents=[agent], tasks=[task]).kickoff().raw

    @listen(decide)
    def refund(self):
        if not self.last_human_feedback.feedback.strip().lower().startswith("y"):
            return "The refund was not approved."
        plan = json.loads(self.last_human_feedback.output)
        issue_refund_impl(plan["order_id"], plan["amount"])
        return f"Refunded ${plan['amount']} for order {plan['order_id']}."

report("crewai", lambda approve: RefundFlow().kickoff())

FAQ

What is the best AI agent framework in 2026?

It depends on your language and on how approvals must work. In our test, the Vercel AI SDK was the simplest for TypeScript and Pydantic AI for Python. LangGraph and Mastra keep paused runs for you. CrewAI fits multi-agent work inside a flow.

Is LangGraph better than CrewAI?

For one agent with tools and approvals, LangGraph gave us per-tool approval with approve, edit, and reject. CrewAI approves whole flow steps. CrewAI is the more natural fit when several role-based agents share a job.

Do I need a framework to build an AI agent?

No. A loop that calls the model, runs the tools it asks for, and stops for approval is a few dozen lines. Frameworks save you the saving and resuming, tracing, and model switching.

Which agent framework is best for TypeScript?

The Vercel AI SDK if you want the smallest piece and will save state yourself. Mastra if you want storage, approvals, and observability in one package. LangChain’s createAgent on LangGraph if you also need checkpoints and editable approvals.

Can I test an agent without paying for API calls?

Yes. All five let you plug in a fake model. We used MockLanguageModelV4 from ai/test, a LangChain BaseChatModel subclass, Pydantic AI’s FunctionModel, and a CrewAI BaseLLM subclass. None of our runs called a paid API.

Dawloom builds AI agents with approval gates and audit trails, from a written scope, billed hourly or by milestone. If you need an agent to reach your own systems from Claude or ChatGPT, see MCP servers and what an MCP server is.