The next article initially appeared on Paolo Perrone’s Substack,The AI Engineer, and is being republished right here with the creator’s permission.
You spent three weeks transport an agent. It labored within the demo. Then manufacturing hit, and also you realized the framework you picked has no checkpointing, the reminiscence layer is a flat vector dump with no temporal reasoning, the browser instrument falls over on any website with a canvas component, and the eval suite is a Notion doc somebody retains forgetting to replace.
The open supply toolkit for constructing brokers in 2026 has solved most of those issues. The catch is that it has solved each in a dozen incompatible methods. The reminiscence framework that wins LoCoMo (the usual long-conversation reminiscence benchmark) runs 340x heavier per dialog than the runner-up, a distinction no benchmark column reveals. The identical hole between benchmark rating and manufacturing conduct reveals up at each layer.
So one of the best ways to zero in on the constraint your system will hit first underneath load: latency finances, audit path, mannequin portability, or language stack. Get this mistaken and also you rewrite your state schemas in week three.
TL;DR
In case you learn “The AI Brokers Stack (2026 Version),” that is the open supply half. Similar seven layers across the think-act-observe loop from “What Is an AI Agent?”: orchestration, reminiscence, instrument interface, browser/CUA, coding brokers, evals and observability, and inference. Right here’s the place to start out at every layer.
The way to decide at every layer
When selecting instruments at every layer, ask three questions:
What’s the dominant constraint? 4 constraints determine most layer picks. Latency finances is what number of tokens or milliseconds you may spend per flip. Audit path is whether or not each motion needs to be traceable for compliance. Mannequin portability is how tied your stack will get to at least one supplier. Language stack is whether or not your crew is Python, TypeScript, or each. Considered one of these normally dominates at every layer.
What’s the rip-out value should you’re mistaken? Swapping an MCP server modifications one config line. Swapping orchestration rewrites your state schemas, your nodes, and your edges. The larger the rewrite, the extra you need to decide by constraint first.
Is it open supply or open core? Open core means the mission ships underneath an open supply license, however manufacturing options (multitenant auth, replication, SSO, audit logs) solely run within the managed cloud product. The repo’s characteristic record tells you which ones aspect of the road you’re shopping for.
Layer 1: Orchestration and runtime management
The orchestration layer runs the agent’s reasoning cycle. The LLM picks an motion, the runtime executes it, the runtime observes the end result, and the LLM picks once more. In case you skip a framework right here, you write the loop your self, which implies reinventing retries, checkpointing, and human-in-the-loop gating earlier than you ship.
LangGraph is the default for Python manufacturing work. Graph-based state machine, sturdy execution by way of PostgresSaver, time-travel debugging, and the most important verified enterprise record within the subject (Klarna, Uber, LinkedIn, JPMorgan, Replit). Graph state maps onto what regulated industries want: Each state transition is an audit log entry, and any failed run rolls again to a previous node and replays from there. The ceiling: It’s verbose. A two-agent circulate nonetheless wants a state schema, nodes, edges, and compilation. For “name three instruments sequentially,” it’s overkill.
CrewAI has the bottom setup overhead of the 4 orchestration frameworks. You declare roles like researcher, author, and reviewer, decide a coordination sample, and run the crew with no state schema to outline first. The ceiling: CrewAI optimizes for prototype velocity at the price of manufacturing sturdiness. The framework can’t resume crashed runs from the place they failed, error dealing with lives on the crew stage quite than per-node, and no inspectable state schema data what the brokers determined and when. Groups transfer from CrewAI to LangGraph when manufacturing state administration begins mattering greater than the position metaphor.
Pydantic AI treats each agent output as a typed Pydantic mannequin, so validation, retries, and downstream serialization come without spending a dime. FastAPI-style decorators for instruments and dependencies. The ceiling: Pydantic has weaker multi-agent primitives than CrewAI or LangGraph. It’s the most effective match when the agent is a single loop that has to return validated knowledge to a downstream service.
Mastra is the TypeScript reply: brokers, workflows, RAG, and evals in a single bundle, constructed by the ex-Gatsby founders, designed to drop into current Subsequent.js apps with no Python sidecar. The ceiling: smaller ecosystem and fewer manufacturing case research than LangGraph. Select Mastra when the crew is already on TypeScript finish to finish and rewriting in Python isn’t on the desk.1
The seller SDKs (Claude Agent SDK, OpenAI Brokers SDK, Google ADK) belong right here too. Every one removes orchestration friction and locks the agent to at least one supplier’s API. Decide one should you’re already dedicated to that supplier and never planning to swap fashions.
Layer 2: Reminiscence and state
The context window isn’t reminiscence. Even at 200K tokens, each flip pays for the complete dialog once more, and nothing survives the session. Manufacturing brokers in 2026 hold reminiscence in a devoted layer that lives outdoors the immediate.2
Mem0 reminiscence could be scoped to a consumer (persists throughout all their classes), a session (simply this dialog), or an agent (shared throughout all customers of 1 agent). Hybrid storage combines vectors and a graph, with mature SDKs that plug into LangGraph, CrewAI, and Mastra. The mission has 48,000+ GitHub stars. Mem0’s ECAI 2025 paper benchmarked Mem0 in opposition to 10 options on LoCoMo and reported 92% decrease latency and 93% fewer tokens versus naive full-context (the baseline each crew replaces by week two), which interprets to roughly 14x cheaper inference on the identical recall.3 The ceiling: Mem0 treats reminiscence as retrieval, returning essentially the most comparable details to a question. Temporal reasoning, like “what did the consumer say final week that contradicts what they mentioned at the moment,” wants a graph that tracks edges between details with timestamps.4
Zep/Graphiti is the temporal graph possibility. The data graph layer handles entity decision: determining that “Alice,” “alice@acme.com,” and “the CEO” all confer with the identical particular person. It additionally tracks how relationships change over time, so the agent can reply, “What did this buyer’s standing appear to be in Q2?” or “When did the contract proprietor change?” The trade-off is that graph development is pricey. Zep’s reminiscence footprint per dialog runs previous 600,000 tokens versus Mem0’s 1,764, and quick postingestion retrieval typically fails as a result of appropriate solutions solely seem after background graph processing completes. Select Zep when the agent must purpose about historical past and you’ll wait seconds, not milliseconds, between turns.
Letta (previously MemGPT) treats reminiscence like an working system. Foremost context is RAM, archival reminiscence is disk, and the agent decides what to advertise into RAM, archive to disk, or overlook. It’s absolutely open supply, mannequin agnostic, and self-hosted from day one. The structure extends an agent’s efficient context far past the LLM’s native window by paging reminiscence out and in, the identical trick working techniques use to provide applications extra digital reminiscence than bodily RAM. The ceiling: You run the storage layer your self. Letta is more durable to deploy than calling a hosted Mem0 endpoint and more durable to debug as a result of reminiscence selections occur contained in the agent at runtime.5
Engineering lesson.“Reminiscence” means two various things in an agent system, and utilizing one instrument for each breaks each. Runtime state is the agent’s scratchpad mid-task: which node it’s at, what instruments it referred to as, what intermediate outcomes it has. LangGraph’s PostgresSaver writes this after each step, so a crashed run resumes from the final node. Information reminiscence is what the agent discovered throughout classes: preferences, prior questions, and details in regards to the consumer. Mem0 and Zep retailer this. Conflate them and also you get an agent that resumes a crashed run appropriately however forgets the consumer the second they open a brand new session, or one which remembers the consumer however can’t recuperate when it crashes mid-task.
Layer 3: Protocols and instruments
Two years in the past this layer was operate calling: Every supplier had its personal JSON schema, and every framework wrapped them in a different way; switching fashions meant rewriting your instruments.
In 2026 this layer is MCP. The Mannequin Context Protocol is the open customary the Claude Agent SDK makes use of, that OpenAI Brokers SDK helps natively, that Google ADK integrates with, that each severe framework now ships a shopper for. In case you’re writing instruments at the moment, you’re writing MCP servers. If MCP itself is fuzzy, “What Is MCP?” is the prerequisite.
There’s no framework to choose at this layer. The orchestration selection from layer 1 already determined how MCP integrates.
FastMCP is the Python framework for writing MCP servers quick. Decorator-based and async-first, it’s the closest factor to FastAPI for MCP. mcp-agent is an orchestration framework constructed round MCP as the first instrument interface. Server lifecycle, multiserver routing, and immediate context dealing with are in-built. With LangGraph or CrewAI, you write that integration code your self. It’s value when your agent connects to a number of MCP servers and the mixing code begins turning into the bottleneck.
Layer 4: Browsers and pc use
When the system the agent has to behave on doesn’t expose an API, the toolkit has to behave by way of screens. The 2026 subject cut up into two architectural approaches: DOM-driven (parse the web page, discover components, and click on them) and vision-driven (screenshot the web page, feed it to a imaginative and prescient mannequin, and click on pixels).
Browser Use is the Python default. With 50,000+ GitHub stars, it’s one of many fastest-growing open supply AI tasks of 2025–2026. The LLM will get full management of the browser by way of an agent loop and integrates with LangChain, CrewAI, and customized frameworks. The ceiling: Each step prices an LLM name, which is okay for novel duties and brutal for repeated workflows. Manufacturing groups cache the repeated 80% in Playwright (the deterministic browser automation library) and go away Browser Use for the 20% that wants reasoning.
Stagehand is the TypeScript reply. It’s an open supply, MIT-licensed SDK from Browserbase, constructed as a layer on high of Playwright. 4 primitives let the developer hold AI inference for the steps that want reasoning and use scripted Playwright code for the remainder. Stagehand v3 (February 2026) rewrote the engine on high of Chrome DevTools Protocol and ships 44% quicker.6 The ceiling: Manufacturing deployment runs by way of Browserbase’s managed cloud. The open supply SDK is the on-ramp.7
Skyvern is the vision-first possibility. Every job runs by way of a three-phase pipeline: Planner breaks the objective into steps, actor sends a screenshot to a imaginative and prescient mannequin and clicks the coordinates it returns, and validator confirms the web page modified. Skyvern scores 85.85% on WebVoyager 2.0, the strongest printed rating on form-filling duties in domains the place the DOM is unreliable: canvas components, React digital DOMs nested in iframes, or antibot equipment. That rating nonetheless interprets to roughly one in seven multistep duties failing. The ceiling: Imaginative and prescient-driven stacks lag DOM-driven ones by 12–17 factors on widespread duties and price 4–8 occasions extra per step.8
The manufacturing sample in 2026 wires each in: DOM-driven as the first path, Skyvern or Anthropic Pc Use or OpenAI CUA because the escape hatch when selectors hold failing on canvas components or antibot screens. Edge surfaces are one of many 4 agent failure modes, and we cowl all 4 in “Why AI Brokers Maintain Failing in Manufacturing.”
Layer 5: Coding brokers and sandboxes
Coding brokers are a class of their very own now. They write code, run it, debug it when it breaks, and browse docs to determine what they acquired mistaken. This layer ships with three issues the opposite six don’t: a sandboxed filesystem to put in writing and edit code with out escaping into the host, terminal entry to run builds, checks, and linters, and a browser instrument as a result of half the work entails studying docs. The class additionally has its personal benchmark, SWE-bench Verified, a curated set of actual GitHub points an agent should resolve right into a working PR. For the closed-source comparability, see “Cursor vs Claude Code.”
OpenHands (previously OpenDevin) is the production-grade autonomous possibility. It has 72,000+ GitHub stars, accomplished a $18.8M Sequence A, and is utilized in manufacturing at AMD, Apple, Google, Amazon, Netflix, and NVIDIA. The event-stream structure strikes by way of 4 states per loop: Agent causes, agent emits an motion, surroundings executes it, surroundings returns an remark. Every session runs in an remoted Docker sandbox. The benchmark query for this class is what share of real-world bug tickets the agent can resolve finish to finish with out human enter. OpenHands scores 53%+ on SWE-bench Verified with Claude 4.5 and as much as 72% with Claude 4 on the printed platform outcomes. The ceiling: The agent has shell entry. Evaluate can’t stay inside OpenHands; it has to stay on the PR.9
Aider is the terminal-native possibility. The unique open supply coding agent, it has 35,000+ GitHub stars and 13,100+ commits throughout 93 releases. It’s Git-integrated by design: Each change turns into a commit with an auto-generated message that names what it touched, so the complete agent session is in your git historical past. Architect/Editor mode splits the work between two fashions: A stronger one plans the edit, whereas a less expensive one writes the code. The cut up cuts value 30%–40% versus working a top-tier mannequin on each token. Aider scores 32% on SWE-bench Verified with Claude 4.5, effectively beneath OpenHands, nevertheless it ships fewer surprises as a result of each motion lands in Git. The ceiling: It’s terminal-only. There’s no IDE integration and no project-wide context past what Aider parses from the recordsdata you cross it.
Cline is the VS Code-native reply. It’s absolutely open supply and modelagnostic, with 38,000+ GitHub stars, and it’s the one possibility right here with a significant market share inside VS Code groups. Plan Mode and Act Mode separate intent from execution: Plan Mode drafts the change record and pauses for approval, and Act Mode executes the permitted plan. Each motion is reviewable earlier than it touches the codebase, which is the design level engineering managers ask about first. Select Cline when the crew lives in VS Code and human evaluate on every step is required by coverage. The ceiling: It’s IDE-locked. JetBrains or Neovim groups ought to have a look at Proceed or the terminal instruments above.
Most groups working manufacturing coding brokers in 2026 run two: one business (Claude Code, Codex) for laborious duties and one open supply for flexibility and outages. “How Cursor Truly Works” reveals what the main business coding agent truly does underneath the hood.
Layer 6: Evals and observability
The evals and observability layer data what the agent did in manufacturing and checks what it may do earlier than transport. Tracing captures each LLM name, instrument invocation, and price, listed by consumer and session, so when an output is mistaken, you may replay the precise context that produced it. Evals are reproducible take a look at suites the agent runs in opposition to fastened inputs with cross/fail standards scored the identical approach each time. Manufacturing-grade agent groups in 2026 wire each in on day one. Skipping this layer is the most costly mistake in agent engineering.
Langfuse is the open supply observability default. It’s open core with a beneficiant self-hosted tier and native integrations with LangGraph, CrewAI, OpenAI Brokers SDK, and Mastra. Each LLM name, instrument invocation, and price will get traced and listed. The ceiling: Managed retention, SSO, and superior eval options run on the SaaS plan. The self-hosted model covers tracing and dashboards.
Arize Phoenix is the OpenTelemetry-native various. Traces circulate into the identical Grafana, Datadog, or Honeycomb dashboards the remainder of your stack already makes use of, so agent telemetry sits subsequent to your API and repair traces as an alternative of in a separate instrument. It’s robust on RAG evals and retrieval high quality. The ceiling: Phoenix doesn’t ship opinionated agent-specific defaults. The pipeline meeting is on you.
Examine AI is the UK AI Safety Institute’s open supply eval framework. The institute wrote it for security evals: testing whether or not the agent refuses jailbreaks, leaks PII, or generates unsafe content material. Frontier labs now use it for functionality and alignment benchmarking too. The ceiling: Examine is for offline analysis. In case you additionally must see what the agent is doing stay in manufacturing, you’ll need Langfuse or Phoenix subsequent to it.
Engineering lesson.Wire tracing in on Day 1, earlier than the primary consumer. Establishing Langfuse or Phoenix at mission begin is a few hours of config work. With out these data, debugging a manufacturing failure means guessing which immediate model, which consumer enter, and which instrument sequence produced it.
Layer 7: Fashions and inference
Each step an agent takes is not less than one inference name, typically extra. The engine working these calls, the software program wrapping the GPU, batching requests, and managing the KV cache, units the price ground for the whole lot else. Hosted API brokers inherit their supplier’s engine. Self-hosted brokers decide their very own, and the decide determines what the agent prices to run at scale.
vLLM is the manufacturing serving default for open-weight fashions. Its core innovation is PagedAttention, a reminiscence administration trick that splits the KV cache into fixed-size blocks so a number of requests share GPU reminiscence with out wasted area. Mixed with steady batching, it produces the very best throughput-per-dollar within the subject. The ceiling: vLLM is GPU solely and optimization heavy, and it assumes the operator is aware of what KV cache means.
Ollama is the native default. After a one-line set up, it downloads quantized fashions from a registry and exposes an OpenAI-compatible API. Quantization compresses weights from 16 bits all the way down to 4 or 8, buying and selling a small accuracy hit for becoming in laptop computer RAM. The ceiling: Ollama isn’t a manufacturing serving layer previous a single consumer.
llama.cpp is the engine Ollama runs on high of. Pure C++ with no GPU dependency, it runs LLMs on CPU, Apple Silicon, Raspberry Pi, and anything with sufficient RAM. The mission additionally outlined GGUF, the file format used to ship quantized open-weight fashions, so the identical mannequin file runs throughout each llama.cpp-based instrument unchanged. The ceiling: CPU throughput sits effectively beneath GPU serving, which makes llama.cpp the fitting decide for native and offline workloads solely.
SGLang is the newer challenger. Two design selections set it aside. First, when many requests share a gap immediate, SGLang caches the computation of that prefix as soon as and reuses it, as an alternative of recomputing it for each name. Second, when the agent wants JSON output, SGLang enforces the schema contained in the inference engine itself, so the mannequin can’t generate invalid JSON within the first place. On agent workloads, SGLang benchmarks quicker than vLLM. The ceiling: There’s a smaller neighborhood and fewer integrations, and it’s much less battle-tested than vLLM in manufacturing at scale.
“What Does NVIDIA Truly Do?” breaks down the {hardware} layer each engine on this part finally runs on.
The seven layers don’t compose
The intuition when studying a seven-layer diagram is to imagine the layers compose vertically: Decide layer 1, that constrains layer 2, which constrains layer 3, and the fitting toolkit is the one the place each field suits collectively.
Most agent rewrites in 2026 hint again to a crew that constructed on that assumption. No ecosystem is finest in school in any respect seven layers, and the integrations between layers have been by no means designed to compose. They meet at skinny seams: a config file, an import, an HTTP name. . .
The seven layers are seven unbiased selections. Every one has a dominant constraint that picks the winner. 4 constraints determine most picks: latency finances, audit path, mannequin portability, and language stack.
The 4 constraints hardly ever level on the identical winner. Latency-first stacks pull towards Mem0 and vLLM. Audit-first stacks pull towards LangGraph and Langfuse. Mannequin portability pulls away from vendor SDKs. Language stack pulls towards Mastra or Pydantic AI. Making an attempt to fulfill all 4 with one ecosystem means selecting the typical instrument at each layer as an alternative of the most effective one at every.
The reframe: An agent’s toolkit is seven small bets, every with a single dominant constraint, and every made independently. The groups transport dependable brokers in 2026 are those who picked the most effective instrument per layer and accepted that integrating the seams is a part of the job.
The agent stack cheat sheet
Earlier than swapping any layer in a manufacturing agent, verify this desk first. The state column tells you ways a lot it’s a must to migrate. The lock-in column tells you what you’re giving up should you change. The demo-to-prod column tells you ways lengthy the swap will truly take.
