The AI agent integration problem

Quick answer

The stitching problem is what happens when you build a multi-agent system by connecting agents one at a time, without designing the team structure first. Each agent works. The joins between them don't — because nobody defined roles, handoffs, tool boundaries, or who owns what when something goes wrong. The fix isn't a better model. It's a design step before you build anything.

How it starts

It always begins with one agent that works. You give it a task, it handles it well, and you move on. Then the task gets bigger, or a second use case emerges alongside it, and the natural thing to do is add another agent.

Adding the second agent feels fine too — it also works. So you wire them together. The first one does its thing, passes output to the second, the second does its thing. You've got a small pipeline. Still feels fine.

Then comes the third, or a fourth. Each one gets added at the point where you need it, connected to whatever came before. The team grows incrementally, one bolt at a time, and the whole thing is running before you stop to ask whether the structure holding it together was ever actually designed.

That's where the problem lives. Not inside any single agent — in the gaps between them.

Where does it break?

Overlapping instructions

When agents are added one at a time without a defined team structure, their instructions drift into each other's territory. The Researcher starts making editorial calls because nobody told it explicitly that the Writer owns those decisions. The Triage agent starts composing responses because its instructions never said "categorise only — don't write." Two agents that each think they're responsible for the same thing produce conflicting output, or one silently overwrites the other's work.

This is a role problem, and it can't be fixed by adjusting the prompt. You can add "don't do X" to every agent's instructions, but without a clear model of what each role owns, the boundary will keep slipping — especially as edge cases accumulate.

Unclear handoffs

A handoff is the moment one agent finishes and another starts. When handoffs aren't designed explicitly, what usually happens is this: the output format of Agent A is whatever made sense when A was built, and Agent B's input expectation is whatever made sense when B was built. They're different. The integration code in between is the patch, and every time either agent changes, the patch breaks.

More subtly: when the handoff isn't defined, nobody specified what a "successful" completion looks like for Agent A. So Agent B sometimes receives good output, sometimes receives partial output from an edge case, and occasionally receives an error that A flagged internally but passed along anyway. Without a defined handoff, Agent B has no idea which one it's dealing with.

Tool conflicts

In a stitched-together system, tools often end up shared across agents in ways that weren't thought through. Two agents both have write access to the same record. A third agent reads it mid-update. Access was never scoped per-agent because the agents were added one at a time, each just asking for what it needed.

The result is a class of failures that are very hard to diagnose: the output is wrong, nothing threw an error, and the logs show every agent "succeeded." Tracing what actually happened requires reading the entire execution linearly to figure out which agent touched what and in what order.

No gates — until after the first mistake

The most common approach to human oversight in a stitched system is to review it manually after it runs. Check the output, catch errors, fix what went wrong. This works at low volume. It doesn't scale, and it doesn't help when the error is an action that can't be undone — an email sent to the wrong person, a record updated incorrectly, a notification fired before the data behind it was ready.

Gates (human review checkpoints that pause the pipeline before a consequential step) are rarely built into stitched systems because nobody stopped to ask "where does this need a human check before it proceeds?" That question has to be answered before the pipeline is built, not added as a layer afterwards.

Why is this a design problem, not an implementation problem?

There's a tempting diagnosis when a multi-agent system fails at the joins: the model isn't smart enough, the prompts need more detail, or the tooling isn't mature enough. These are all implementation-layer fixes, and they don't address the actual cause.

The joins fail because nobody designed what should happen there. A smarter model won't invent a handoff contract that was never defined. A longer prompt won't create a clear role boundary where one wasn't drawn. Better tooling won't scope access per-agent if nobody decided which agent should own what.

The structure of the team — who does what, who hands off to whom, what data each agent can touch, where a human needs to be in the loop — is a design question. It has to be answered before any code runs, not worked out by patching a running system.

This is also why the choice of platform matters early. The platform trap describes the pattern of building bottom-up around available tools and discovering the design gaps later. Composable AI is the alternative framing: design the system first, then choose the components that fit it.

What does upfront design actually look like?

A properly designed multi-agent team has the same kind of clarity you'd expect from a well-run human team:

Defined roles. Each agent has a written role — what it's responsible for and, explicitly, what it isn't. A Researcher is not the Writer. A Triage agent doesn't also compose responses. Overlapping authority is removed at the design stage, not debugged in production.

Explicit handoffs. Every connection between agents specifies what the upstream agent produces and what the downstream agent expects. Format, structure, and the definition of "done" are agreed before either agent is built.

Scoped access. Each agent is given exactly the tool access its role requires and no more. Access is assigned per-agent, not shared across the team by default.

Pipeline shape. The order agents run is based on real dependencies — what genuinely needs to happen before something else can start — and not just "the order we happened to build them in." Steps that don't depend on each other run concurrently. Steps that do have explicit dependencies between them.

Gates placed deliberately. The pipeline identifies which steps could cause irreversible harm or require human judgment before proceeding. Those get a gate — a real pause, not a log entry — with a defined timeout and clear on-reject behaviour.

This isn't documentation-as-afterthought. It's the specification that makes the implementation possible in the first place.

Is there a way to tell if a system has the stitching problem?

A few patterns are reliable signals:

Debugging requires reading every agent's logs linearly. In a well-designed team, you can trace an output back to the step and agent that produced it. If debugging means reading the entire execution history to piece together what happened, the observability was never designed in.

Adding a new agent breaks something unexpected. When the team structure isn't explicit, adding a new agent creates new role ambiguity or a new dependency that wasn't obvious until it collided with the existing pipeline.

Failures happen "at the joins." Each agent reports success when run in isolation. Problems only appear when they run together — which means the problem is in the interface between agents, not in any individual agent. This is the stitching problem, and it only gets worse as the team scales.

Human oversight is added after the fact. If the current plan is "we'll review the output after it runs," that's a plan that doesn't scale, and one that doesn't catch irreversible errors before they happen.

Where Envelope fits

Envelope is the design step — the part that happens before any code runs. You describe the team you want in natural language, and Envelope designs the structure: sub-agents with clear roles, a pipeline with the right steps in the right order, gates placed at the points where they actually matter, and access scoped per-agent.

The output is a spec, not a prototype. Something you can hand to an engineer or a platform and have them implement correctly, because the design decisions were made before anyone started building. The spec conforms to an open standard — not a proprietary format locked to a single runtime.

If you want to understand the structure the spec captures — roles, models, pipelines, gates — the full anatomy is in The anatomy of a multi-agent. The specific design decisions around gates and human oversight are covered in more depth in Human in the loop and the autonomous agent problem.

Design your AI agents in Envelope

Envelope turns a plain-language description of your workflow into a complete AI agent system — agents with named roles, model assignments, tool access, handoffs, and human review gates. Free to start, no code required.

Start designing →

Frequently asked questions

What is the stitching problem in multi-agent AI?

The stitching problem is what happens when a multi-agent system is built by connecting agents one at a time, without designing the team structure first. Each agent works individually, but the joins between them fail — because roles, handoffs, tool boundaries, and ownership were never defined.

Why do most multi-agent systems fail?

Most multi-agent systems fail at the connections between agents, not inside them. The causes are predictable: overlapping instructions, undefined handoffs, unscoped tool access, and no gates for human oversight. These are design problems — they can't be fixed by better models or longer prompts.

How do you fix the stitching problem?

The fix is a design step before any implementation begins. Define each agent's role and what it explicitly doesn't own. Specify the handoff between every pair of connected agents. Scope tool access per-agent. Map the pipeline shape based on real dependencies. Place gates where irreversible actions or human judgment are required.

Can better AI models prevent the stitching problem?

No. Smarter models don't invent handoff contracts that were never defined, or create role boundaries that were never drawn. The stitching problem is structural — it's a failure to design the team before building it. Model capability is irrelevant until the structure is right.

What is a gate in a multi-agent pipeline?

A gate is a checkpoint where the pipeline stops and waits for a human decision before continuing. It's placed deliberately before a consequential, potentially irreversible step — not added as a review mechanism after something goes wrong. A properly designed gate includes a timeout and defined on-reject behaviour.

How do you know if your multi-agent system has the stitching problem?

Common signals: debugging requires reading every agent's execution log linearly; adding a new agent breaks something unexpected elsewhere; each agent works in isolation but failures appear when they run together; human oversight is a manual review step after the pipeline runs rather than a designed checkpoint within it.

What should a multi-agent design spec include?

A complete design spec defines every sub-agent's role and capabilities, the pipeline order and concurrency model, handoff contracts between agents, access scope per-agent, and gate placement with timeout and rejection handling. It's the blueprint that makes the build possible — not documentation written afterwards.