You must bake a cake before decorating it, but you can prepare decorations while the cake is baking. Dependencies—not speed alone—decide whether work can run in parallel.
Sequential: A → B → C
Parallel: A → B and C together → joinWhat You Will Learn
- How to identify real dependencies between subtasks.
- How parallel execution changes latency, cost, and coordination.
- How race conditions and stale state silently corrupt results.
Every pattern covered in Modules 7 through 9 has to answer one more question underneath the topology: does this specific work happen in order, or at the same time?
User Request
↓
Planner
↓
┌────────────┼────────────┐
↓ ↓ ↓
Search Analyze Validate
↓ ↓ ↓
└────────────┼────────────┘
↓
SynthesizerThis looks like a simple fan-out. Whether it’s actually correct depends entirely on one thing: do Search, Analyze, and Validate have no dependency on each other’s results?
What each execution model actually looks like, step by step
Sequential:
Planner
↓
Search completes fully
↓
Analyze starts, using Search's output
↓
Validate starts, using Analyze's output
↓
SynthesizerParallel:
Planner
↓
Search, Analyze, and Validate all start simultaneously
↓
(no agent waits on another — each works from the original request alone)
↓
Synthesizer waits for all three, then combinesThe structural difference is exactly one thing: in sequential execution, each stage’s input includes the previous stage’s output. In parallel execution, every stage’s input is limited to what was available before any of them started — which is precisely why dependency between stages is the one fact that determines which of these two diagrams is actually correct for a given task, not a stylistic preference between them.
Dependencies decide this, not preference
Module 4 already introduced this distinction briefly. It’s worth being precise and complete about it here, because getting it wrong in either direction has a real, measurable cost.
independent work — three subtasks that don’t need each other’s output to proceed — is safe to parallelize. Running them sequentially anyway wastes real wall-clock time for no benefit.
dependent work — where one subtask’s output is a required input to another — cannot be safely parallelized, no matter how tempting the speed gain looks. Forcing it into parallel execution doesn’t just fail to help; it produces a race condition, covered in depth below.
The mistake worth naming directly: assuming a task is parallelizable because it looks like three separate steps, without actually checking whether any of them secretly depends on another’s result.
The real speed and cost trade-off
Parallel execution’s benefit is wall-clock time — three subtasks running simultaneously finish in roughly the time of the slowest one, not the sum of all three. Its cost is different from sequential execution’s cost, not simply larger or smaller.
Production cost analysis frames this precisely: parallel fan-out patterns have the highest peak cost, because every branch fires simultaneously and consumes resources at the same moment, but the lowest wall-clock time. Sequential pipelines spread the same total cost out over a longer period, with lower peak resource demand at any given instant.
This matters for real infrastructure planning, not just an abstract trade-off — a system provisioned for sequential execution’s steady, moderate load can be overwhelmed by the same total workload arriving as a parallel burst instead.
Race conditions: the real mechanics
This is worth defining precisely before discussing what to do about it. The formal, industry-standard definition: a race condition occurs when two or more processes access a shared resource concurrently, at least one access is a write, and the outcome depends on the order those accesses happen to occur in. (CWE-362, MITRE)
Translated into agent terms, and stated as plainly as the risk deserves:
“In multi-agent systems, where parallel execution is the whole point, race conditions aren’t edge cases. They’re expected guests. Understanding how to handle them is less about being defensive and more about building systems that assume chaos by default.” — MachineLearningMastery, Handling Race Conditions in Multi-Agent Orchestration
Why this is more dangerous than it sounds
Here’s the part that makes this worth real attention rather than a passing warning: race conditions in agent systems frequently don’t crash anything or throw a visible error. They corrupt data silently, and the corruption looks exactly like a model reasoning error instead of what it actually is — a concurrency bug. (TianPan.co, The Silent Corruption Problem in Parallel Agent Systems)
A concrete, worked example makes this vivid: “Agent A reads a document, Agent B updates it half a second later, and Agent A writes back a stale version with no error thrown anywhere. The system looks fine. The data is compromised.” (MachineLearningMastery)
Read that sequence again slowly. Nothing failed. No exception was thrown. No log entry flagged anything unusual. And the shared document is now wrong, with the only trace of what happened being the timing of two writes that, on any other run, might have happened in the opposite order and produced a correct result.
This is precisely why Module 6’s “cognitive debugging” instinct — asking “why did the agent fail to reason” — can actively mislead you here. The agent didn’t fail to reason. The infrastructure underneath it failed to synchronize.
A second, common real scenario: a shared ticket queue, multiple agents pulling from it without coordination. Two agents claim the same ticket simultaneously, and the customer receives two separate, possibly conflicting responses. (MachineLearningMastery)
Real mitigations, with real trade-offs
None of these are exotic — they’re borrowed directly from distributed systems engineering, applied to agent-specific shared state.
- Locking. A resource is claimed exclusively while one agent works with it, blocking others until it’s released. This guarantees correctness. The real cost: if many agents compete for the same lock, throughput drops quickly — you’re trading back some of parallelism’s speed benefit specifically to buy back correctness.
- Atomic updates. Rather than a read-modify-write sequence an agent performs itself (exactly where the stale-write example above went wrong), the update is delegated to infrastructure that guarantees it as one indivisible operation — a database or key-value store’s native atomic operations. This removes the race entirely rather than managing around it.
- Optimistic locking via versioning. Instead of blocking access upfront, each write includes a check against the version it read — if that version has since changed, the write is rejected and the agent retries against current data. This preserves more parallelism than locking, at the cost of occasional wasted work when a conflict is actually detected.
A more sophisticated real answer: transactional tool use
Current research goes further than ad-hoc locking, addressing a harder version of this problem: agents don’t just read and write shared data, they call tools that mutate real external state — editing code, updating databases, sending emails. A 2026 paper frames the core question directly: *“when may a tool effect become permanent?”
Their answer combines tool-level idempotency keys — directly Module 6’s idempotency guard, now named precisely — with Saga-style compensations, a distributed-systems pattern where a multi-step action that partially completes can be undone through a defined, reverse sequence of compensating actions, rather than left in an inconsistent, half-finished state. (Atomix: Timely, Transactional Tool Use for Reliable Agentic Workflows, arXiv)
This matters specifically for parallel agents whose actions have real, external, hard-to-reverse consequences — sending an email can’t be “rolled back” the way a database write can, which is exactly why this research treats the question of permanence as a first-class design problem, not an afterthought.
The subtler cost: parallelism can waste effort even without a race
It’s worth knowing a more nuanced problem than data corruption, because it applies even when your synchronization is technically correct.
Consider three agents simultaneously investigating a production incident — one reading logs, one checking deployment history, one correlating recent code changes. The log agent finds a anomaly at a specific timestamp. That timestamp is load-bearing — it would dramatically narrow what the other two agents actually need to search. But under pure parallel execution, that finding stays locked inside the log agent’s own context until all three finish and the orchestrator collates everything.
The deployment and code-change agents keep searching the entire space the whole time, wasting effort, and potentially missing the connection the log agent’s finding would have pointed them toward directly. (Medium, Parallel Agents Are Just Multithreading)
This is worth holding as a separate lesson from the race-condition risk above. Pure, uncoordinated parallelism isn’t just a correctness risk when agents share mutable state — it’s an efficiency loss even when nothing is technically wrong, because early results that would help other agents work smarter simply aren’t shared until everyone is already done.
The practical implication: a hybrid approach — parallel execution with a lightweight, periodic checkpoint where agents can surface load-bearing findings early — often outperforms either pure sequential or pure parallel execution for tasks with this shape, even though it doesn’t fit cleanly into either category.
A decision framework
- Are the subtasks independent? If any of them needs another’s output, that’s a hard dependency — sequential execution for that specific pair, regardless of what the rest of the pipeline does.
- Does a subtask mutate shared state? If yes, parallelizing it requires one of this module’s real mitigations — locking, atomic updates, or optimistic versioning — not just running it alongside others and hoping for the best.
- Would an early result from one subtask help another? If yes, consider a hybrid checkpoint approach rather than pure parallel fan-out, even when the subtasks are technically independent enough to run simultaneously.
- Does peak resource cost matter more than wall-clock time, or the reverse? This is a infrastructure question, not just a task-shape question — the same logical parallelism can be the right or wrong choice depending on what your system can actually absorb at once.
None of these four questions has a universal answer that transfers cleanly from one task to the next — that’s precisely why this module treats them as a framework to apply deliberately, not a rule to memorize once and reuse without checking it against the specific task in front of you.
Applying this to the recurring scenario
The legal-contract checklist items — payment terms, liability, termination — are independent of each other in content, which is exactly why Module 7 and 8 treated parallel Executor runs as safe. It’s worth checking this module’s specific risks against that assumption honestly.
Shared state risk: if every Executor writes its comparison directly into one shared report document rather than its own isolated output, that’s exactly the stale-write pattern this module described — two Executors finishing close together could overwrite each other’s contribution. The fix is structural, not aspirational: each Executor writes to its own designated slot in a shared schema (Module 6’s structured aggregation), never to a shared mutable document directly.
The subtler cost: if the Payment Terms Executor discovers the contract references an unusual, non-standard payment structure, that finding might be relevant to how the Termination Executor should interpret an early-termination refund clause. Pure parallel execution means neither Executor benefits from the other’s finding until the Critic reviews both afterward — precisely the load-bearing-information problem this module described, now costing the pipeline a potentially missed connection between two related clauses.
A disciplined version of this pipeline would give the Planner a lightweight mid-run checkpoint — not full synchronization, just a single point where an Executor can flag “this finding may be relevant to another checklist item” before continuing. That’s a real, deliberate design choice, not a default behavior any orchestration framework provides automatically. Skipping it doesn’t break the pipeline. It just means the system occasionally misses a connection a more coordinated version would have caught, without ever surfacing that gap as a visible error anyone would notice.
Interview-relevant framing
Q: How would you diagnose whether a bug in a multi-agent system is a race condition or a reasoning error?
Ans: I’d look at whether the failure is reproducible with the same inputs. A reasoning error tends to happen consistently given the same context — the model makes the same mistake again. A race condition often doesn’t reproduce cleanly, because it depends on the timing of concurrent writes, which can vary between runs even with identical inputs. If a bug seems to appear and disappear unpredictably on the same test case, that’s a real signal to look at shared-state access patterns before assuming it’s a model quality issue.
Q: When would you choose a hybrid execution model over pure sequential or pure parallel?
Ans: When subtasks are independent enough to run in parallel, but an early result from one would meaningfully help another do its job better or faster. Pure parallel execution in that case is technically safe but wastes real effort — each agent searches its full space blind to what a peer has already found. A lightweight checkpoint where agents can surface load-bearing findings mid-execution often beats both pure approaches for tasks with cross-subtask relevance.
A third question worth preparing for:
Q: A teammate wants to parallelize a pipeline stage that writes to a shared report document. What would you check before approving that change?
Ans: First, whether that stage has no dependency on another stage’s output — if it does, parallelizing it is wrong regardless of the shared-write question. Assuming that’s clear, I’d check exactly how the write happens: if multiple agents write directly to the same mutable document, that’s the stale-write pattern that produces silent corruption with no error thrown. I’d want each agent writing to its own isolated slot in a structured schema instead, or a atomic update at the database level — not a read-modify-write sequence performed by the agent itself.
Common Misconception
Incorrect idea: Independent-looking tasks are always safe to run in parallel.
Why it is incorrect: Tasks may read or modify the same hidden state. Parallel execution is safe only after dependencies and shared writes are understood.
Key takeaways
- Whether work can be parallelized is decided entirely by dependency, not by how the task happens to be described — three steps that look separate can still secretly depend on each other’s output.
- Parallel execution trades lower wall-clock time for higher peak resource cost; sequential execution spreads the same total cost over more time at lower peak demand — a infrastructure planning question, not just a speed question.
- Race conditions in agent systems are frequently silent — no crash, no error, just quietly corrupted shared state that looks exactly like a model reasoning mistake rather than the concurrency bug it actually is.
- Real mitigations — locking, atomic updates, optimistic versioning, and Saga-style compensations for tool effects that mutate real external state — are borrowed directly from distributed systems engineering, each with a different trade-off.
- Pure parallel execution has a subtler cost even without any race condition: a load-bearing finding from one agent stays siloed until everyone finishes, wasting real effort other agents could have avoided with earlier access to it.
- A hybrid model — parallelism with a lightweight checkpoint for surfacing early findings — often outperforms both pure sequential and pure parallel execution for tasks where subtasks are independent but related.
Module 11 covers a concern that applies across every pattern and execution model covered so far: agent routing — how a system dynamically decides which specific agent, or which entire pattern, a given request should actually go to, and how that decision itself needs the same rigor this module just applied to execution order.
- Author
- TechByteByByte Editorial Team
- Reviewed by
- TechByteByByte Admin
- Published
- Last reviewed