OrchestraBench presents a novel evaluation framework for multi-agent orchestration, emphasizing the quantification of failure modes, recovery abilities, and decomposition quality. Unlike traditional benchmarks that only measure task accuracy, OrchestraBench probes the root causes of failure within orchestration pipelines, offering a systematic approach to understanding where and why breakdowns occur.
The framework utilizes a controlled failure-injection harness capable of seed-reproducible outcomes with templated enterprise workflows. It introduces two key metrics: cascade radius and per-failure-mode recovery. A comparative analysis of various routing policies revealed significant discrepancies in performance. Specifically, a keyword/flag router failed entirely on adversarial cases with misleading or absent surface flags, achieving a score of 0%. In contrast, an intent-reasoning model router successfully matched all gold-labels, achieving a perfect score of 100%.
Further investigation involved probes using the Claude agent across defined arithmetic dependency chains, uncovering three tiers of failure handling across five multi-agent system tasks. Tool faults demonstrated complete recovery (1.0), while ambiguities in delegation led to partial recovery (0.30), and three categories of latent or semantic faults indicated no recovery (0.0). These observations remained consistent even when altering the context to scenarios like loan approvals, although the overall recovery rates varied.
The findings suggest that common strategies such as blind retries may exacerbate fault detection times, reinforcing the need for improved detection and attribution mechanisms to enhance operational containment. Notably, the cascade radius, reflecting the impact of pipeline depth on fault propagation, increased from an average of 0.9 to 4.7 as depth expanded from 3 to 7, indicating the complexities inherent in deeper orchestration structures. A trusted-state repair analysis revealed that most containment improvements greatly depended on trusted-state signals rather than on autonomous detection capabilities.