Learning to Hand Off: Provably Convergent Workflow Learning under Interface Constraints

TL;DR

This paper introduces a provably convergent decentralized Q-learning algorithm for workflow learning in multi-agent systems with interface constraints, providing finite-sample guarantees and validating through diverse experiments.

Contribution

It formalizes the interface-constrained semi-Markov decision process and develops IC-Q, the first finite-sample neural Q-learning guarantee for decentralized partial observability.

Findings

IC-Q matches centralized oracle performance without joint trajectory access.

The finite-sample bound decomposes into neural approximation, interface gap, and mixing-time errors.

Experiments validate the theoretical error scaling and effectiveness across multiple tasks.

Abstract

We study workflow learning in a setting where specialized agents hand off control through a shared artifact, each agent observes only a local function of that artifact and its own private state, and no centralized learner accesses joint trajectories -- the operating regime of multi-agent LLM pipelines that span organizational, vendor, or trust boundaries. We formalize this regime as an interface-constrained semi-Markov decision process (IC-SMDP), whose decision epochs occur at handoff times, and design IC-$Q$, an asynchronous decentralized $Q$-learning algorithm in which cross-agent coordination at every handoff is exactly one scalar. Our main result is a finite-sample bound for neural IC-$Q$ that decomposes into three independently controllable error sources: neural function-approximation error, interface representation gap, and a mixing-time residual, under the random option-duration discount. Establishing this bound requires lifting the approximate information state (AIS) framework from single-agent primitive-step MDPs to multi-agent SMDPs and controlling Markovian noise under random duration, neither of which has been done in prior work. To our knowledge this is the first finite-sample guarantee for neural $Q$-learning under decentralized partial observability. Four experiments: a controlled synthetic IC-SMDP that validates the bound term-by-term, multi-LLM mathematical reasoning, multi-agent routing, and multi-agent CPU programming, show that IC-$Q$ matches a centralized oracle without any agent observing joint trajectories, with each of the three error sources scaling along its corresponding axis as the bound predicts.