IdleSpec: Exploiting Idle Time via Speculative Planning for LLM Agents

TL;DR

IdleSpec is a scalable inference method that exploits idle time in LLM agents to generate and refine plans, significantly enhancing performance across various complex tasks.

Contribution

It introduces a novel approach that utilizes idle periods for speculative planning, improving agent efficiency and accuracy without increasing latency.

Findings

IdleSpec improves agent accuracy by up to 55.6% on GAIA and FRAMES.

It achieves up to 9.1% performance gains on MLE-Bench.

The method effectively utilizes idle time for long-horizon tasks.

Abstract

Large language model (LLM)-based agents solve complex tasks by leveraging multi-step reasoning with iterative tool calls and environment interactions, which incur idle time while waiting for observations. Despite the prevalence of idle time in most agentic scenarios, existing works treat it as an unavoidable overhead or propose restricted solutions that overlook varying computational budgets across different tool calls and future observation uncertainty, thereby leading to suboptimal utilization of idle time. In this paper, we introduce IdleSpec, a scalable and generic inference approach that leverages idle-time computation to improve agent performance while minimizing latency overhead. Specifically, IdleSpec iteratively generates plan candidates during idle periods and, once observations become available, aggregates them to guide the next reasoning step. For effective plan generation under observation uncertainty, IdleSpec samples between complementary drafting strategies (i.e., progressive and recovery) from a learned distribution that is updated via posterior feedback. Our experiments demonstrate that IdleSpec significantly improves agent performance in various agentic scenarios by effectively utilizing idle time. In particular, on the GAIA and FRAMES, IdleSpec achieves 55.6% average accuracy with Gemini-2.5-Flash, surpassing the vanilla baseline without idle-time usage by 5.1%. Furthermore, for MLE-Bench, which involves substantial delay from code executions, IdleSpec achieves performance gains of up to 9.1% on the Any Medal rate, highlighting its generalizability to long-horizon tasks.