TVCACHE: A Stateful Tool-Value Cache for Post-Training LLM Agents

TL;DR

TVCACHE is a novel stateful cache for post-training LLM agents that maintains sequence trees to ensure correct reuse of tool outputs, significantly reducing tool call times without harming reward performance.

Contribution

It introduces a tree-based, state-aware caching mechanism that guarantees correctness and efficiency in reusing tool outputs during LLM agent post-training.

Findings

Achieves up to 70% cache hit rate

Reduces median tool call time by up to 6.9X

No degradation in reward accumulation

Abstract

In RL post-training of LLM agents, calls to external tools take several seconds or even minutes, leaving allocated GPUs idle and inflating post-training time and cost. While many tool invocations repeat across parallel rollouts and could in principle be cached, naively caching their outputs for reuse is incorrect since tool outputs depend on the environment state induced by prior agent interactions. We present TVCACHE, a stateful tool-value cache for LLM agent post-training. TVCACHE maintains a tree of observed tool-call sequences and performs longest-prefix matching for cache lookups: a hit occurs only when the agent's full tool history matches a previously executed sequence, guaranteeing identical environment state. On three diverse workloads-terminal-based tasks, SQL generation, and video understanding. TVCACHE achieves cache hit rates of up to 70% and reduces median tool call execution time by up to 6.9X, with no degradation in post-training reward accumulation.