CORE: Code-based Inverse Self-Training Framework with Graph Expansion for Virtual Agents

TL;DR

CORE introduces a novel training framework for virtual agents that combines imitation and exploration by automatically inferring reward functions and expanding behavioral strategies, leading to improved performance and generalization.

Contribution

The paper proposes CORE, a code-based inverse self-training framework with graph expansion, integrating semantic code abstraction, strategy graph expansion, and trajectory-guided extrapolation to enhance behavioral diversity without manual reward design.

Findings

Significantly improves performance on Web and Android platforms.

Enhances behavioral diversity through strategy graph expansion.

Achieves better generalization in virtual agent training.

Abstract

The development of Multimodal Virtual Agents has made significant progress through the integration of Multimodal Large Language Models. However, mainstream training paradigms face key challenges: Behavior Cloning is simple and effective through imitation but suffers from low behavioral diversity, while Reinforcement Learning is capable of discovering novel strategies through exploration but heavily relies on manually designed reward functions. To address the conflict between these two methods, we present CORE, a Code-based Inverse Self-Training Framework with Graph Expansion that bridges imitation and exploration, offering a novel training framework that promotes behavioral diversity while eliminating the reliance on manually reward design. Specifically, we introduce Semantic Code Abstraction to automatically infers reward functions from expert demonstrations without manual design. The inferred reward function, referred to as the Label Function, is executable code that verifies one key step within a task. Building on this, we propose Strategy Graph Expansion to enhance in-domain behavioral diversity, which constructs a multi-path graph called Strategy Graph that captures diverse valid solutions beyond expert demonstrations. Furthermore, we introduce Trajectory-Guided Extrapolation, which enriches out-of-domain behavioral diversity by utilizing both successful and failed trajectories to expand the task space. Experiments on Web and Android platforms demonstrate that CORE significantly improves both overall performance and generalization, highlighting its potential as a robust and generalizable training paradigm for building powerful virtual agents.