Structured Preference Optimization for Vision-Language Long-Horizon Task Planning

TL;DR

This paper introduces Structured Preference Optimization (SPO), a novel approach that enhances vision-language long-horizon task planning by improving reasoning quality and decision accuracy through preference evaluation and curriculum training.

Contribution

The paper proposes SPO, a new method for long-horizon vision-language planning, and introduces ExtendaBench, a comprehensive benchmark for evaluating such tasks.

Findings

SPO outperforms prior methods on long-horizon tasks.

SPO achieves approximately 6% GCR improvement in VirtualHome.

SPO achieves over 2% SR improvement in Habitat.

Abstract

Existing methods for vision-language task planning excel in short-horizon tasks but often fall short in complex, long-horizon planning within dynamic environments. These challenges primarily arise from the difficulty of effectively training models to produce high-quality reasoning processes for long-horizon tasks. To address this, we propose Structured Preference Optimization (SPO), which aims to enhance reasoning and action selection in long-horizon task planning through structured preference evaluation and optimized training strategies. Specifically, SPO introduces: 1) Preference-Based Scoring and Optimization, which systematically evaluates reasoning chains based on task relevance, visual grounding, and historical consistency; and 2) Curriculum-Guided Training, where the model progressively adapts from simple to complex tasks, improving its generalization ability in long-horizon scenarios and enhancing reasoning robustness. To advance research in vision-language long-horizon task planning, we introduce ExtendaBench, a comprehensive benchmark covering 1,509 tasks across VirtualHome and Habitat 2.0, categorized into ultra-short, short, medium, and long tasks. Experimental results demonstrate that SPO significantly improves reasoning quality and final decision accuracy, outperforming prior methods on long-horizon tasks and underscoring the effectiveness of preference-driven optimization in vision-language task planning. Specifically, SPO achieves a +5.98% GCR and +4.68% SR improvement in VirtualHome and a +3.30% GCR and +2.11% SR improvement in Habitat over the best-performing baselines.