Start Small, Think Big: Curriculum-based Relative Policy Optimization for Visual Grounding

TL;DR

This paper introduces CuRPO, a curriculum-based training method for visual grounding that uses reasoning complexity indicators to improve performance and robustness, especially with complex data and limited samples.

Contribution

We propose CuRPO, a novel curriculum learning strategy that leverages CoT length and gIoU rewards to progressively train models from simple to complex examples in visual grounding.

Findings

CuRPO outperforms existing methods with up to +12.52 mAP improvement.

It enhances robustness and efficiency in few-shot learning scenarios.

The approach effectively handles complex and ambiguous textual descriptions.

Abstract

Chain-of-Thought (CoT) prompting has recently shown significant promise across various NLP and computer vision tasks by explicitly generating intermediate reasoning steps. However, we find that reinforcement learning (RL)-based fine-tuned CoT reasoning can paradoxically degrade performance in Visual Grounding tasks, particularly as CoT outputs become lengthy or complex. Additionally, our analysis reveals that increased dataset size does not always enhance performance due to varying data complexities. Motivated by these findings, we propose Curriculum-based Relative Policy Optimization (CuRPO), a novel training strategy that leverages CoT length and generalized Intersection over Union (gIoU) rewards as complexity indicators to progressively structure training data from simpler to more challenging examples. Extensive experiments on RefCOCO, RefCOCO+, RefCOCOg, and LISA datasets demonstrate the effectiveness of our approach. CuRPO consistently outperforms existing methods, including Visual-RFT, with notable improvements of up to +12.52 mAP on RefCOCO. Moreover, CuRPO exhibits exceptional efficiency and robustness, delivering strong localization performance even in few-shot learning scenarios, particularly benefiting tasks characterized by ambiguous and intricate textual descriptions.The code is released on https://github.com/qyoung-yan/CuRPO.