PuzzleCraft: Exploration-Aware Curriculum Learning for Puzzle-Based RLVR in VLMs

TL;DR

PuzzleCraft introduces an exploration-aware curriculum for puzzle-based RLVR in vision-language models, improving reasoning consistency and robustness without supervision by leveraging puzzle environments and a new metric.

Contribution

It presents PuzzleCraft, a supervision-free framework that scales puzzle-based RLVR with a novel curriculum considering difficulty and exploration, and introduces RAC for measuring reasoning-answer consistency.

Findings

Improves RAC and downstream performance on vision benchmarks.

Enhances robustness and reasoning consistency in VLMs.

Achieves consistent gains on Qwen2.5-VL and Qwen3-VL models.

Abstract

RL post-training with verifiable rewards (RLVR) has become a practical route to eliciting chain-of-thought reasoning in vision--language models (VLMs), but scaling it in the visual domain remains challenging due to costly or noisy supervision and reliance on external verifiers. Puzzle-based RLVR is a promising alternative, yet existing approaches often treat puzzle rewards as flat or sparse, which weakens group-relative learning signal. Existing curriculum strategies are overly restrictive: they rely mainly on reward statistics and do not account for exploration in the solution space, which can lead to collapsed rollout dynamics. Further, RL post-training can induce reasoning--answer inconsistency as training progresses. To address these shortcomings, we present PuzzleCraft, a supervision-free framework that scales vision-centric RLVR using a set of lightweight puzzle environments with built-in verification. PuzzleCraft instantiates three puzzles inspired by classic visual pretext tasks: PatchFit, Rotation, and Jigsaw. We introduce a curriculum that combines difficulty with an exploration signal derived from solution-space dispersion, and use it to downweight collapsed prompt groups. In addition, we introduce a new post-training metric, Reasoning-Answer Consistency (RAC), to measure the degree that the chain-of-though supports the answer, and show our exploration-aware curriculum improves RAC and downstream performance. Across a broad suite of vision-centric benchmarks, PuzzleCraft improves robustness and reasoning consistency, yielding consistent downstream gains on both Qwen2.5-VL and Qwen3-VL backbones. Overall, our results suggest that scalable puzzle-based RLVR benefits from curricula that account for both difficulty and solution-space collapse, together with explicit consistency-enhancing schemes.