EQA-RM: A Generative Embodied Reward Model with Test-time Scaling

TL;DR

EQA-RM is a novel generative reward model designed for embodied question answering tasks, offering interpretable feedback and test-time scaling, with high efficiency and strong performance on a new benchmark.

Contribution

The paper introduces EQA-RM, a generative multimodal reward model for EQA, trained with C-GRPO, and presents EQARewardBench for standardized evaluation.

Findings

EQA-RM achieves 61.9% accuracy with only 700 samples.

EQA-RM outperforms proprietary and open-source baselines.

Test-time scaling enables dynamic evaluation granularity.

Abstract

Reward Models (RMs), vital for large model alignment, are underexplored for complex embodied tasks like Embodied Question Answering (EQA) where nuanced evaluation of agents' spatial, temporal, and logical understanding is critical yet not considered by generic approaches. We introduce EQA-RM, a novel generative multimodal reward model specifically architected for EQA, trained via our innovative Contrastive Group Relative Policy Optimization (C-GRPO) strategy to learn fine-grained behavioral distinctions. The generative nature of EQA-RM provides interpretable, structured reward feedback (beyond simple scalars), uniquely enabling test-time scaling to dynamically adjust evaluation granularity, from concise scores to detailed critiques of reasoning and grounding, at inference without retraining. Concurrently, we introduce EQARewardBench, a new benchmark built on OpenEQA for standardized EQA reward model assessment. Demonstrating high sample efficiency, EQA-RM (fine-tuning Qwen2-VL-2B-Instruct) achieves 61.9\% accuracy on EQA-RM-Bench with only 700 samples, outperforming strong proprietary baselines, including Gemini-2.5-Flash, GPT-4o, Claude-3.5-Haiku, and open-sourced state-of-the-art models such as RoVRM and VisualPRM. The code and dataset can be found here https://github.com/UNITES-Lab/EQA-RM.