Perception Test 2025: Challenge Summary and a Unified VQA Extension

TL;DR

Perception Test 2025 benchmarked multimodal video models through unified perception tasks, emphasizing task unification challenges and introducing novel video QA formats to evaluate current state-of-the-art capabilities.

Contribution

It introduces a unified benchmarking framework for diverse perception tasks, reformulating traditional tasks as multiple-choice video QA to test model versatility.

Findings

Unified video QA reformulates perception tasks as multiple-choice questions.

Models face significant challenges when tackling diverse perception tasks.

The challenge highlights the need for more versatile multimodal models.

Abstract

The Third Perception Test challenge was organised as a full-day workshop alongside the IEEE/CVF International Conference on Computer Vision (ICCV) 2025. Its primary goal is to benchmark state-of-the-art video models and measure the progress in multimodal perception. This year, the workshop featured 2 guest tracks as well: KiVA (an image understanding challenge) and Physic-IQ (a video generation challenge). In this report, we summarise the results from the main Perception Test challenge, detailing both the existing tasks as well as novel additions to the benchmark. In this iteration, we placed an emphasis on task unification, as this poses a more challenging test for current SOTA multimodal models. The challenge included five consolidated tracks: unified video QA, unified object and point tracking, unified action and sound localisation, grounded video QA, and hour-long video QA, alongside an analysis and interpretability track that is still open for submissions. Notably, the unified video QA track introduced a novel subset that reformulates traditional perception tasks (such as point tracking and temporal action localisation) as multiple-choice video QA questions that video-language models can natively tackle. The unified object and point tracking merged the original object tracking and point tracking tasks, whereas the unified action and sound localisation merged the original temporal action localisation and temporal sound localisation tracks. Accordingly, we required competitors to use unified approaches rather than engineered pipelines with task-specific models. By proposing such a unified challenge, Perception Test 2025 highlights the significant difficulties existing models face when tackling diverse perception tasks through unified interfaces.