MARS: Technical Report for the CASTLE Challenge at EgoVis 2026

TL;DR

MARS is a multimodal reasoning system designed for the CASTLE Challenge, integrating diverse sources like videos, transcripts, and auxiliary data to answer complex questions over multiple days.

Contribution

The paper introduces MARS, a novel agentic evidence-selection approach that effectively handles multimodal data for complex reasoning tasks in egocentric benchmarks.

Findings

Achieved second place on the CASTLE Challenge leaderboard.

Effectively integrates multiple modalities including videos, transcripts, gaze, and thermal imagery.

Uses GPT-5.4 as a decision agent for evidence selection and reasoning.

Abstract

This report presents MARS, short for Multimodal Agentic Reasoning with Source selection, our system for the CASTLE Challenge at EgoVis 2026. Participants must answer 185 closed-form questions over the CASTLE 2024 dataset. In contrast to prior single-video egocentric benchmarks, CASTLE requires reasoning over four days of activity, 15 synchronized perspectives, official transcripts, and multiple auxiliary modalities, including personal photos, auxiliary videos, gaze, thermal imagery, and heartrate measurements. MARS therefore treats the task as an agentic evidence-selection problem over multimodal sources rather than a purely text-only pipeline. MARS first follows the official CASTLE directory organization to build evidence memories from two primary sources, videos and transcripts, and four auxiliary sources, gaze, heartrate, photos, and thermal imagery. Long videos are converted into captions and DeepSeek-based summaries only because CASTLE videos are too long to fit directly into the model context for every question; this step compresses temporal evidence while keeping photos and other auxiliary media available as source-specific evidence. At inference time, a GPT-5.4 decision agent repeatedly chooses whether to continue reasoning, request a specific missing modality, produce an answer, or fall back to a random option when the evidence remains insufficient. The resulting system achieved second place on the final CASTLE Challenge leaderboard. Our codes are available at https://github.com/Hyu-Zhang/MARS.