Reasoning Resides in Layers: Restoring Temporal Reasoning in Video-Language Models with Layer-Selective Merging

TL;DR

MERIT is a training-free framework that restores temporal reasoning in video-language models by layer-wise merging, improving reasoning without retraining and maintaining perceptual abilities.

Contribution

Introduces MERIT, a novel layer-merging method that enhances temporal reasoning in VLMs while preserving perception, without requiring retraining.

Findings

MERIT improves temporal reasoning across multiple VLMs and benchmarks.

MERIT preserves or enhances temporal perception in merged models.

Layer selection is crucial for effective reasoning restoration.

Abstract

Multimodal adaptation equips large language models (LLMs) with perceptual capabilities, but often weakens the reasoning ability inherited from language-only pretraining. This trade-off is especially pronounced in video-language models (VLMs), where visual alignment can impair temporal reasoning (TR) over sequential events. We propose MERIT, a training-free, task-driven model merging framework for restoring TR in VLMs. MERIT searches over layer-wise self-attention merging recipes between a VLM and its paired text-only backbone using an objective that improves TR while penalizing degradation in temporal perception (TP). Across three representative VLMs and multiple challenging video benchmarks, MERIT consistently improves TR, preserves or improves TP, and generalizes beyond the search set to four distinct benchmarks. It also outperforms uniform full-model merging and random layer selection, showing that effective recovery depends on selecting the right layers. Interventional masking and frame-level attribution further show that the selected layers are disproportionately important for reasoning and shift model decisions toward temporally and causally relevant evidence. These results show that targeted, perception-aware model merging can effectively restore TR in VLMs without retraining.