TemporalDoRA: Temporal PEFT for Robust Surgical Video Question Answering

TL;DR

TemporalDoRA introduces a temporally-aware PEFT method for surgical VideoQA, enhancing robustness to linguistic variation and improving temporal grounding by integrating lightweight temporal attention within the low-rank adaptation framework.

Contribution

It proposes a novel temporal PEFT approach that incorporates temporal attention into low-rank adaptation, improving VideoQA robustness and temporal grounding capabilities.

Findings

Improves Out-of-Template performance on surgical VideoQA datasets.

Enhances robustness to linguistic variation in question phrasing.

Validates effectiveness on multiple surgical VideoQA benchmarks.

Abstract

Surgical Video Question Answering (VideoQA) requires accurate temporal grounding while remaining robust to natural variation in how clinicians phrase questions, where linguistic bias can arise. Standard Parameter Efficient Fine Tuning (PEFT) methods adapt pretrained projections without explicitly modeling frame-to-frame interactions within the adaptation pathway, limiting their ability to exploit sparse temporal evidence. We introduce TemporalDoRA, a video-specific PEFT formulation that extends Weight-Decomposed Low-Rank Adaptation by (i) inserting lightweight temporal Multi-Head Attention (MHA) inside the low-rank bottleneck of the vision encoder and (ii) selectively applying weight decomposition only to the trainable low-rank branch rather than the full adapted weight. This design enables temporally-aware updates while preserving a frozen backbone and stable scaling. By mixing information across frames within the adaptation subspace, TemporalDoRA steers updates toward temporally consistent visual cues and improves robustness with minimal parameter overhead. To benchmark this setting, we present REAL-Colon-VQA, a colonoscopy VideoQA dataset with 6,424 clip--question pairs, including paired rephrased Out-of-Template questions to evaluate sensitivity to linguistic variation. TemporalDoRA improves Out-of-Template performance, and ablation studies confirm that temporal mixing inside the low-rank branch is the primary driver of these gains. We also validate on EndoVis18-VQA adapted to short clips and observe consistent improvements on the Out-of-Template split. Code and dataset available at~\href{https://anonymous.4open.science/r/TemporalDoRA-BFC8/}{Anonymous GitHub}.