StructAlign: Structured Cross-Modal Alignment for Continual Text-to-Video Retrieval

TL;DR

StructAlign is a novel method that uses geometric priors and relation-preserving losses to improve continual text-to-video retrieval by mitigating feature drift and modality misalignment.

Contribution

It introduces a structured cross-modal alignment framework with a simplex ETF geometry and novel loss functions to address catastrophic forgetting in CTVR.

Findings

Outperforms state-of-the-art continual retrieval methods on benchmark datasets.

Effectively mitigates intra-modal and cross-modal feature drift.

Enhances stability and accuracy in incremental learning of text-video associations.

Abstract

Continual Text-to-Video Retrieval (CTVR) is a challenging multimodal continual learning setting, where models must incrementally learn new semantic categories while maintaining accurate text-video alignment for previously learned ones, thus making it particularly prone to catastrophic forgetting. A key challenge in CTVR is feature drift, which manifests in two forms: intra-modal feature drift caused by continual learning within each modality, and non-cooperative feature drift across modalities that leads to modality misalignment. To mitigate these issues, we propose StructAlign, a structured cross-modal alignment method for CTVR. First, StructAlign introduces a simplex Equiangular Tight Frame (ETF) geometry as a unified geometric prior to mitigate modality misalignment. Building upon this geometric prior, we design a cross-modal ETF alignment loss that aligns text and video features with category-level ETF prototypes, encouraging the learned representations to form an approximate simplex ETF geometry. In addition, to suppress intra-modal feature drift, we design a Cross-modal Relation Preserving loss, which leverages complementary modalities to preserve cross-modal similarity relations, providing stable relational supervision for feature updates. By jointly addressing non-cooperative feature drift across modalities and intra-modal feature drift, StructAlign effectively alleviates catastrophic forgetting in CTVR. Extensive experiments on benchmark datasets demonstrate that our method consistently outperforms state-of-the-art continual retrieval approaches.