Drop-DTW: Aligning Common Signal Between Sequences While Dropping Outliers

TL;DR

Drop-DTW is a new algorithm that improves sequence alignment by automatically dropping outliers, making it robust for noisy data and effective for various applications like video analysis and cross-modal retrieval.

Contribution

We introduce Drop-DTW, a differentiable dynamic programming algorithm that aligns common signals while dropping outliers, enhancing robustness in sequence matching tasks.

Findings

Drop-DTW outperforms standard DTW in noisy sequence alignment.

It achieves state-of-the-art results in weakly-supervised video localization.

Effective as a training loss for diverse applications.

Abstract

In this work, we consider the problem of sequence-to-sequence alignment for signals containing outliers. Assuming the absence of outliers, the standard Dynamic Time Warping (DTW) algorithm efficiently computes the optimal alignment between two (generally) variable-length sequences. While DTW is robust to temporal shifts and dilations of the signal, it fails to align sequences in a meaningful way in the presence of outliers that can be arbitrarily interspersed in the sequences. To address this problem, we introduce Drop-DTW, a novel algorithm that aligns the common signal between the sequences while automatically dropping the outlier elements from the matching. The entire procedure is implemented as a single dynamic program that is efficient and fully differentiable. In our experiments, we show that Drop-DTW is a robust similarity measure for sequence retrieval and demonstrate its effectiveness as a training loss on diverse applications. With Drop-DTW, we address temporal step localization on instructional videos, representation learning from noisy videos, and cross-modal representation learning for audio-visual retrieval and localization. In all applications, we take a weakly- or unsupervised approach and demonstrate state-of-the-art results under these settings.