A Neural Multi-sequence Alignment TeCHnique (NeuMATCH)

TL;DR

This paper introduces NeuMATCH, an end-to-end neural architecture for aligning heterogeneous sequential data like video and text, overcoming limitations of traditional methods by supporting complex alignment tasks.

Contribution

NeuMATCH is a novel neural multi-sequence alignment technique that enables flexible, end-to-end training for various complex alignment scenarios.

Findings

Outperforms state-of-the-art baselines on semi-synthetic datasets

Supports a wide range of alignment types including non-monotonic

Demonstrates effectiveness on real datasets

Abstract

The alignment of heterogeneous sequential data (video to text) is an important and challenging problem. Standard techniques for this task, including Dynamic Time Warping (DTW) and Conditional Random Fields (CRFs), suffer from inherent drawbacks. Mainly, the Markov assumption implies that, given the immediate past, future alignment decisions are independent of further history. The separation between similarity computation and alignment decision also prevents end-to-end training. In this paper, we propose an end-to-end neural architecture where alignment actions are implemented as moving data between stacks of Long Short-term Memory (LSTM) blocks. This flexible architecture supports a large variety of alignment tasks, including one-to-one, one-to-many, skipping unmatched elements, and (with extensions) non-monotonic alignment. Extensive experiments on semi-synthetic and real datasets show that our algorithm outperforms state-of-the-art baselines.