Attention Mechanism in Randomized Time Warping

TL;DR

This paper demonstrates that Randomized Time Warping (RTW) functions similarly to self-attention mechanisms in Transformers, offering a novel interpretation and showing RTW's superior performance in motion recognition tasks.

Contribution

It reveals that RTW can be interpreted as a form of self-attention, providing a new perspective on RTW's role in sequential pattern analysis and its advantages over traditional self-attention.

Findings

RTW weights correlate highly with self-attention weights (average correlation 0.80).

RTW achieves a 5% performance boost over Transformers on the Something-Something V2 dataset.

RTW operates on entire sequences, unlike local self-attention, leading to performance benefits.

Abstract

This paper reveals that we can interpret the fundamental function of Randomized Time Warping (RTW) as a type of self-attention mechanism, a core technology of Transformers in motion recognition. The self-attention is a mechanism that enables models to identify and weigh the importance of different parts of an input sequential pattern. On the other hand, RTW is a general extension of Dynamic Time Warping (DTW), a technique commonly used for matching and comparing sequential patterns. In essence, RTW searches for optimal contribution weights for each element of the input sequential patterns to produce discriminative features. Although the two approaches look different, these contribution weights can be interpreted as self-attention weights. In fact, the two weight patterns look similar, producing a high average correlation of 0.80 across the ten smallest canonical angles. However, they work in different ways: RTW attention operates on an entire input sequential pattern, while self-attention focuses on only a local view which is a subset of the input sequential pattern because of the computational costs of the self-attention matrix. This targeting difference leads to an advantage of RTW against Transformer, as demonstrated by the 5\% performance improvement on the Something-Something V2 dataset.