# Temporal Cycle-Consistency Learning

## TL;DR

This paper presents a self-supervised learning method called temporal cycle consistency (TCC) that learns video representations for alignment and action phase classification, reducing supervision needs and enabling various video analysis applications.

## Contribution

The introduction of TCC as a novel self-supervised approach for learning video embeddings through cycle consistency, improving alignment and classification tasks.

## Key findings

- Embeddings enable few-shot action phase classification.
- TCC complements existing self-supervised methods.
- Applications include video alignment, metadata transfer, and anomaly detection.

## Abstract

We introduce a self-supervised representation learning method based on the task of temporal alignment between videos. The method trains a network using temporal cycle consistency (TCC), a differentiable cycle-consistency loss that can be used to find correspondences across time in multiple videos. The resulting per-frame embeddings can be used to align videos by simply matching frames using the nearest-neighbors in the learned embedding space. To evaluate the power of the embeddings, we densely label the Pouring and Penn Action video datasets for action phases. We show that (i) the learned embeddings enable few-shot classification of these action phases, significantly reducing the supervised training requirements; and (ii) TCC is complementary to other methods of self-supervised learning in videos, such as Shuffle and Learn and Time-Contrastive Networks. The embeddings are also used for a number of applications based on alignment (dense temporal correspondence) between video pairs, including transfer of metadata of synchronized modalities between videos (sounds, temporal semantic labels), synchronized playback of multiple videos, and anomaly detection. Project webpage: https://sites.google.com/view/temporal-cycle-consistency .