PFML: Self-Supervised Learning of Time-Series Data Without Representation Collapse

TL;DR

PFML is a novel self-supervised learning algorithm for time-series data that predicts statistical functionals of masked inputs to avoid representation collapse, demonstrating superior performance across multiple real-world applications.

Contribution

Introduces PFML, a self-supervised learning method for time-series data that prevents representation collapse and is applicable across diverse modalities.

Findings

PFML outperforms similar SSL methods in classification tasks.

PFML matches state-of-the-art SSL performance while being simpler.

PFML is effective across multiple data modalities like sensor, speech, and EEG data.

Abstract

Self-supervised learning (SSL) is a data-driven learning approach that utilizes the innate structure of the data to guide the learning process. In contrast to supervised learning, which depends on external labels, SSL utilizes the inherent characteristics of the data to produce its own supervisory signal. However, one frequent issue with SSL methods is representation collapse, where the model outputs a constant input-invariant feature representation. This issue hinders the potential application of SSL methods to new data modalities, as trying to avoid representation collapse wastes researchers' time and effort. This paper introduces a novel SSL algorithm for time-series data called Prediction of Functionals from Masked Latents (PFML). Instead of predicting masked input signals or their latent representations directly, PFML operates by predicting statistical functionals of the input signal corresponding to masked embeddings, given a sequence of unmasked embeddings. The algorithm is designed to avoid representation collapse, rendering it straightforwardly applicable to different time-series data domains, such as novel sensor modalities in clinical data. We demonstrate the effectiveness of PFML through complex, real-life classification tasks across three different data modalities: infant posture and movement classification from multi-sensor inertial measurement unit data, emotion recognition from speech data, and sleep stage classification from EEG data. The results show that PFML is superior to a conceptually similar SSL method and a contrastive learning-based SSL method. Additionally, PFML is on par with the current state-of-the-art SSL method, while also being conceptually simpler and without suffering from representation collapse.