rETF-semiSL: Semi-Supervised Learning for Neural Collapse in Temporal Data

TL;DR

This paper introduces a semi-supervised pre-training method for neural networks on time series data that enforces neural collapse, improving classification performance by aligning embeddings with theoretical geometry.

Contribution

The paper proposes a novel semi-supervised pre-training strategy that enforces neural collapse in temporal data representations, combining generative tasks and a new augmentation method.

Findings

Outperforms previous pretext tasks on multiple models and datasets

Enforces neural collapse to improve embedding separability

Enhances temporal models' classification accuracy

Abstract

Deep neural networks for time series must capture complex temporal patterns, to effectively represent dynamic data. Self- and semi-supervised learning methods show promising results in pre-training large models, which -- when finetuned for classification -- often outperform their counterparts trained from scratch. Still, the choice of pretext training tasks is often heuristic and their transferability to downstream classification is not granted, thus we propose a novel semi-supervised pre-training strategy to enforce latent representations that satisfy the Neural Collapse phenomenon observed in optimally trained neural classifiers. We use a rotational equiangular tight frame-classifier and pseudo-labeling to pre-train deep encoders with few labeled samples. Furthermore, to effectively capture temporal dynamics while enforcing embedding separability, we integrate generative pretext tasks with our method, and we define a novel sequential augmentation strategy. We show that our method significantly outperforms previous pretext tasks when applied to LSTMs, transformers, and state-space models on three multivariate time series classification datasets. These results highlight the benefit of aligning pre-training objectives with theoretically grounded embedding geometry.