COCOA: Cross Modality Contrastive Learning for Sensor Data

TL;DR

COCOA introduces a novel self-supervised contrastive learning method for multisensor data, effectively learning cross-modality representations that outperform existing models in classification tasks with limited labeled data.

Contribution

The paper presents COCOA, a new cross-modality contrastive learning framework that leverages cross-correlation for multisensor data, addressing limitations of existing single-modality contrastive methods.

Findings

COCOA outperforms state-of-the-art self-supervised models in classification accuracy.

COCOA is significantly more label-efficient, requiring only one-tenth of labeled data.

The method effectively learns from multiple sensor modalities, enhancing representation quality.

Abstract

Self-Supervised Learning (SSL) is a new paradigm for learning discriminative representations without labelled data and has reached comparable or even state-of-the-art results in comparison to supervised counterparts. Contrastive Learning (CL) is one of the most well-known approaches in SSL that attempts to learn general, informative representations of data. CL methods have been mostly developed for applications in computer vision and natural language processing where only a single sensor modality is used. A majority of pervasive computing applications, however, exploit data from a range of different sensor modalities. While existing CL methods are limited to learning from one or two data sources, we propose COCOA (Cross mOdality COntrastive leArning), a self-supervised model that employs a novel objective function to learn quality representations from multisensor data by computing the cross-correlation between different data modalities and minimizing the similarity between irrelevant instances. We evaluate the effectiveness of COCOA against eight recently introduced state-of-the-art self-supervised models, and two supervised baselines across five public datasets. We show that COCOA achieves superior classification performance to all other approaches. Also, COCOA is far more label-efficient than the other baselines including the fully supervised model using only one-tenth of available labelled data.