Finding Meaning in Points: Weakly Supervised Semantic Segmentation for Event Cameras

TL;DR

This paper introduces EV-WSSS, a weakly supervised method for event camera semantic segmentation using sparse point annotations, dual-student learning, and contrastive feature learning to achieve dense segmentation without pixel-level labels.

Contribution

The paper proposes a novel weakly supervised framework for event-based semantic segmentation that leverages dual-student learning and contrastive prototype-based feature aggregation.

Findings

Achieves substantial segmentation accuracy with sparse point annotations.

Effectively utilizes temporal information from event data.

Demonstrates strong performance on multiple datasets, including newly introduced DSEC Night-Point.

Abstract

Event cameras excel in capturing high-contrast scenes and dynamic objects, offering a significant advantage over traditional frame-based cameras. Despite active research into leveraging event cameras for semantic segmentation, generating pixel-wise dense semantic maps for such challenging scenarios remains labor-intensive. As a remedy, we present EV-WSSS: a novel weakly supervised approach for event-based semantic segmentation that utilizes sparse point annotations. To fully leverage the temporal characteristics of event data, the proposed framework performs asymmetric dual-student learning between 1) the original forward event data and 2) the longer reversed event data, which contain complementary information from the past and the future, respectively. Besides, to mitigate the challenges posed by sparse supervision, we propose feature-level contrastive learning based on class-wise prototypes, carefully aggregated at both spatial region and sample levels. Additionally, we further excavate the potential of our dual-student learning model by exchanging prototypes between the two learning paths, thereby harnessing their complementary strengths. With extensive experiments on various datasets, including DSEC Night-Point with sparse point annotations newly provided by this paper, the proposed method achieves substantial segmentation results even without relying on pixel-level dense ground truths. The code and dataset are available at https://github.com/Chohoonhee/EV-WSSS.