Doubly Contrastive End-to-End Semantic Segmentation for Autonomous Driving under Adverse Weather

TL;DR

This paper introduces a lightweight, end-to-end contrastive learning method to enhance real-time semantic segmentation for autonomous driving in adverse weather, achieving improved accuracy without heavy computational costs.

Contribution

It proposes a novel doubly contrastive approach that improves lightweight segmentation models under challenging weather conditions without requiring large memory banks or pretraining.

Findings

Up to 1.34% improvement in mIoU on ACDC dataset

Achieves 66.7 FPS on a single GPU

Self-supervision with clear weather images yields comparable results

Abstract

Road scene understanding tasks have recently become crucial for self-driving vehicles. In particular, real-time semantic segmentation is indispensable for intelligent self-driving agents to recognize roadside objects in the driving area. As prior research works have primarily sought to improve the segmentation performance with computationally heavy operations, they require far significant hardware resources for both training and deployment, and thus are not suitable for real-time applications. As such, we propose a doubly contrastive approach to improve the performance of a more practical lightweight model for self-driving, specifically under adverse weather conditions such as fog, nighttime, rain and snow. Our proposed approach exploits both image- and pixel-level contrasts in an end-to-end supervised learning scheme without requiring a memory bank for global consistency or the pretraining step used in conventional contrastive methods. We validate the effectiveness of our method using SwiftNet on the ACDC dataset, where it achieves up to 1.34%p improvement in mIoU (ResNet-18 backbone) at 66.7 FPS (2048x1024 resolution) on a single RTX 3080 Mobile GPU at inference. Furthermore, we demonstrate that replacing image-level supervision with self-supervision achieves comparable performance when pre-trained with clear weather images.