Rain Removal in Traffic Surveillance: Does it Matter?

TL;DR

This paper evaluates the effectiveness of rain removal algorithms in traffic surveillance videos, showing that while some improve segmentation, they often do not enhance feature tracking, highlighting the need for real-world testing.

Contribution

It introduces a new dataset and evaluation protocol for assessing rain removal algorithms in real traffic videos, bridging the gap between synthetic testing and real-world application.

Findings

Single-frame rain removal improves segmentation by 19.7%.

Video-based rain removal enhances feature tracking by 7.72%.

Rain removal algorithms show mixed results on instance segmentation.

Abstract

Varying weather conditions, including rainfall and snowfall, are generally regarded as a challenge for computer vision algorithms. One proposed solution to the challenges induced by rain and snowfall is to artificially remove the rain from images or video using rain removal algorithms. It is the promise of these algorithms that the rain-removed image frames will improve the performance of subsequent segmentation and tracking algorithms. However, rain removal algorithms are typically evaluated on their ability to remove synthetic rain on a small subset of images. Currently, their behavior is unknown on real-world videos when integrated with a typical computer vision pipeline. In this paper, we review the existing rain removal algorithms and propose a new dataset that consists of 22 traffic surveillance sequences under a broad variety of weather conditions that all include either rain or snowfall. We propose a new evaluation protocol that evaluates the rain removal algorithms on their ability to improve the performance of subsequent segmentation, instance segmentation, and feature tracking algorithms under rain and snow. If successful, the de-rained frames of a rain removal algorithm should improve segmentation performance and increase the number of accurately tracked features. The results show that a recent single-frame-based rain removal algorithm increases the segmentation performance by 19.7% on our proposed dataset, but it eventually decreases the feature tracking performance and showed mixed results with recent instance segmentation methods. However, the best video-based rain removal algorithm improves the feature tracking accuracy by 7.72%.