# Optical Flow augmented Semantic Segmentation networks for Automated Driving

## TL;DR

This paper explores how integrating optical flow into semantic segmentation networks enhances the detection and segmentation of moving objects in automated driving, demonstrating significant improvements across multiple datasets and architectures.

## Contribution

It systematically evaluates four architectures combining RGB and optical flow, showing that flow augmentation notably improves segmentation accuracy for moving objects in autonomous driving scenarios.

## Key findings

- Two-stream RGB + flow architecture achieves 4% IoU improvement with ground truth flow.
- Significant IoU increases for moving objects: trucks (38%), vans (28%), cars (6%).
- Flow augmentation improves segmentation in Cityscapes, especially for motorcycles and trains.

## Abstract

Motion is a dominant cue in automated driving systems. Optical flow is typically computed to detect moving objects and to estimate depth using triangulation. In this paper, our motivation is to leverage the existing dense optical flow to improve the performance of semantic segmentation. To provide a systematic study, we construct four different architectures which use RGB only, flow only, RGBF concatenated and two-stream RGB + flow. We evaluate these networks on two automotive datasets namely Virtual KITTI and Cityscapes using the state-of-the-art flow estimator FlowNet v2. We also make use of the ground truth optical flow in Virtual KITTI to serve as an ideal estimator and a standard Farneback optical flow algorithm to study the effect of noise. Using the flow ground truth in Virtual KITTI, two-stream architecture achieves the best results with an improvement of 4% IoU. As expected, there is a large improvement for moving objects like trucks, vans and cars with 38%, 28% and 6% increase in IoU. FlowNet produces an improvement of 2.4% in average IoU with larger improvement in the moving objects corresponding to 26%, 11% and 5% in trucks, vans and cars. In Cityscapes, flow augmentation provided an improvement for moving objects like motorcycle and train with an increase of 17% and 7% in IoU.