On Efficient Real-Time Semantic Segmentation: A Survey

TL;DR

This survey reviews efficient real-time semantic segmentation models suitable for autonomous vehicles, analyzing their design, performance, and trade-offs on embedded hardware to facilitate scene understanding with limited resources.

Contribution

It provides a comprehensive taxonomy and evaluation of recent compact semantic segmentation models optimized for real-time deployment on low-memory embedded systems.

Findings

Many models achieve real-time inference on embedded hardware

Trade-off observed between model accuracy and latency

Evaluation under consistent hardware setups highlights performance differences

Abstract

Semantic segmentation is the problem of assigning a class label to every pixel in an image, and is an important component of an autonomous vehicle vision stack for facilitating scene understanding and object detection. However, many of the top performing semantic segmentation models are extremely complex and cumbersome, and as such are not suited to deployment onboard autonomous vehicle platforms where computational resources are limited and low-latency operation is a vital requirement. In this survey, we take a thorough look at the works that aim to address this misalignment with more compact and efficient models capable of deployment on low-memory embedded systems while meeting the constraint of real-time inference. We discuss several of the most prominent works in the field, placing them within a taxonomy based on their major contributions, and finally we evaluate the inference speed of the discussed models under consistent hardware and software setups that represent a typical research environment with high-end GPU and a realistic deployed scenario using low-memory embedded GPU hardware. Our experimental results demonstrate that many works are capable of real-time performance on resource-constrained hardware, while illustrating the consistent trade-off between latency and accuracy.