Structuring a Training Strategy to Robustify Perception Models with Realistic Image Augmentations

TL;DR

This paper presents a novel training methodology using physics-based augmentations to improve perception model robustness in challenging autonomous driving scenarios, demonstrating significant performance gains on standard benchmarks.

Contribution

It introduces a comprehensive framework for customizing augmentations and optimizing training strategies to enhance model resilience in diverse operational conditions.

Findings

Improved mAP and mIoU metrics on object detection and segmentation datasets.

Enhanced model robustness to challenging environmental conditions.

Customization of augmentations leads to better performance in real-world scenarios.

Abstract

Advancing Machine Learning (ML)-based perception models for autonomous systems necessitates addressing weak spots within the models, particularly in challenging Operational Design Domains (ODDs). These are environmental operating conditions of an autonomous vehicle which can contain difficult conditions, e.g., lens flare at night or objects reflected in a wet street. This report introduces a novel methodology for training with augmentations to enhance model robustness and performance in such conditions. The proposed approach leverages customized physics-based augmentation functions, to generate realistic training data that simulates diverse ODD scenarios. We present a comprehensive framework that includes identifying weak spots in ML models, selecting suitable augmentations, and devising effective training strategies. The methodology integrates hyperparameter optimization and latent space optimization to fine-tune augmentation parameters, ensuring they maximally improve the ML models' performance. Experimental results demonstrate improvements in model performance, as measured by commonly used metrics such as mean Average Precision (mAP) and mean Intersection over Union (mIoU) on open-source object detection and semantic segmentation models and datasets. Our findings emphasize that optimal training strategies are model- and data-specific and highlight the benefits of integrating augmentations into the training pipeline. By incorporating augmentations, we observe enhanced robustness of ML-based perception models, making them more resilient to edge cases encountered in real-world ODDs. This work underlines the importance of customized augmentations and offers an effective solution for improving the safety and reliability of autonomous driving functions.