Improving Image Data Leakage Detection in Automotive Software

TL;DR

This paper presents a new method for detecting image data leakage in automotive perception systems, improving model evaluation accuracy and preventing erroneous real-world predictions.

Contribution

The study introduces a novel detection method for image data leakage in automotive datasets, validated on both proprietary and public datasets, revealing previously unknown leakage in Kitti.

Findings

Effective detection of data leakage in Kitti dataset

Method successfully applied to proprietary Volvo dataset

Enhances reliability of automotive perception model evaluation

Abstract

Data leakage is a very common problem that is often overlooked during splitting data into train and test sets before training any ML/DL model. The model performance gets artificially inflated with the presence of data leakage during the evaluation phase which often leads the model to erroneous prediction on real-time deployment. However, detecting the presence of such leakage is challenging, particularly in the object detection context of perception systems where the model needs to be supplied with image data for training. In this study, we conduct a computational experiment on the Cirrus dataset from our industrial partner Volvo Cars to develop a method for detecting data leakage. We then evaluate the method on another public dataset, Kitti, which is a popular and widely accepted benchmark dataset in the automotive domain. The results show that thanks to our proposed method we are able to detect data leakage in the Kitti dataset, which was previously unknown.