Novelty-based Generalization Evaluation for Traffic Light Detection

TL;DR

This paper introduces a novelty-aware evaluation framework for CNNs in autonomous driving, specifically for traffic light detection, which accounts for the novelty of test samples to better assess generalization.

Contribution

It proposes a novel scoring method that combines prediction accuracy with sample novelty, enhancing the evaluation of CNN generalization in autonomous driving tasks.

Findings

The framework effectively differentiates between known and novel objects.

It provides more meaningful insights into CNN performance on unseen data.

Visualization aids interpretability of novelty assessments.

Abstract

The advent of Convolutional Neural Networks (CNNs) has led to their application in several domains. One noteworthy application is the perception system for autonomous driving that relies on the predictions from CNNs. Practitioners evaluate the generalization ability of such CNNs by calculating various metrics on an independent test dataset. A test dataset is often chosen based on only one precondition, i.e., its elements are not a part of the training data. Such a dataset may contain objects that are both similar and novel w.r.t. the training dataset. Nevertheless, existing works do not reckon the novelty of the test samples and treat them all equally for evaluating generalization. Such novelty-based evaluations are of significance to validate the fitness of a CNN in autonomous driving applications. Hence, we propose a CNN generalization scoring framework that considers novelty of objects in the test dataset. We begin with the representation learning technique to reduce the image data into a low-dimensional space. It is on this space we estimate the novelty of the test samples. Finally, we calculate the generalization score as a combination of the test data prediction performance and novelty. We perform an experimental study of the same for our traffic light detection application. In addition, we systematically visualize the results for an interpretable notion of novelty.