Building Resilience to Out-of-Distribution Visual Data via Input Optimization and Model Finetuning

TL;DR

This paper introduces an Input Optimization Network that preprocesses images to improve out-of-distribution data resilience in semantic segmentation, outperforming traditional finetuning and adversarial methods, especially in edge cases.

Contribution

The work presents a novel input optimization approach that enhances model robustness to out-of-distribution data and combines it with finetuning for superior performance.

Findings

Input optimization achieves comparable performance to finetuning.

Combined input optimization and finetuning outperform individual methods.

Joint training of input optimization and target model yields significant gains.

Abstract

A major challenge in machine learning is resilience to out-of-distribution data, that is data that exists outside of the distribution of a model's training data. Training is often performed using limited, carefully curated datasets and so when a model is deployed there is often a significant distribution shift as edge cases and anomalies not included in the training data are encountered. To address this, we propose the Input Optimisation Network, an image preprocessing model that learns to optimise input data for a specific target vision model. In this work we investigate several out-of-distribution scenarios in the context of semantic segmentation for autonomous vehicles, comparing an Input Optimisation based solution to existing approaches of finetuning the target model with augmented training data and an adversarially trained preprocessing model. We demonstrate that our approach can enable performance on such data comparable to that of a finetuned model, and subsequently that a combined approach, whereby an input optimization network is optimised to target a finetuned model, delivers superior performance to either method in isolation. Finally, we propose a joint optimisation approach, in which input optimization network and target model are trained simultaneously, which we demonstrate achieves significant further performance gains, particularly in challenging edge-case scenarios. We also demonstrate that our architecture can be reduced to a relatively compact size without a significant performance impact, potentially facilitating real time embedded applications.