Dynamic Test-Time Augmentation via Differentiable Functions

TL;DR

DynTTA is a differentiable image enhancement technique that improves deep learning model robustness to distribution shifts without retraining, by generating recognition-friendly images through learned augmentations.

Contribution

We introduce DynTTA, a novel differentiable data augmentation method that enhances robustness to distribution shifts without retraining the recognition model.

Findings

DynTTA improves robustness on various datasets and models.

It maintains accuracy on clean images while enhancing robustness.

Estimating training-time augmentations further boosts performance.

Abstract

Distribution shifts, which often occur in the real world, degrade the accuracy of deep learning systems, and thus improving robustness to distribution shifts is essential for practical applications. To improve robustness, we study an image enhancement method that generates recognition-friendly images without retraining the recognition model. We propose a novel image enhancement method, DynTTA, which is based on differentiable data augmentation techniques and generates a blended image from many augmented images to improve the recognition accuracy under distribution shifts. In addition to standard data augmentations, DynTTA also incorporates deep neural network-based image transformation, further improving the robustness. Because DynTTA is composed of differentiable functions, it can be directly trained with the classification loss of the recognition model. In experiments with widely used image recognition datasets using various classification models, DynTTA improves the robustness with almost no reduction in classification accuracy for clean images, thus outperforming the existing methods. Furthermore, the results show that robustness is significantly improved by estimating the training-time augmentations for distribution-shifted datasets using DynTTA and retraining the recognition model with the estimated augmentations. DynTTA is a promising approach for applications that require both clean accuracy and robustness. Our code is available at \url{https://github.com/s-enmt/DynTTA}.