Understanding vision transformer robustness through the lens of out-of-distribution detection

TL;DR

This paper investigates how quantized vision transformers behave in out-of-distribution detection, revealing that large-scale pretraining can reduce quantization robustness in OOD scenarios.

Contribution

It provides the first analysis of quantized vision transformers' OOD detection performance, highlighting the impact of pretraining datasets on quantization robustness.

Findings

Quantized models show initial instabilities, especially on larger datasets.

Pretraining on large datasets can decrease low-bit quantization robustness in OOD detection.

Data augmentation may improve quantization robustness in OOD scenarios.

Abstract

Vision transformers have shown remarkable performance in vision tasks, but enabling them for accessible and real-time use is still challenging. Quantization reduces memory and inference costs at the risk of performance loss. Strides have been made to mitigate low precision issues mainly by understanding in-distribution (ID) task behaviour, but the attention mechanism may provide insight on quantization attributes by exploring out-of-distribution (OOD) situations. We investigate the behaviour of quantized small-variant popular vision transformers (DeiT, DeiT3, and ViT) on common OOD datasets. ID analyses show the initial instabilities of 4-bit models, particularly of those trained on the larger ImageNet-22k, as the strongest FP32 model, DeiT3, sharply drop 17% from quantization error to be one of the weakest 4-bit models. While ViT shows reasonable quantization robustness for ID calibration, OOD detection reveals more: ViT and DeiT3 pretrained on ImageNet-22k respectively experienced a 15.0% and 19.2% average quantization delta in AUPR-out between full precision to 4-bit while their ImageNet-1k-only counterparts experienced a 9.5% and 12.0% delta. Overall, our results suggest pretraining on large scale datasets may hinder low-bit quantization robustness in OOD detection and that data augmentation may be a more beneficial option.