QT-DoG: Quantization-aware Training for Domain Generalization

TL;DR

This paper introduces QT-DoG, a quantization-aware training method that uses weight quantization as a regularizer to find flatter minima, improving domain generalization and model robustness without extra computational costs.

Contribution

QT-DoG is the first to leverage quantization as an implicit regularizer for domain generalization, promoting flatter minima and enhancing model robustness across domains.

Findings

Quantization leads to flatter minima in the loss landscape.

Ensemble of quantized models outperforms state-of-the-art DG methods.

Model size is reduced without sacrificing accuracy.

Abstract

A key challenge in Domain Generalization (DG) is preventing overfitting to source domains, which can be mitigated by finding flatter minima in the loss landscape. In this work, we propose Quantization-aware Training for Domain Generalization (QT-DoG) and demonstrate that weight quantization effectively leads to flatter minima in the loss landscape, thereby enhancing domain generalization. Unlike traditional quantization methods focused on model compression, QT-DoG exploits quantization as an implicit regularizer by inducing noise in model weights, guiding the optimization process toward flatter minima that are less sensitive to perturbations and overfitting. We provide both an analytical perspective and empirical evidence demonstrating that quantization inherently encourages flatter minima, leading to better generalization across domains. Moreover, with the benefit of reducing the model size through quantization, we demonstrate that an ensemble of multiple quantized models further yields superior accuracy than the state-of-the-art DG approaches with no computational or memory overheads. Code is released at: https://saqibjaved1.github.io/QT_DoG/.