Fisher-aware Quantization for DETR Detectors with Critical-category Objectives

TL;DR

This paper introduces Fisher-aware quantization techniques for DETR object detectors, focusing on critical categories to mitigate quantization-induced performance drops, especially in larger models and datasets.

Contribution

It proposes Fisher-informed mixed-precision quantization and regularization methods to improve critical-category performance in quantized DETR models.

Findings

Significant mAP improvements on critical classes in COCO dataset.

Fisher-aware methods reduce overfitting in quantized models.

Enhanced performance is more pronounced in larger models and with more classes.

Abstract

The impact of quantization on the overall performance of deep learning models is a well-studied problem. However, understanding and mitigating its effects on a more fine-grained level is still lacking, especially for harder tasks such as object detection with both classification and regression objectives. This work defines the performance for a subset of task-critical categories, i.e. the critical-category performance, as a crucial yet largely overlooked fine-grained objective for detection tasks. We analyze the impact of quantization at the category-level granularity, and propose methods to improve performance for the critical categories. Specifically, we find that certain critical categories have a higher sensitivity to quantization, and are prone to overfitting after quantization-aware training (QAT). To explain this, we provide theoretical and empirical links between their performance gaps and the corresponding loss landscapes with the Fisher information framework. Using this evidence, we apply a Fisher-aware mixed-precision quantization scheme, and a Fisher-trace regularization for the QAT on the critical-category loss landscape. The proposed methods improve critical-category metrics of the quantized transformer-based DETR detectors. They are even more significant in case of larger models and higher number of classes where the overfitting becomes more severe. For example, our methods lead to 10.4% and 14.5% mAP gains for, correspondingly, 4-bit DETR-R50 and Deformable DETR on the most impacted critical classes in the COCO Panoptic dataset.