Boost Test-Time Performance with Closed-Loop Inference

TL;DR

This paper introduces a Closed-Loop Inference method that iteratively refines predictions on hard test samples, improving model accuracy on both in-distribution and out-of-distribution datasets.

Contribution

It proposes a novel looped inference framework with a filtering criterion and auxiliary calibration tasks to enhance test-time performance.

Findings

Improves accuracy on ImageNet and ImageNet-C datasets.

Effective for both in-distribution and out-of-distribution samples.

Enhances pre-trained model performance with minimal additional effort.

Abstract

Conventional deep models predict a test sample with a single forward propagation, which, however, may not be sufficient for predicting hard-classified samples. On the contrary, we human beings may need to carefully check the sample many times before making a final decision. During the recheck process, one may refine/adjust the prediction by referring to related samples. Motivated by this, we propose to predict those hard-classified test samples in a looped manner to boost the model performance. However, this idea may pose a critical challenge: how to construct looped inference, so that the original erroneous predictions on these hard test samples can be corrected with little additional effort. To address this, we propose a general Closed-Loop Inference (CLI) method. Specifically, we first devise a filtering criterion to identify those hard-classified test samples that need additional inference loops. For each hard sample, we construct an additional auxiliary learning task based on its original top-$K$ predictions to calibrate the model, and then use the calibrated model to obtain the final prediction. Promising results on ImageNet (in-distribution test samples) and ImageNet-C (out-of-distribution test samples) demonstrate the effectiveness of CLI in improving the performance of any pre-trained model.