Focus on Likely Classes for Test-Time Prediction

TL;DR

This paper introduces two test-time fine-tuning methods that focus on likely classes to improve uncertain model predictions, demonstrating accuracy gains across text and image models without relying on hand-engineered augmentations.

Contribution

The paper proposes novel test-time fine-tuning techniques that refine predictions by focusing on likely classes, leveraging shared features among classes without auxiliary tasks.

Findings

Accuracy improvements on diverse models

Effective in both text and image domains

Refinement via gradient steps enhances predictions

Abstract

We ask: Can focusing on likely classes of a single, in-domain sample improve model predictions? Prior work argued ``no''. We put forward a novel rationale in favor of ``yes'': Sharedness of features among classes indicates their reliability for a single sample. We aim for an affirmative answer without using hand-engineered augmentations or auxiliary tasks. We propose two novel test-time fine-tuning methods to improve uncertain model predictions. Instead of greedily selecting the most likely class, we introduce an additional step, \emph{focus on the likely classes}, to refine predictions. By applying a single gradient descent step with a large learning rate, we refine predictions when an initial forward pass indicates high uncertainty. The experimental evaluation demonstrates accuracy gains for one of our methods on average, which emphasizes shared features among likely classes. The gains are confirmed across diverse text and image domain models.