Fine-tune your Classifier: Finding Correlations With Temperature

TL;DR

This paper investigates how dataset statistics correlate with optimal temperature hyperparameters in neural network classification, proposing a heuristic for setting temperature without extensive hyperparameter tuning.

Contribution

It introduces a method to analyze dataset features to predict suitable temperature values, reducing reliance on hyperparameter optimization.

Findings

Strong correlation between dataset statistics and optimal temperatures

Potential for a general heuristic to set temperature values

Preliminary results across diverse datasets and extractors

Abstract

Temperature is a widely used hyperparameter in various tasks involving neural networks, such as classification or metric learning, whose choice can have a direct impact on the model performance. Most of existing works select its value using hyperparameter optimization methods requiring several runs to find the optimal value. We propose to analyze the impact of temperature on classification tasks by describing a dataset as a set of statistics computed on representations on which we can build a heuristic giving us a default value of temperature. We study the correlation between these extracted statistics and the observed optimal temperatures. This preliminary study on more than a hundred combinations of different datasets and features extractors highlights promising results towards the construction of a general heuristic for temperature.