Heated-Up Softmax Embedding

TL;DR

This paper introduces a 'heating-up' strategy that progressively increases the temperature in softmax classifiers, resulting in more effective embeddings for metric learning tasks, achieving state-of-the-art results.

Contribution

The paper proposes a novel temperature scheduling method during training that enhances embedding quality for metric learning.

Findings

Higher softmax temperatures produce more spread-out features.

Heating-up training improves embedding compactness and separability.

State-of-the-art performance on multiple benchmarks.

Abstract

Metric learning aims at learning a distance which is consistent with the semantic meaning of the samples. The problem is generally solved by learning an embedding for each sample such that the embeddings of samples of the same category are compact while the embeddings of samples of different categories are spread-out in the feature space. We study the features extracted from the second last layer of a deep neural network based classifier trained with the cross entropy loss on top of the softmax layer. We show that training classifiers with different temperature values of softmax function leads to features with different levels of compactness. Leveraging these insights, we propose a "heating-up" strategy to train a classifier with increasing temperatures, leading the corresponding embeddings to achieve state-of-the-art performance on a variety of metric learning benchmarks.