Trade-offs in Top-k Classification Accuracies on Losses for Deep Learning

TL;DR

This paper analyzes the limitations of cross entropy loss in optimizing top-k classification accuracy in deep learning and introduces a novel top-k transition loss that improves top-k predictions, especially for larger k values.

Contribution

The paper proposes a new top-k transition loss that enhances top-k accuracy in deep learning models, addressing the shortcomings of cross entropy in complex data distributions.

Findings

CE is not always optimal for top-k prediction in complex distributions.

The proposed loss improves top-k accuracy over CE for larger k values.

ResNet18 trained with the new loss achieves 99% accuracy with fewer candidates.

Abstract

This paper presents an experimental analysis about trade-offs in top-k classification accuracies on losses for deep leaning and proposal of a novel top-k loss. Commonly-used cross entropy (CE) is not guaranteed to optimize top-k prediction without infinite training data and model complexities. The objective is to clarify when CE sacrifices top-k accuracies to optimize top-1 prediction, and to design loss that improve top-k accuracy under such conditions. Our novel loss is basically CE modified by grouping temporal top-k classes as a single class. To obtain a robust decision boundary, we introduce an adaptive transition from normal CE to our loss, and thus call it top-k transition loss. It is demonstrated that CE is not always the best choice to learn top-k prediction in our experiments. First, we explore trade-offs between top-1 and top-k (=2) accuracies on synthetic datasets, and find a failure of CE in optimizing top-k prediction when we have complex data distribution for a given model to represent optimal top-1 prediction. Second, we compare top-k accuracies on CIFAR-100 dataset targeting top-5 prediction in deep learning. While CE performs the best in top-1 accuracy, in top-5 accuracy our loss performs better than CE except using one experimental setup. Moreover, our loss has been found to provide better top-k accuracies compared to CE at k larger than 10. As a result, a ResNet18 model trained with our loss reaches 99 % accuracy with k=25 candidates, which is a smaller candidate number than that of CE by 8.