Cyclical Focal Loss

TL;DR

The paper introduces a cyclical focal loss that enhances deep neural network training by improving performance across balanced and imbalanced datasets, demonstrating superior results with minimal implementation effort.

Contribution

It proposes a new cyclical focal loss function that is more universal and effective than existing loss functions for various dataset distributions.

Findings

Superior performance on CIFAR-10/100, ImageNet, and long-tailed datasets.

Easy to implement with minimal code changes.

No additional training time required.

Abstract

The cross-entropy softmax loss is the primary loss function used to train deep neural networks. On the other hand, the focal loss function has been demonstrated to provide improved performance when there is an imbalance in the number of training samples in each class, such as in long-tailed datasets. In this paper, we introduce a novel cyclical focal loss and demonstrate that it is a more universal loss function than cross-entropy softmax loss or focal loss. We describe the intuition behind the cyclical focal loss and our experiments provide evidence that cyclical focal loss provides superior performance for balanced, imbalanced, or long-tailed datasets. We provide numerous experimental results for CIFAR-10/CIFAR-100, ImageNet, balanced and imbalanced 4,000 training sample versions of CIFAR-10/CIFAR-100, and ImageNet-LT and Places-LT from the Open Long-Tailed Recognition (OLTR) challenge. Implementing the cyclical focal loss function requires only a few lines of code and does not increase training time. In the spirit of reproducibility, our code is available at \url{https://github.com/lnsmith54/CFL}.