Distribution-Free Pretraining of Classification Losses via Evolutionary Dynamics

TL;DR

This paper introduces EDL, a novel evolutionary framework for pretraining classification losses without real data, using synthetic predictions and a mutation strategy to enhance exploration.

Contribution

The paper presents EDL, a transferable loss learning method optimized via evolutionary strategies with chaotic mutation, improving pretraining and performance over traditional losses.

Findings

EDL can replace cross-entropy in CIFAR-10 classification tasks.

Chaotic mutation accelerates convergence and enhances synthetic pretraining metrics.

EDL achieves competitive or superior accuracy compared to standard methods.

Abstract

We propose Evolutionary Dynamic Loss (EDL), a framework that learns a transferable classification loss in the probability space using unlimited synthetic prediction-label pairs, without accessing real samples during the main loss pretraining stage. EDL parameterizes the loss as a lightweight network and is trained with a semantics-free ranking-consistency objective that assigns larger penalties for more erroneous predictions. To robustly explore the space of loss functions, we optimize EDL via an evolutionary strategy and introduce chaotic mutation to improve exploration under noisy fitness evaluations. Experiments on CIFAR-10 with ResNet backbones show that EDL can serve as a drop-in replacement for cross-entropy and achieves competitive or improved accuracy, while ablation studies confirm that chaotic mutation yields faster convergence and better synthetic pretraining metrics than standard Gaussian mutation.