Learning Symbolic Model-Agnostic Loss Functions via Meta-Learning

TL;DR

This paper introduces a meta-learning framework that automatically discovers symbolic, model-agnostic loss functions using a hybrid neuro-symbolic search, significantly improving model performance across various tasks.

Contribution

It presents a novel hybrid neuro-symbolic meta-learning approach for automatically learning symbolic loss functions that are model-agnostic and outperform existing methods.

Findings

Meta-learned loss functions outperform cross-entropy and state-of-the-art methods.

The framework is effective across diverse neural architectures and datasets.

Discovered loss functions enhance training performance in supervised learning.

Abstract

In this paper, we develop upon the emerging topic of loss function learning, which aims to learn loss functions that significantly improve the performance of the models trained under them. Specifically, we propose a new meta-learning framework for learning model-agnostic loss functions via a hybrid neuro-symbolic search approach. The framework first uses evolution-based methods to search the space of primitive mathematical operations to find a set of symbolic loss functions. Second, the set of learned loss functions are subsequently parameterized and optimized via an end-to-end gradient-based training procedure. The versatility of the proposed framework is empirically validated on a diverse set of supervised learning tasks. Results show that the meta-learned loss functions discovered by the newly proposed method outperform both the cross-entropy loss and state-of-the-art loss function learning methods on a diverse range of neural network architectures and datasets.