Learning Symbolic Expressions via Gumbel-Max Equation Learner Networks

TL;DR

This paper introduces Gumbel-Max Equation Learner (GMEQL), a neural network architecture that effectively extracts symbolic mathematical expressions from data, outperforming existing methods on benchmark problems.

Contribution

The paper proposes GMEQL, a novel neural network architecture utilizing the Gumbel-Max trick for structure relaxation and a two-stage training process, advancing symbolic regression capabilities.

Findings

GMEQL outperforms several state-of-the-art approaches on benchmark problems.

The two-stage training process effectively optimizes structure and regression parameters.

GMEQL successfully extracts high-level mathematical expressions from complex datasets.

Abstract

Most of the neural networks (NNs) learned via state-of-the-art machine learning techniques are black-box models. For a widespread success of machine learning in science and engineering, it is important to develop new NN architectures to effectively extract high-level mathematical knowledge from complex datasets. Motivated by this understanding, this paper develops a new NN architecture called the Gumbel-Max Equation Learner (GMEQL) network. Different from previously proposed Equation Learner (EQL) networks, GMEQL applies continuous relaxation to the network structure via the Gumbel-Max trick and introduces two types of trainable parameters: structure parameters and regression parameters. This paper also proposes a two-stage training process with new techniques to train structure parameters in both online and offline settings based on an elite repository. On 8 benchmark symbolic regression problems, GMEQL is experimentally shown to outperform several cutting-edge machine learning approaches.