E-Gen: Leveraging E-Graphs to Improve Continuous Representations of Symbolic Expressions

TL;DR

E-Gen introduces an e-graph-based dataset generation method to create large, diverse mathematical expression datasets, improving embedding quality and outperforming existing models in mathematical language tasks.

Contribution

The paper presents E-Gen, a novel e-graph-based dataset generation scheme that enhances diversity and size of mathematical expression datasets for better embeddings.

Findings

E-Gen produces larger, more diverse datasets than prior methods.

Embedding models trained with E-Gen outperform previous approaches.

Our approach surpasses state-of-the-art LLMs on mathematical tasks.

Abstract

Vector representations have been pivotal in advancing natural language processing (NLP), with prior research focusing on embedding techniques for mathematical expressions using mathematically equivalent formulations. While effective, these approaches are constrained by the size and diversity of training data. In this work, we address these limitations by introducing E-Gen, a novel e-graph-based dataset generation scheme that synthesizes large and diverse mathematical expression datasets, surpassing prior methods in size and operator variety. Leveraging this dataset, we train embedding models using two strategies: (1) generating mathematically equivalent expressions, and (2) contrastive learning to explicitly group equivalent expressions. We evaluate these embeddings on both in-distribution and out-of-distribution mathematical language processing tasks, comparing them against prior methods. Finally, we demonstrate that our embedding-based approach outperforms state-of-the-art large language models (LLMs) on several tasks, underscoring the necessity of optimizing embedding methods for the mathematical data modality. The source code and datasets are available at https://github.com/MLPgroup/E-Gen.