ChemAlgebra: Algebraic Reasoning on Chemical Reactions

TL;DR

ChemAlgebra is a new benchmark designed to evaluate the reasoning abilities of deep learning models by predicting balanced chemical reactions, involving complex objects and algebraic constraints.

Contribution

It introduces ChemAlgebra, a novel benchmark for assessing deep learning models' reasoning on chemical reactions with algebraic constraints.

Findings

Benchmark effectively measures reasoning robustness.

Models struggle with algebraic constraints in chemical reactions.

ChemAlgebra promotes development of reasoning-focused models.

Abstract

While showing impressive performance on various kinds of learning tasks, it is yet unclear whether deep learning models have the ability to robustly tackle reasoning tasks. than by learning the underlying reasoning process that is actually required to solve the tasks. Measuring the robustness of reasoning in machine learning models is challenging as one needs to provide a task that cannot be easily shortcut by exploiting spurious statistical correlations in the data, while operating on complex objects and constraints. reasoning task. To address this issue, we propose ChemAlgebra, a benchmark for measuring the reasoning capabilities of deep learning models through the prediction of stoichiometrically-balanced chemical reactions. ChemAlgebra requires manipulating sets of complex discrete objects -- molecules represented as formulas or graphs -- under algebraic constraints such as the mass preservation principle. We believe that ChemAlgebra can serve as a useful test bed for the next generation of machine reasoning models and as a promoter of their development.