Solving Linear Algebra by Program Synthesis

TL;DR

This paper demonstrates that interactive program synthesis using OpenAI Codex can accurately solve university-level linear algebra problems, generate visualizations, and create new questions, advancing automated STEM education tools.

Contribution

It introduces a novel approach of transforming linear algebra questions into programming tasks and uses zero-shot Codex synthesis to solve them without overfitting.

Findings

Achieved perfect accuracy on MIT and Columbia linear algebra courses

Generated visually appealing plots as part of solutions

Automatically created new course questions from sample problems

Abstract

We solve MIT's Linear Algebra 18.06 course and Columbia University's Computational Linear Algebra COMS3251 courses with perfect accuracy by interactive program synthesis. This surprisingly strong result is achieved by turning the course questions into programming tasks and then running the programs to produce the correct answers. We use OpenAI Codex with zero-shot learning, without providing any examples in the prompts, to synthesize code from questions. We quantify the difference between the original question text and the transformed question text that yields a correct answer. Since all COMS3251 questions are not available online the model is not overfitting. We go beyond just generating code for questions with numerical answers by interactively generating code that also results visually pleasing plots as output. Finally, we automatically generate new questions given a few sample questions which may be used as new course content. This work is a significant step forward in solving quantitative math problems and opens the door for solving many university level STEM courses by machine.