Evaluating Large Language Models on Solved and Unsolved Problems in Graph Theory: Implications for Computing Education

TL;DR

This study evaluates how large language models assist in understanding and exploring graph theory problems, showing they excel at established concepts but are limited in generating new mathematical insights, with implications for computing education.

Contribution

It provides a detailed analysis of LLM performance on solved and unsolved graph theory problems using an authentic inquiry protocol, highlighting strengths and limitations.

Findings

Strong performance on solved problem with correct proofs

Generated plausible strategies for open problem without fabricating results

Acknowledged uncertainty, avoiding unsupported claims

Abstract

Large Language Models are increasingly used by students to explore advanced material in computer science, including graph theory. As these tools become integrated into undergraduate and graduate coursework, it is important to understand how reliably they support mathematically rigorous thinking. This study examines the performance of a LLM on two related graph theoretic problems: a solved problem concerning the gracefulness of line graphs and an open problem for which no solution is currently known. We use an eight stage evaluation protocol that reflects authentic mathematical inquiry, including interpretation, exploration, strategy formation, and proof construction. The model performed strongly on the solved problem, producing correct definitions, identifying relevant structures, recalling appropriate results without hallucination, and constructing a valid proof confirmed by a graph theory expert. For the open problem, the model generated coherent interpretations and plausible exploratory strategies but did not advance toward a solution. It did not fabricate results and instead acknowledged uncertainty, which is consistent with the explicit prompting instructions that directed the model to avoid inventing theorems or unsupported claims. These findings indicate that LLMs can support exploration of established material but remain limited in tasks requiring novel mathematical insight or critical structural reasoning. For computing education, this distinction highlights the importance of guiding students to use LLMs for conceptual exploration while relying on independent verification and rigorous argumentation for formal problem solving.