TuringQ: Benchmarking AI Comprehension in Theory of Computation

TL;DR

TuringQ is a comprehensive benchmark with 4,006 questions designed to evaluate and improve large language models' reasoning skills in the theory of computation, combining human and automated assessments.

Contribution

It introduces the first dedicated benchmark for AI reasoning in the theory of computation, including an automated evaluation system and fine-tuning results showing improved reasoning.

Findings

GPT-4 and open-source LLMs perform variably on TuringQ

Fine-tuning Llama3-8B improves reasoning and out-of-domain performance

Automated LLM-based evaluation is competitive with human assessment

Abstract

We present TuringQ, the first benchmark designed to evaluate the reasoning capabilities of large language models (LLMs) in the theory of computation. TuringQ consists of 4,006 undergraduate and graduate-level question-answer pairs, categorized into four difficulty levels and covering seven core theoretical areas. We evaluate several open-source LLMs, as well as GPT-4, using Chain of Thought prompting and expert human assessment. Additionally, we propose an automated LLM-based evaluation system that demonstrates competitive accuracy when compared to human evaluation. Fine-tuning a Llama3-8B model on TuringQ shows measurable improvements in reasoning ability and out-of-domain tasks such as algebra. TuringQ serves as both a benchmark and a resource for enhancing LLM performance in complex computational reasoning tasks. Our analysis offers insights into LLM capabilities and advances in AI comprehension of theoretical computer science.