Evaluating LLMs in the Context of a Functional Programming Course: A Comprehensive Study

TL;DR

This study evaluates the effectiveness of large language models in a low-resource functional programming course using OCaml, introducing new benchmarks and manual grading to assess their capabilities and limitations.

Contribution

It presents three comprehensive benchmarks for evaluating LLMs in OCaml educational tasks and provides insights into their strengths and weaknesses in a low-resource language setting.

Findings

LLMs are effective in syntax correction and conceptual questions

They solve fewer homework problems in low-resource settings compared to Python and Java

Top LLMs perform well across all tasks in the course

Abstract

Large-Language Models (LLMs) are changing the way learners acquire knowledge outside the classroom setting. Previous studies have shown that LLMs seem effective in generating to short and simple questions in introductory CS courses using high-resource programming languages such as Java or Python. In this paper, we evaluate the effectiveness of LLMs in the context of a low-resource programming language -- OCaml, in an educational setting. In particular, we built three benchmarks to comprehensively evaluate 9 state-of-the-art LLMs: 1) $\lambda$CodeGen (a benchmark containing natural-language homework programming problems); 2) $\lambda$Repair (a benchmark containing programs with syntax, type, and logical errors drawn from actual student submissions); 3) $\lambda$Explain (a benchmark containing natural language questions regarding theoretical programming concepts). We grade each LLMs responses with respect to correctness using the OCaml compiler and an autograder. And our evaluation goes beyond common evaluation methodology by using manual grading to assess the quality of the responses. Our study shows that the top three LLMs are effective on all tasks within a typical functional programming course, although they solve much fewer homework problems in the low-resource setting compared to their success on introductory programming problems in Python and Java. The strength of LLMs lies in correcting syntax and type errors as well as generating answers to basic conceptual questions. While LLMs may not yet match dedicated language-specific tools in some areas, their convenience as a one-stop tool for multiple programming languages can outweigh the benefits of more specialized systems. We hope our benchmarks can serve multiple purposes: to assess the evolving capabilities of LLMs, to help instructors raise awareness among students about the limitations of LLM-generated solutions, and to inform programming language researchers about opportunities to integrate domain-specific reasoning into LLMs and develop more powerful code synthesis and repair tools for low-resource languages.