Perish or Flourish? A Holistic Evaluation of Large Language Models for Code Generation in Functional Programming

TL;DR

This paper introduces FPEval, a comprehensive framework for evaluating large language models on functional programming tasks, revealing performance gaps and style issues in code generation for Haskell, OCaml, and Scala.

Contribution

It presents FPEval, a new benchmark and evaluation framework for assessing LLMs on functional programming languages, including static analysis and repair capabilities.

Findings

LLMs perform better on newer models but still have high error rates in functional languages.

Generated code often follows imperative patterns, affecting maintainability.

Static analysis feedback helps LLMs partially self-repair code issues.

Abstract

Functional programming provides strong foundations for developing reliable and secure software systems, yet its adoption remains not widespread due to the steep learning curve. Recent advances in Large Language Models (LLMs) for code generation present new opportunities to lower these barriers. However, extensive evaluations of LLMs largely focus on imperative programming languages, and their capabilities in functional programming languages (FP) remain underexplored. To address this gap, we introduce FPEval, a holistic evaluation framework built on FPBench, a new benchmark of 721 programming tasks across three difficulty levels on three mainstream FP languages: Haskell, Ocaml and Scala. FPEval provides compehensive evaluation infrastructures with both test validations with comprehensive test suites and static analysis tools to assess both functional correctness and code style and maintainability. Using this framework, we evaluate state-of-the-art LLMs, including GPT-3.5, GPT-4o, and GPT-5, for code generation in functional programming languages and Java as an imperative baseline. Our results demonstrate that LLM performance in functional programming improves substantially with model advancement; however, error rates remain significantly higher in purely functional languages (Haskell and OCaml) than in hybrid (Scala) or imperative (Java) languages. Moreover, LLMs frequently generate non-idiomatic functional code that follows imperative patterns, raising concerns about code style and long-term maintainability. Finally, we show that LLMs can partially self-repair both correctness and quality issues when provided with static analysis feedback and hand-crafted instructions for common types of issues.