CrossPL: Evaluating Large Language Models on Cross Programming Language Code Generation

TL;DR

CrossPL is a new benchmark for evaluating large language models' ability to generate code that enables interoperability across multiple programming languages, highlighting current models' limitations in this complex task.

Contribution

We introduce CrossPL, the first comprehensive benchmark for assessing LLMs' performance on cross-programming-language code generation involving IPC mechanisms.

Findings

State-of-the-art LLMs perform poorly on CPL tasks.

Existing models struggle with multi-language code generation.

Benchmark reveals significant gaps in current LLM capabilities.

Abstract

As large language models (LLMs) become increasingly embedded in software engineering workflows, a critical capability remains underexplored: generating correct code that enables cross-programming-language (CPL) interoperability. This skill is essential for building complex systems that integrate components written in multiple languages via mechanisms like inter-process communication (IPC). To bridge this gap, we present CrossPL, the first benchmark designed to systematically evaluate LLMs' ability to generate CPL-interoperating code. CrossPL comprises 1,982 tasks centered around IPC, covering six widely-used programming languages and seven representative CPL techniques. We construct this benchmark by (i) analyzing 19,169 multi-language GitHub repositories using 156 hand-crafted finite state machines (FSMs), and (ii) developing an LLM-based pipeline that automatically extracts CPL code snippets, generates task instructions, and validates functional correctness. We evaluate 14 state-of-the-art general-purpose LLMs and 6 code-oriented LLMs released in the past three years on CrossPL via FSM-based validation. Results reveal that even the best-performing models struggle with CPL scenarios, underscoring the need for more targeted research in this space. Our benchmark and code are available at: https://anonymous.4open.science/r/crosspl-2814.