Syntax Without Semantics: Teaching Large Language Models to Code in an Unseen Language

TL;DR

This paper investigates whether large language models can transfer coding skills to an unseen language, revealing a gap between understanding algorithms and expressing them in new languages.

Contribution

The study introduces PyLang, a new minimal language, and demonstrates that fine-tuning improves syntax but not semantic transfer, highlighting the implementation fidelity gap.

Findings

Fine-tuning teaches syntax quickly but not semantics.

Models perform 19% better on Python than PyLang.

Internal representations are similar across languages, but output diverges.

Abstract

Large language models (LLMs) achieve high pass rates on code generation benchmarks, yet whether they can transfer this ability to languages absent from pretraining remains poorly understood. We introduce PyLang, a minimal imperative language absent from all pretraining corpora, and evaluate frontier models zero-shot and fine-tuned Qwen3 (4B, 8B, 32B) on 352 problems. We find that fine-tuning quickly teaches syntax but fails to transfer semantic competence: Python outperforms PyLang by up to 19% across all configurations, and no intervention (multi-task learning, preference tuning, code infilling, or latent-space objectives) closes the gap. An LLM judge reveals that frontier models select an identical algorithm to Python 80% of the time, yet cannot translate it into a working PyLang implementation., and CKA analysis confirms that fine-tuned models converge to nearly identical internal representations across languages (CKA > 0.97) while diverging at the output stage. We term this the implementation fidelity gap: models possess language-agnostic algorithmic understanding but cannot express it in an unfamiliar language. Our findings highlight the need for training methods that decouple reasoning from language-specific realization.