Bridging the Language Gap: Enhancing Multilingual Prompt-Based Code Generation in LLMs via Zero-Shot Cross-Lingual Transfer

TL;DR

This paper introduces a zero-shot cross-lingual method using neural projection and LASER embeddings to improve multilingual code generation in LLMs, addressing biases and disparities for non-English prompts.

Contribution

It proposes a novel zero-shot cross-lingual approach that enhances multilingual code generation in LLMs without requiring additional training data for each language.

Findings

Significant improvements in code quality for non-English prompts

Simple translation and fine-tuning approaches are inadequate

Method scales effectively across multiple languages

Abstract

The use of Large Language Models (LLMs) for program code generation has gained substantial attention, but their biases and limitations with non-English prompts challenge global inclusivity. This paper investigates the complexities of multilingual prompt-based code generation. Our evaluations of LLMs, including CODELLAMA and CODEGEMMA, reveal significant disparities in code quality for non-English prompts; we also demonstrate the inadequacy of simple approaches like prompt translation, bootstrapped data augmentation, and fine-tuning. To address this, we propose a zero-shot cross-lingual approach using a neural projection technique, integrating a cross-lingual encoder like LASER to map multilingual embeddings from it into the LLM's token space. This method requires training only on English data and scales effectively to other languages. Results on a translated and quality-checked MBPP dataset show substantial improvements in code quality. This research promotes a more inclusive code generation landscape by empowering LLMs with multilingual capabilities to support the diverse linguistic spectrum in programming.