TALL -- A Trainable Architecture for Enhancing LLM Performance in Low-Resource Languages

TL;DR

TALL is a novel architecture that enhances low-resource language performance in LLMs by integrating translation models and efficient training strategies, demonstrated effectively on Hebrew.

Contribution

Introduces TALL, a trainable architecture combining translation models with LLMs, utilizing parameter-efficient training for low-resource languages.

Findings

Significant performance improvements on Hebrew.

Effective low-resource language handling with minimal additional training.

Parameter-efficient design balances performance and computational cost.

Abstract

Large Language Models (LLMs) excel in high-resource languages but struggle with low-resource languages due to limited training data. This paper presents TALL (Trainable Architecture for Enhancing LLM Performance in Low-Resource Languages), which integrates an LLM with two bilingual translation models. TALL transforms low-resource inputs into high-resource representations, leveraging the LLM's capabilities while preserving linguistic features through dimension alignment layers and custom transformers. Our experiments on Hebrew demonstrate significant improvements over several baselines, including direct use, naive translation, and fine-tuning approaches. The architecture employs a parameter-efficient strategy, freezing pre-trained components while training only lightweight adapter modules, balancing computational efficiency with performance gains.