LayAlign: Enhancing Multilingual Reasoning in Large Language Models via Layer-Wise Adaptive Fusion and Alignment Strategy

TL;DR

LayAlign is a novel framework that enhances multilingual reasoning in large language models by integrating and aligning representations across all encoder layers through a layer-wise adaptive fusion strategy, leading to improved performance.

Contribution

The paper introduces LayAlign, a new method that leverages layer-wise interactions between LLMs and multilingual encoders, utilizing all encoder layers for better multilingual reasoning.

Findings

Consistently outperforms existing baselines on multilingual reasoning tasks.

Effectively utilizes representations from all encoder layers.

Demonstrates improved alignment and interaction between models.

Abstract

Despite being pretrained on multilingual corpora, large language models (LLMs) exhibit suboptimal performance on low-resource languages. Recent approaches have leveraged multilingual encoders alongside LLMs by introducing trainable parameters connecting the two models. However, these methods typically focus on the encoder's output, overlooking valuable information from other layers. We propose \aname (\mname), a framework that integrates representations from all encoder layers, coupled with the \attaname mechanism to enable layer-wise interaction between the LLM and the multilingual encoder. Extensive experiments on multilingual reasoning tasks, along with analyses of learned representations, show that our approach consistently outperforms existing baselines.