Understanding the role of FFNs in driving multilingual behaviour in LLMs

TL;DR

This paper provides an in-depth analysis of how Feed-Forward Networks in Large Language Models influence multilingual capabilities, revealing architectural impacts and phenomena like over-layerization affecting performance.

Contribution

It introduces novel metrics for probing multilingual behavior and uncovers the effects of architecture and layer depth on multilingual processing in LLMs.

Findings

Different multilingual processing patterns in FFN sublayers

Identification of over-layerization phenomenon

Interplay between architecture, layer depth, and multilingual capabilities

Abstract

Multilingualism in Large Language Models (LLMs) is an yet under-explored area. In this paper, we conduct an in-depth analysis of the multilingual capabilities of a family of a Large Language Model, examining its architecture, activation patterns, and processing mechanisms across languages. We introduce novel metrics to probe the model's multilingual behaviour at different layers and shed light on the impact of architectural choices on multilingual processing. Our findings reveal different patterns of multilinugal processing in the sublayers of Feed-Forward Networks of the models. Furthermore, we uncover the phenomenon of "over-layerization" in certain model configurations, where increasing layer depth without corresponding adjustments to other parameters may degrade model performance. Through comparisons within and across languages, we demonstrate the interplay between model architecture, layer depth, and multilingual processing capabilities of LLMs trained on multiple languages.