Adapters for Enhanced Modeling of Multilingual Knowledge and Text

TL;DR

This paper introduces a lightweight adapter method to enhance multilingual language models with knowledge from multilingual knowledge graphs, improving performance on knowledge and language understanding tasks across many languages, especially low-resource ones.

Contribution

It presents a novel adapter-based approach to integrate multilingual knowledge graphs into MLLMs, enabling better factual reasoning and language understanding in low-resource languages.

Findings

Improved knowledge graph completion and entity alignment, especially in low-resource languages.

Enhanced MLLM performance on multilingual factual knowledge tasks.

Maintained performance on general language tasks.

Abstract

Large language models appear to learn facts from the large text corpora they are trained on. Such facts are encoded implicitly within their many parameters, making it difficult to verify or manipulate what knowledge has been learned. Language models have recently been extended to multilingual language models (MLLMs), enabling knowledge to be learned across hundreds of languages. Meanwhile, knowledge graphs contain facts in an explicit triple format, which require careful and costly curation and are only available in a few high-resource languages, restricting their research and application. To address these issues, we propose to enhance MLLMs with knowledge from multilingual knowledge graphs (MLKGs) so as to tackle language and knowledge graph tasks across many languages, including low-resource ones. Specifically, we introduce a lightweight adapter set to enhance MLLMs with cross-lingual entity alignment and facts from MLKGs for many languages. Experiments on common benchmarks show that such enhancement benefits both MLLMs and MLKGs, achieving: (1) comparable or improved performance for knowledge graph completion and entity alignment relative to baselines, especially for low-resource languages (for which knowledge graphs are unavailable); and (2) improved MLLM performance on language understanding tasks that require multilingual factual knowledge; all while maintaining performance on other general language tasks.