RetrieveAll: A Multilingual Named Entity Recognition Framework with Large Language Models

TL;DR

RetrieveAll is a novel multilingual NER framework utilizing dynamic LoRA and hierarchical prompting, significantly improving performance across languages, especially low-resource ones, without high computational costs.

Contribution

It introduces a scalable, efficient multilingual NER method with dynamic adaptability and a knowledge augmentation technique that enhances low-resource language recognition.

Findings

Achieves 12.1% higher F1 on PAN-X dataset.

Outperforms existing multilingual NER baselines.

Demonstrates effective knowledge utilization without external resources.

Abstract

The rise of large language models has led to significant performance breakthroughs in named entity recognition (NER) for high-resource languages, yet there remains substantial room for improvement in low- and medium-resource languages. Existing multilingual NER methods face severe language interference during the multi-language adaptation process, manifested in feature conflicts between different languages and the competitive suppression of low-resource language features by high-resource languages. Although training a dedicated model for each language can mitigate such interference, it lacks scalability and incurs excessive computational costs in real-world applications. To address this issue, we propose RetrieveAll, a universal multilingual NER framework based on dynamic LoRA. The framework decouples task-specific features across languages and demonstrates efficient dynamic adaptability. Furthermore, we introduce a cross-granularity knowledge augmented method that fully exploits the intrinsic potential of the data without relying on external resources. By leveraging a hierarchical prompting mechanism to guide knowledge injection, this approach advances the paradigm from "prompt-guided inference" to "prompt-driven learning." Experimental results show that RetrieveAll outperforms existing baselines; on the PAN-X dataset, it achieves an average F1 improvement of 12.1 percent.