DAHRS: Divergence-Aware Hallucination-Remediated SRL Projection

TL;DR

DAHRS introduces a linguistically-informed, divergence-aware method to improve multilingual semantic role labeling projection, significantly reducing hallucinations and outperforming existing large language model-based approaches in accuracy and explainability.

Contribution

The paper presents DAHRS, a novel divergence-aware SRL projection method that enhances accuracy without additional transformer models and extends to phrase-level projection across multiple languages.

Findings

DAHRS outperforms XSRL in accuracy on CoNLL-2009 benchmarks.

DAHRS achieves higher human and automatic evaluation scores.

The divergence metric enables adaptation to various language pairs.

Abstract

Semantic role labeling (SRL) enriches many downstream applications, e.g., machine translation, question answering, summarization, and stance/belief detection. However, building multilingual SRL models is challenging due to the scarcity of semantically annotated corpora for multiple languages. Moreover, state-of-the-art SRL projection (XSRL) based on large language models (LLMs) yields output that is riddled with spurious role labels. Remediation of such hallucinations is not straightforward due to the lack of explainability of LLMs. We show that hallucinated role labels are related to naturally occurring divergence types that interfere with initial alignments. We implement Divergence-Aware Hallucination-Remediated SRL projection (DAHRS), leveraging linguistically-informed alignment remediation followed by greedy First-Come First-Assign (FCFA) SRL projection. DAHRS improves the accuracy of SRL projection without additional transformer-based machinery, beating XSRL in both human and automatic comparisons, and advancing beyond headwords to accommodate phrase-level SRL projection (e.g., EN-FR, EN-ES). Using CoNLL-2009 as our ground truth, we achieve a higher word-level F1 over XSRL: 87.6% vs. 77.3% (EN-FR) and 89.0% vs. 82.7% (EN-ES). Human phrase-level assessments yield 89.1% (EN-FR) and 91.0% (EN-ES). We also define a divergence metric to adapt our approach to other language pairs (e.g., English-Tagalog).