SteerEval: Inference-time Interventions Strengthen Multilingual Generalization in Neural Summarization Metrics

TL;DR

This paper introduces inference-time interventions for multilingual neural summarization metrics, improving their correlation with human judgments by steering model activations towards an English pivot, thereby enhancing multilingual evaluation accuracy.

Contribution

It demonstrates that test-time intervention methods can effectively improve multilingual neural metrics by aligning their internal representations with an English pivot.

Findings

Interventions increase metric effectiveness across diverse languages.

Steering activations towards English improves correlation with human judgments.

Methods are effective for both encoder- and decoder-based metrics.

Abstract

An increasing body of work has leveraged multilingual language models for Natural Language Generation tasks such as summarization. A major empirical bottleneck in this area is the shortage of accurate and robust evaluation metrics for many languages, which hinders progress. Recent studies suggest that multilingual language models often use English as an internal pivot language, and that misalignment with this pivot can lead to degraded downstream performance. Motivated by the hypothesis that this mismatch could also apply to multilingual neural metrics, we ask whether steering their activations toward an English pivot can improve correlation with human judgments. We experiment with encoder- and decoder-based metrics and find that test-time intervention methods are effective across the board, increasing metric effectiveness for diverse languages.