Domain-Specific Quality Estimation for Machine Translation in Low-Resource Scenarios

TL;DR

This paper explores domain-specific quality estimation for low-resource machine translation, comparing prompting techniques across models and proposing adaptations like ALOPE and LoRMA to improve robustness in diverse domains.

Contribution

It introduces ALOPE and LoRMA frameworks for enhancing LLM-based quality estimation in low-resource, domain-specific translation scenarios, demonstrating improved performance.

Findings

Closed-weight models perform well with prompting.

Open-weight models are fragile with prompt-only approaches.

Intermediate-layer adaptation improves QE performance.

Abstract

Quality Estimation (QE) is essential for assessing machine translation quality in reference-less settings, particularly for domain-specific and low-resource language scenarios. In this paper, we investigate sentence-level QE for English to Indic machine translation across four domains (Healthcare, Legal, Tourism, and General) and five language pairs. We systematically compare zero-shot, few-shot, and guideline-anchored prompting across selected closed-weight and open-weight LLMs. Findings indicate that while closed-weight models achieve strong performance via prompting alone, prompt-only approaches remain fragile for open-weight models, especially in high-risk domains. To address this, we adopt ALOPE, a framework for LLM-based QE that uses Low-Rank Adaptation with regression heads attached to selected intermediate Transformer layers. We also extend ALOPE with recently proposed Low-Rank Multiplicative Adaptation (LoRMA). Our results show that intermediate-layer adaptation consistently improves QE performance, with gains in semantically complex domains, indicating a path toward more robust QE in practical scenarios. We release code and domain-specific QE datasets publicly to support further research.