Word Order in English-Japanese Simultaneous Interpretation: Analyses and Evaluation using Chunk-wise Monotonic Translation

TL;DR

This paper investigates the challenges of monotonic word order translation in English-Japanese simultaneous interpreting, analyzing grammatical factors affecting translation and evaluating current models' performance with new datasets.

Contribution

It introduces the analysis of chunk-wise monotonic translation features and evaluates existing speech translation models using a specialized dataset, revealing potential underestimations in current evaluation methods.

Findings

Existing SI test sets may underestimate model performance.

CMT sentences as references favor simulST models over ST models.

Offline test sets may underestimate simulST model performance.

Abstract

This paper analyzes the features of monotonic translations, which follow the word order of the source language, in simultaneous interpreting (SI). Word order differences are one of the biggest challenges in SI, especially for language pairs with significant structural differences like English and Japanese. We analyzed the characteristics of chunk-wise monotonic translation (CMT) sentences using the NAIST English-to-Japanese Chunk-wise Monotonic Translation Evaluation Dataset and identified some grammatical structures that make monotonic translation difficult in English-Japanese SI. We further investigated the features of CMT sentences by evaluating the output from the existing speech translation (ST) and simultaneous speech translation (simulST) models on the NAIST English-to-Japanese Chunk-wise Monotonic Translation Evaluation Dataset as well as on existing test sets. The results indicate the possibility that the existing SI-based test set underestimates the model performance. The results also suggest that using CMT sentences as references gives higher scores to simulST models than ST models, and that using an offline-based test set to evaluate the simulST models underestimates the model performance.