Underreporting of errors in NLG output, and what to do about it

TL;DR

This paper highlights the significant under-reporting of errors in Natural Language Generation systems, emphasizing the need for detailed error analysis to guide system improvements and proposing recommendations for better error reporting practices.

Contribution

It quantifies error under-reporting in NLG and offers guidelines for more comprehensive error identification, analysis, and reporting.

Findings

Error under-reporting is widespread in NLG research.

Current metrics do not reflect specific system weaknesses.

Recommendations improve transparency and error analysis in NLG.

Abstract

We observe a severe under-reporting of the different kinds of errors that Natural Language Generation systems make. This is a problem, because mistakes are an important indicator of where systems should still be improved. If authors only report overall performance metrics, the research community is left in the dark about the specific weaknesses that are exhibited by `state-of-the-art' research. Next to quantifying the extent of error under-reporting, this position paper provides recommendations for error identification, analysis and reporting.