An evaluation of template and ML-based generation of user-readable text from a knowledge graph

TL;DR

This study compares template-based and machine learning-based natural language generation methods for knowledge graphs, finding both approaches produce acceptable human-perceived quality despite different error types, informing future NLG system design.

Contribution

It provides an empirical evaluation of the impact of different error types on human judgments, showing no significant difference between template and ML-based approaches in perceived quality.

Findings

No significant link between errors and low human quality ratings.

Both template and ML-based texts are viable for knowledge graph NLG.

Errors like content dropping or hallucination do not strongly affect human judgments.

Abstract

Typical user-friendly renderings of knowledge graphs are visualisations and natural language text. Within the latter HCI solution approach, data-driven natural language generation systems receive increased attention, but they are often outperformed by template-based systems due to suffering from errors such as content dropping, hallucination, or repetition. It is unknown which of those errors are associated significantly with low quality judgements by humans who the text is aimed for, which hampers addressing errors based on their impact on improving human evaluations. We assessed their possible association with an experiment availing of expert and crowdsourced evaluations of human authored text, template generated text, and sequence-to-sequence model generated text. The results showed that there was no significant association between human authored texts with errors and the low human judgements of naturalness and quality. There was also no significant association between machine learning generated texts with dropped or hallucinated slots and the low human judgements of naturalness and quality. Thus, both approaches appear to be viable options for designing a natural language interface for knowledge graphs.