Promoting Graph Awareness in Linearized Graph-to-Text Generation

TL;DR

This paper investigates how pretrained transformers encode graph structures in graph-to-text tasks and proposes denoising scaffolds to improve their implicit graph understanding, especially in low-resource scenarios.

Contribution

It introduces graph-denoising objectives within a multi-task framework to enhance graph encoding in linearized graph-to-text models, addressing invariance and reconstruction issues.

Findings

Denoising scaffolds improve generation quality in low-resource settings.

Linearized models show invariance to graph linearization strategies.

Models can reconstruct corrupted graph inputs effectively.

Abstract

Generating text from structured inputs, such as meaning representations or RDF triples, has often involved the use of specialized graph-encoding neural networks. However, recent applications of pretrained transformers to linearizations of graph inputs have yielded state-of-the-art generation results on graph-to-text tasks. Here, we explore the ability of these linearized models to encode local graph structures, in particular their invariance to the graph linearization strategy and their ability to reconstruct corrupted inputs. Our findings motivate solutions to enrich the quality of models' implicit graph encodings via scaffolding. Namely, we use graph-denoising objectives implemented in a multi-task text-to-text framework. We find that these denoising scaffolds lead to substantial improvements in downstream generation in low-resource settings.