The Road to Quality is Paved with Good Revisions: A Detailed Evaluation Methodology for Revision Policies in Incremental Sequence Labelling

TL;DR

This paper introduces a formal framework and evaluation metrics for revision policies in incremental sequence labeling, analyzing Transformer encoders to improve output revision strategies in dialogue models.

Contribution

It formalizes edits and revisions in incremental sequence labeling and proposes metrics to evaluate revision policies, applying them to Transformer encoders for better incremental behavior.

Findings

Formalization of edits and revisions in incremental labeling

Development of metrics for revision policy evaluation

Profiling of Transformer encoders' incremental behavior

Abstract

Incremental dialogue model components produce a sequence of output prefixes based on incoming input. Mistakes can occur due to local ambiguities or to wrong hypotheses, making the ability to revise past outputs a desirable property that can be governed by a policy. In this work, we formalise and characterise edits and revisions in incremental sequence labelling and propose metrics to evaluate revision policies. We then apply our methodology to profile the incremental behaviour of three Transformer-based encoders in various tasks, paving the road for better revision policies.