You Only Forward Once: Prediction and Rationalization in A Single Forward Pass

TL;DR

The paper introduces YOFO, a single-pass framework that simultaneously predicts and rationalizes model outputs by gradually removing unimportant tokens, improving interpretability and accuracy over previous two-phase methods.

Contribution

YOFO is a novel single-phase approach that performs prediction and rationale extraction together, addressing interlocking and spurious correlation issues in unsupervised rationale extraction.

Findings

Up to 18.4% improvement in token-level F1 over state-of-the-art methods.

YOFO extracts precise rationales while removing unimportant tokens.

Demonstrates effectiveness on BeerAdvocate and Hotel Review datasets.

Abstract

Unsupervised rationale extraction aims to extract concise and contiguous text snippets to support model predictions without any annotated rationale. Previous studies have used a two-phase framework known as the Rationalizing Neural Prediction (RNP) framework, which follows a generate-then-predict paradigm. They assumed that the extracted explanation, called rationale, should be sufficient to predict the golden label. However, the assumption above deviates from the original definition and is too strict to perform well. Furthermore, these two-phase models suffer from the interlocking problem and spurious correlations. To solve the above problems, we propose a novel single-phase framework called You Only Forward Once (YOFO), derived from a relaxed version of rationale where rationales aim to support model predictions rather than make predictions. In our framework, A pre-trained language model like BERT is deployed to simultaneously perform prediction and rationalization with less impact from interlocking or spurious correlations. Directly choosing the important tokens in an unsupervised manner is intractable. Instead of directly choosing the important tokens, YOFO gradually removes unimportant tokens during forward propagation. Through experiments on the BeerAdvocate and Hotel Review datasets, we demonstrate that our model is able to extract rationales and make predictions more accurately compared to RNP-based models. We observe an improvement of up to 18.4\% in token-level F1 compared to previous state-of-the-art methods. We also conducted analyses and experiments to explore the extracted rationales and token decay strategies. The results show that YOFO can extract precise and important rationales while removing unimportant tokens in the middle part of the model.