FR: Folded Rationalization with a Unified Encoder

TL;DR

This paper introduces Folded Rationalization (FR), a unified encoder approach that improves text semantic extraction by integrating generator and predictor, overcoming degeneration issues in traditional two-phase models.

Contribution

FR unifies the generator and predictor into a single encoder, enhancing the quality of rationales and prediction accuracy in text analysis.

Findings

FR improves F1 score by up to 10.3% over state-of-the-art methods.

Unified encoder facilitates better information flow between components.

Addresses degeneration problem in rationale models.

Abstract

Conventional works generally employ a two-phase model in which a generator selects the most important pieces, followed by a predictor that makes predictions based on the selected pieces. However, such a two-phase model may incur the degeneration problem where the predictor overfits to the noise generated by a not yet well-trained generator and in turn, leads the generator to converge to a sub-optimal model that tends to select senseless pieces. To tackle this challenge, we propose Folded Rationalization (FR) that folds the two phases of the rationale model into one from the perspective of text semantic extraction. The key idea of FR is to employ a unified encoder between the generator and predictor, based on which FR can facilitate a better predictor by access to valuable information blocked by the generator in the traditional two-phase model and thus bring a better generator. Empirically, we show that FR improves the F1 score by up to 10.3% as compared to state-of-the-art methods.