SPECTRA: Sparse Structured Text Rationalization

TL;DR

This paper introduces SPECTRA, a deterministic framework for extracting structured rationales from text using constrained inference, simplifying training and improving explanation quality over stochastic methods.

Contribution

The paper proposes a unified, differentiable approach for structured rationale extraction that outperforms prior stochastic methods in performance and plausibility.

Findings

Deterministic extraction improves rationale quality and training stability.

The framework outperforms stochastic methods in classification and inference tasks.

Provides a comprehensive comparison of stochastic and deterministic rationale methods.

Abstract

Selective rationalization aims to produce decisions along with rationales (e.g., text highlights or word alignments between two sentences). Commonly, rationales are modeled as stochastic binary masks, requiring sampling-based gradient estimators, which complicates training and requires careful hyperparameter tuning. Sparse attention mechanisms are a deterministic alternative, but they lack a way to regularize the rationale extraction (e.g., to control the sparsity of a text highlight or the number of alignments). In this paper, we present a unified framework for deterministic extraction of structured explanations via constrained inference on a factor graph, forming a differentiable layer. Our approach greatly eases training and rationale regularization, generally outperforming previous work on what comes to performance and plausibility of the extracted rationales. We further provide a comparative study of stochastic and deterministic methods for rationale extraction for classification and natural language inference tasks, jointly assessing their predictive power, quality of the explanations, and model variability.