WR-ONE2SET: Towards Well-Calibrated Keyphrase Generation

TL;DR

This paper introduces WR-ONE2SET, a novel approach to keyphrase generation that reduces calibration errors by adaptively weighting training instances and re-assigning targets, leading to more accurate and reliable keyphrase predictions.

Contribution

It proposes WR-ONE2SET, an extension of ONE2SET, with adaptive weighting and target re-assignment mechanisms to improve calibration and reduce over-estimation of no-keyphrase tokens.

Findings

Significantly reduces calibration errors in keyphrase generation.

Improves the accuracy of keyphrase predictions across datasets.

Demonstrates the effectiveness and generality of the proposed methods.

Abstract

Keyphrase generation aims to automatically generate short phrases summarizing an input document. The recently emerged ONE2SET paradigm (Ye et al., 2021) generates keyphrases as a set and has achieved competitive performance. Nevertheless, we observe serious calibration errors outputted by ONE2SET, especially in the over-estimation of $\varnothing$ token (means "no corresponding keyphrase"). In this paper, we deeply analyze this limitation and identify two main reasons behind: 1) the parallel generation has to introduce excessive $\varnothing$ as padding tokens into training instances; and 2) the training mechanism assigning target to each slot is unstable and further aggravates the $\varnothing$ token over-estimation. To make the model well-calibrated, we propose WR-ONE2SET which extends ONE2SET with an adaptive instance-level cost Weighting strategy and a target Re-assignment mechanism. The former dynamically penalizes the over-estimated slots for different instances thus smoothing the uneven training distribution. The latter refines the original inappropriate assignment and reduces the supervisory signals of over-estimated slots. Experimental results on commonly-used datasets demonstrate the effectiveness and generality of our proposed paradigm.