Learning to Plan and Generate Text with Citations
Constanza Fierro, Reinald Kim Amplayo, Fantine Huot, Nicola De Cao,, Joshua Maynez, Shashi Narayan, Mirella Lapata
TL;DR
This paper investigates plan-based models for generating verifiable, well-attributed text with citations, demonstrating that planning enhances attribution quality and citation accuracy in long-form question-answering tasks.
Contribution
It introduces two novel attribution models utilizing question-based blueprints, improving the faithfulness and accuracy of generated citations compared to non-planning approaches.
Findings
Planning improves attribution quality in generated text.
Blueprint models produce more accurate citations.
Experiments confirm the effectiveness of plan-based generation.
Abstract
The increasing demand for the deployment of LLMs in information-seeking scenarios has spurred efforts in creating verifiable systems, which generate responses to queries along with supporting evidence. In this paper, we explore the attribution capabilities of plan-based models which have been recently shown to improve the faithfulness, grounding, and controllability of generated text. We conceptualize plans as a sequence of questions which serve as blueprints of the generated content and its organization. We propose two attribution models that utilize different variants of blueprints, an abstractive model where questions are generated from scratch, and an extractive model where questions are copied from the input. Experiments on long-form question-answering show that planning consistently improves attribution quality. Moreover, the citations generated by blueprint models are more…
Peer Reviews
Decision·Submitted to ICLR 2024
The proposed method is decent and achieves good performance, and the proposed abstraction and extractive modules are demonstrated performant.
1. The major concern is the lack of novelty in the proposed method. The question generation modules are not new. 2. In the experiments on the ALCE benchmark, the correctness score falls far behind the LLM-based few-shot prompting methods. 3. The writing of the paper should be significantly improved. There are too many redundant narratives but less focus on the overall technical contribution.
This is an interesting paper that addresses a really important and challenging problem -- answering complex questions with citations. The citation piece is an excellent step towards enabling people to verify for themselves what LLMs seem to be telling them. The paper itself is easy to read, and the experimental results are rich and interesting.
I don't have weaknesses to point out per se. I am supportive of publishing the paper.
- The paper introduces an interesting approach to incorporating attribution (i.e generating citations) into text generation models through blueprints, which can potentially improve the trustworthiness and quality of generated content. - This work also evaluates model performance in transfer settings, demonstrating the robustness and generalization capabilities of the blueprint models.
- The paper is hard to follow, referencing other papers and models without providing detailed contextual information, making it challenging for readers unfamiliar with these works to fully understand the content. - It is also not clear what are the main contributions of the model wrt Narayan et al (2023). It seems that the main differences are just the formulation of the blueprints as a set of questions rather than the Question-Answer pairs and re-ranking of passages before generation. It would
Videos
Taxonomy
TopicsAdvanced Text Analysis Techniques · Intelligent Tutoring Systems and Adaptive Learning