No Stupid Questions: An Analysis of Question Query Generation for Citation Recommendation

TL;DR

This paper explores using GPT-4o-mini to generate question queries for citation recommendation, demonstrating that these questions can outperform traditional keyword queries in retrieving relevant scientific articles.

Contribution

It introduces a novel question-based query generation method for citation recommendation and proposes MMR-RBO to select effective questions, enhancing retrieval performance.

Findings

Generated questions sometimes outperform keyword queries in retrieval tasks.

MMR-RBO effectively identifies high-quality question queries.

All question queries produce unique result sets, supporting the idea that no questions are 'stupid'.

Abstract

Existing techniques for citation recommendation are constrained by their adherence to article contents and metadata. We leverage GPT-4o-mini's latent expertise as an inquisitive assistant by instructing it to ask questions which, when answered, could expose new insights about an excerpt from a scientific article. We evaluate the utility of these questions as retrieval queries, measuring their effectiveness in retrieving and ranking masked target documents. In some cases, generated questions ended up being better queries than extractive keyword queries generated by the same model. We additionally propose MMR-RBO, a variation of Maximal Marginal Relevance (MMR) using Rank-Biased Overlap (RBO) to identify which questions will perform competitively with the keyword baseline. As all question queries yield unique result sets, we contend that there are no stupid questions.