Selecting Better Samples from Pre-trained LLMs: A Case Study on Question Generation

TL;DR

This paper introduces prompt-based methods for selecting high-quality questions generated by large language models, improving diversity and quality without requiring model modifications or human references.

Contribution

It proposes novel prompt-based selection techniques for LLM outputs under black-box and reference-free constraints, validated through empirical and human evaluations.

Findings

Selected questions are of higher quality than greedy outputs.

The approach works effectively without model modifications.

Both automatic and human evaluations confirm improvements.

Abstract

Large Language Models (LLMs) have in recent years demonstrated impressive prowess in natural language generation. A common practice to improve generation diversity is to sample multiple outputs from the model. However, there lacks a simple and robust way of selecting the best output from these stochastic samples. As a case study framed in the context of question generation, we propose two prompt-based approaches to selecting high-quality questions from a set of LLM-generated candidates. Our method works under the constraints of 1) a black-box (non-modifiable) question generation model and 2) lack of access to human-annotated references -- both of which are realistic limitations for real-world deployment of LLMs. With automatic as well as human evaluations, we empirically demonstrate that our approach can effectively select questions of higher qualities than greedy generation.