HintEval: A Comprehensive Framework for Hint Generation and Evaluation for Questions

TL;DR

HintEval is a comprehensive Python toolkit that consolidates datasets and evaluation methods to advance research in automatic hint generation and assessment for question-answering systems.

Contribution

It introduces a unified framework and library that simplifies access to datasets and evaluation tools for hint generation research, promoting consistency and progress.

Findings

Provides a unified toolkit for hint datasets and evaluation methods

Enables consistent and multi-faceted evaluation of hint quality

Facilitates research and comparison in hint generation methods

Abstract

Large Language Models (LLMs) are transforming how people find information, and many users turn nowadays to chatbots to obtain answers to their questions. Despite the instant access to abundant information that LLMs offer, it is still important to promote critical thinking and problem-solving skills. Automatic hint generation is a new task that aims to support humans in answering questions by themselves by creating hints that guide users toward answers without directly revealing them. In this context, hint evaluation focuses on measuring the quality of hints, helping to improve the hint generation approaches. However, resources for hint research are currently spanning different formats and datasets, while the evaluation tools are missing or incompatible, making it hard for researchers to compare and test their models. To overcome these challenges, we introduce HintEval, a Python library that makes it easy to access diverse datasets and provides multiple approaches to generate and evaluate hints. HintEval aggregates the scattered resources into a single toolkit that supports a range of research goals and enables a clear, multi-faceted, and reliable evaluation. The proposed library also includes detailed online documentation, helping users quickly explore its features and get started. By reducing barriers to entry and encouraging consistent evaluation practices, HintEval offers a major step forward for facilitating hint generation and analysis research within the NLP/IR community.