GUIDEQ: Framework for Guided Questioning for progressive informational collection and classification

TL;DR

GUIDEQ is a novel framework that uses guided questioning based on explainability and classifier outputs to iteratively gather relevant information, significantly improving text classification accuracy in QA systems.

Contribution

This work introduces GUIDEQ, a new framework that leverages explainability and classifier labels to generate targeted questions, enhancing information collection and classification performance.

Findings

GUIDEQ outperforms baseline LLM-based methods in F1-Score.

The framework generates higher quality, relevant questions.

Experimental results show improved classification accuracy.

Abstract

Question Answering (QA) is an important part of tasks like text classification through information gathering. These are finding increasing use in sectors like healthcare, customer support, legal services, etc., to collect and classify responses into actionable categories. LLMs, although can support QA systems, they face a significant challenge of insufficient or missing information for classification. Although LLMs excel in reasoning, the models rely on their parametric knowledge to answer. However, questioning the user requires domain-specific information aiding to collect accurate information. Our work, GUIDEQ, presents a novel framework for asking guided questions to further progress a partial information. We leverage the explainability derived from the classifier model for along with LLMs for asking guided questions to further enhance the information. This further information helps in more accurate classification of a text. GUIDEQ derives the most significant key-words representative of a label using occlusions. We develop GUIDEQ's prompting strategy for guided questions based on the top-3 classifier label outputs and the significant words, to seek specific and relevant information, and classify in a targeted manner. Through our experimental results, we demonstrate that GUIDEQ outperforms other LLM-based baselines, yielding improved F1-Score through the accurate collection of relevant further information. We perform various analytical studies and also report better question quality compared to our method.