In-Context Ability Transfer for Question Decomposition in Complex QA

TL;DR

This paper introduces ICAT, a method that transfers reasoning abilities to large language models for complex question answering without fine-tuning or manual annotations, by selecting relevant examples from related tasks.

Contribution

ICAT enables in-context transfer of question decomposition skills to LLMs without training or manual annotations, using automated example selection from related data sources.

Findings

ICAT outperforms existing prompt-based methods on various complex QA tasks.

It effectively transfers reasoning abilities without model fine-tuning.

The approach is scalable and adaptable to different complex QA scenarios.

Abstract

Answering complex questions is a challenging task that requires question decomposition and multistep reasoning for arriving at the solution. While existing supervised and unsupervised approaches are specialized to a certain task and involve training, recently proposed prompt-based approaches offer generalizable solutions to tackle a wide variety of complex question-answering (QA) tasks. However, existing prompt-based approaches that are effective for complex QA tasks involve expensive hand annotations from experts in the form of rationales and are not generalizable to newer complex QA scenarios and tasks. We propose, icat (In-Context Ability Transfer) which induces reasoning capabilities in LLMs without any LLM fine-tuning or manual annotation of in-context samples. We transfer the ability to decompose complex questions to simpler questions or generate step-by-step rationales to LLMs, by careful selection from available data sources of related tasks. We also propose an automated uncertainty-aware exemplar selection approach for selecting examples from transfer data sources. Finally, we conduct large-scale experiments on a variety of complex QA tasks involving numerical reasoning, compositional complex QA, and heterogeneous complex QA which require decomposed reasoning. We show that ICAT convincingly outperforms existing prompt-based solutions without involving any model training, showcasing the benefits of re-using existing abilities.