Large Language Models Are Also Good Prototypical Commonsense Reasoners

TL;DR

This paper demonstrates that carefully designed prompts can enable large language models to excel at commonsense reasoning tasks, achieving new state-of-the-art results without fine-tuning.

Contribution

The authors introduce novel prompt engineering techniques, including task relevance and supportive evidence generation, to enhance large language models' reasoning capabilities.

Findings

Achieved new SOTA on ProtoQA with an 8% improvement in Max Answer@1

Surpassed previous models on StrategyQA and CommonsenseQA2.0 by 3% and 1%

Enhanced interpretability through Chain-of-Thought and knowledge prompts

Abstract

Commonsense reasoning is a pivotal skill for large language models, yet it presents persistent challenges in specific tasks requiring this competence. Traditional fine-tuning approaches can be resource-intensive and potentially compromise a model's generalization capacity. Furthermore, state-of-the-art language models like GPT-3.5 and Claude are primarily accessible through API calls, which makes fine-tuning models challenging. To address these challenges, we draw inspiration from the outputs of large models for tailored tasks and semi-automatically developed a set of novel prompts from several perspectives, including task-relevance, supportive evidence generation (e.g. chain-of-thought and knowledge), diverse path decoding to aid the model. Experimental results on ProtoQA dataset demonstrate that with better designed prompts we can achieve the new state-of-art(SOTA) on the ProtoQA leaderboard, improving the Max Answer@1 score by 8%, Max Incorrect@1 score by 4% (breakthrough 50% for the first time) compared to the previous SOTA model and achieved an improvement on StrategyQA and CommonsenseQA2.0 (3% and 1%, respectively). Furthermore, with the generated Chain-of-Thought and knowledge, we can improve the interpretability of the model while also surpassing the previous SOTA models. We hope that our work can provide insight for the NLP community to develop better prompts and explore the potential of large language models for more complex reasoning tasks.