Hierarchical Prompting Taxonomy: A Universal Evaluation Framework for Large Language Models Aligned with Human Cognitive Principles

TL;DR

This paper introduces the Hierarchical Prompting Taxonomy (HPT), a framework based on human cognitive principles, to evaluate large language models' problem-solving abilities and task complexity across diverse datasets.

Contribution

It proposes a novel hierarchical prompting framework and index to assess LLMs' cognitive demands, providing a standardized metric for task complexity and model performance evaluation.

Findings

HPF improves LLM performance by up to 63% over baselines.

GSM8k identified as the most cognitively complex dataset.

HPI effectively measures the cognitive demands of tasks on LLMs.

Abstract

Assessing the effectiveness of large language models (LLMs) in performing different tasks is crucial for understanding their strengths and weaknesses. This paper presents Hierarchical Prompting Taxonomy (HPT), grounded on human cognitive principles and designed to assess LLMs by examining the cognitive demands of various tasks. The HPT utilizes the Hierarchical Prompting Framework (HPF), which structures five unique prompting strategies in a hierarchical order based on their cognitive requirement on LLMs when compared to human mental capabilities. It assesses the complexity of tasks with the Hierarchical Prompting Index (HPI), which demonstrates the cognitive competencies of LLMs across diverse datasets and offers insights into the cognitive demands that datasets place on different LLMs. This approach enables a comprehensive evaluation of an LLMs problem solving abilities and the intricacy of a dataset, offering a standardized metric for task complexity. Extensive experiments with multiple datasets and LLMs show that HPF enhances LLM performance by 2% to 63% compared to baseline performance, with GSM8k being the most cognitively complex task among reasoning and coding tasks with an average HPI of 3.20 confirming the effectiveness of HPT. To support future research and reproducibility in this domain, the implementations of HPT and HPF are available here.