Evaluating LLMs Across Multi-Cognitive Levels: From Medical Knowledge Mastery to Scenario-Based Problem Solving

TL;DR

This study introduces a multi-cognitive-level evaluation framework based on Bloom's Taxonomy to assess LLMs' medical knowledge and problem-solving abilities, revealing performance drops at higher cognitive levels.

Contribution

It proposes a novel multi-cognitive-level evaluation framework for medical LLMs and systematically assesses multiple models across these levels, highlighting performance challenges.

Findings

Performance declines as cognitive complexity increases.

Model size impacts higher-level cognitive performance more.

Insights for improving LLMs in real-world medical tasks.

Abstract

Large language models (LLMs) have demonstrated remarkable performance on various medical benchmarks, but their capabilities across different cognitive levels remain underexplored. Inspired by Bloom's Taxonomy, we propose a multi-cognitive-level evaluation framework for assessing LLMs in the medical domain in this study. The framework integrates existing medical datasets and introduces tasks targeting three cognitive levels: preliminary knowledge grasp, comprehensive knowledge application, and scenario-based problem solving. Using this framework, we systematically evaluate state-of-the-art general and medical LLMs from six prominent families: Llama, Qwen, Gemma, Phi, GPT, and DeepSeek. Our findings reveal a significant performance decline as cognitive complexity increases across evaluated models, with model size playing a more critical role in performance at higher cognitive levels. Our study highlights the need to enhance LLMs' medical capabilities at higher cognitive levels and provides insights for developing LLMs suited to real-world medical applications.