MedCoG: Maximizing LLM Inference Density in Medical Reasoning via Meta-Cognitive Regulation

TL;DR

MedCoG introduces a meta-cognitive approach for LLMs in medical reasoning, dynamically regulating knowledge use to improve inference efficiency and accuracy, especially on challenging medical benchmarks.

Contribution

This paper presents MedCoG, a novel meta-cognitive agent that enhances LLM reasoning by self-assessing task complexity and knowledge density to optimize inference.

Findings

Achieves 5.5x inference density on medical benchmarks

Effectively filters distractive knowledge to improve accuracy

Meta-cognitive regulation shows significant potential in Oracle study

Abstract

Large Language Models (LLMs) have shown strong potential in complex medical reasoning yet face diminishing gains under inference scaling laws. While existing studies augment LLMs with various knowledge types, it remains unclear how effectively the additional costs translate into accuracy. In this paper, we explore how meta-cognition of LLMs, i.e., their self-awareness of their own knowledge states, can regulate the reasoning process. Specifically, we propose MedCoG, a Medical Meta-Cognition Agent with Knowledge Graph, where the meta-cognitive assessments of task complexity, familiarity, and knowledge density dynamically regulate utilization of procedural, episodic, and factual knowledge. The LLM-centric on-demand reasoning aims to mitigate scaling laws by (1) reducing costs via avoiding indiscriminate scaling, (2) improving accuracy via filtering out distractive knowledge. To validate this, we empirically characterize the scaling curve and introduce inference density to quantify inference efficiency, defined as the ratio of theoretically effective cost to actual cost. Experiments demonstrate the effectiveness and efficiency of MedCoG on five hard sets of medical benchmarks, yielding 5.5x inference density. Furthermore, the Oracle study highlights the significant potential of meta-cognitive regulation.