Human-Centric Evaluation for Foundation Models

TL;DR

This paper introduces a human-centric evaluation framework for foundation models, emphasizing subjective assessments of problem-solving, information quality, and interaction, supported by a large dataset from diverse human-model collaborations.

Contribution

It proposes a novel human-centric evaluation framework and provides a comprehensive dataset from extensive human-model interactions, addressing limitations of traditional objective metrics.

Findings

Grok 3 outperforms other models in human evaluations

Deepseek R1 and Gemini 2.5 show strong performance

OpenAI o3 mini lags behind in human-centric assessments

Abstract

Currently, nearly all evaluations of foundation models focus on objective metrics, emphasizing quiz performance to define model capabilities. While this model-centric approach enables rapid performance assessment, it fails to reflect authentic human experiences. To address this gap, we propose a Human-Centric subjective Evaluation (HCE) framework, focusing on three core dimensions: problem-solving ability, information quality, and interaction experience. Through experiments involving Deepseek R1, OpenAI o3 mini, Grok 3, and Gemini 2.5, we conduct over 540 participant-driven evaluations, where humans and models collaborate on open-ended research tasks, yielding a comprehensive subjective dataset. This dataset captures diverse user feedback across multiple disciplines, revealing distinct model strengths and adaptability. Our findings highlight Grok 3's superior performance, followed by Deepseek R1 and Gemini 2.5, with OpenAI o3 mini lagging behind. By offering a novel framework and a rich dataset, this study not only enhances subjective evaluation methodologies but also lays the foundation for standardized, automated assessments, advancing LLM development for research and practical scenarios. Our dataset link is https://github.com/yijinguo/Human-Centric-Evaluation.