SCORE: Specificity, Context Utilization, Robustness, and Relevance for Reference-Free LLM Evaluation

TL;DR

This paper introduces SCORE, a multi-dimensional, reference-free evaluation framework for LLMs that assesses specificity, robustness, relevance, and context use, addressing gaps in current evaluation methods for high-stakes, domain-specific tasks.

Contribution

The paper proposes a novel multi-dimensional evaluation framework and a curated dataset for systematic, domain-sensitive assessment of LLM outputs in high-stakes settings.

Findings

No single metric fully captures answer quality.

Structured, multi-metric evaluation is essential for high-stakes deployment.

Human evaluation reveals subjectivity in domain-specific assessments.

Abstract

Large language models (LLMs) are increasingly used to support question answering and decision-making in high-stakes, domain-specific settings such as natural hazard response and infrastructure planning, where effective answers must convey fine-grained, decision-critical details. However, existing evaluation frameworks for retrieval-augmented generation (RAG) and open-ended question answering primarily rely on surface-level similarity, factual consistency, or semantic relevance, and often fail to assess whether responses provide the specific information required for domain-sensitive decisions. To address this gap, we propose a multi-dimensional, reference-free evaluation framework that assesses LLM outputs along four complementary dimensions: specificity, robustness to paraphrasing and semantic perturbations, answer relevance, and context utilization. We introduce a curated dataset of 1,412 domain-specific question-answer pairs spanning 40 professional roles and seven natural hazard types to support systematic evaluation. We further conduct human evaluation to assess inter-annotator agreement and alignment between model outputs and human judgments, which highlights the inherent subjectivity of open-ended, domain-specific evaluation. Our results show that no single metric sufficiently captures answer quality in isolation and demonstrate the need for structured, multi-metric evaluation frameworks when deploying LLMs in high-stakes applications.