SMILE: A Composite Lexical-Semantic Metric for Question-Answering Evaluation

TL;DR

SMILE is a new evaluation metric for question-answering tasks that combines lexical and semantic analysis to better align with human judgment, addressing limitations of existing metrics.

Contribution

It introduces a composite metric that balances lexical exactness with semantic understanding, improving QA evaluation accuracy.

Findings

Highly correlated with human judgments across multiple QA tasks

Computationally lightweight compared to LLM-based evaluators

Effectively balances lexical and semantic evaluation aspects

Abstract

Traditional evaluation metrics for textual and visual question answering, like ROUGE, METEOR, and Exact Match (EM), focus heavily on n-gram based lexical similarity, often missing the deeper semantic understanding needed for accurate assessment. While measures like BERTScore and MoverScore leverage contextual embeddings to address this limitation, they lack flexibility in balancing sentence-level and keyword-level semantics and ignore lexical similarity, which remains important. Large Language Model (LLM) based evaluators, though powerful, come with drawbacks like high costs, bias, inconsistency, and hallucinations. To address these issues, we introduce SMILE: Semantic Metric Integrating Lexical Exactness, a novel approach that combines sentence-level semantic understanding with keyword-level semantic understanding and easy keyword matching. This composite method balances lexical precision and semantic relevance, offering a comprehensive evaluation. Extensive benchmarks across text, image, and video QA tasks show SMILE is highly correlated with human judgments and computationally lightweight, bridging the gap between lexical and semantic evaluation.