Meta Semantic Template for Evaluation of Large Language Models

TL;DR

This paper introduces MSTemp, a novel method for evaluating large language models' semantic understanding by generating out-of-distribution evaluation sets through semantic templates, revealing models' limitations.

Contribution

MSTemp provides a flexible, dynamic, and cost-effective approach to create OOD evaluation datasets for assessing LLMs' semantic comprehension.

Findings

MSTemp-generated samples significantly reduce LLM performance.

The approach effectively tests LLMs beyond existing benchmark datasets.

Initial experiments demonstrate the method's potential for LLM evaluation.

Abstract

Do large language models (LLMs) genuinely understand the semantics of the language, or just memorize the training data? The recent concern on potential data contamination of LLMs has raised awareness of the community to conduct research on LLMs evaluation. In this paper, we propose MSTemp, an approach that creates meta semantic templates to evaluate the semantic understanding ability of LLMs. The core of MSTemp is not to perform evaluation directly on existing benchmark datasets, but to generate new out-of-distribution (OOD) evaluation sets using existing datasets as seeds. Specifically, for a given sentence, MSTemp leverages another language model to generate new samples while preserving its semantics. The new samples are called semantic templates to the original sentence. Then, MSTemp generates evaluation samples via sentence parsing and random word replacement on the semantic templates. MSTemp is highly flexible, dynamic, and cost-effective. Our initial experiments show that MSTemp-generated samples can significantly reduce the performance of LLMs using existing datasets as seeds. We hope this initial work can shed light on future research of LLMs evaluation.