NewTerm: Benchmarking Real-Time New Terms for Large Language Models with Annual Updates

TL;DR

This paper introduces NewTerm, an adaptive benchmark for evaluating large language models on real-time new terms, highlighting their performance decline and analyzing challenges in updating models with recent information.

Contribution

We propose a highly automated, annually updated benchmark for real-time new term evaluation, addressing limitations of existing outdated content benchmarks.

Findings

LLMs experience over 20% performance drop on new terms

Knowledge cutoff updates only partially cover new terms

Certain types of new terms are more challenging for LLMs

Abstract

Despite their remarkable abilities in various tasks, large language models (LLMs) still struggle with real-time information (e.g., new facts and terms) due to the knowledge cutoff in their development process. However, existing benchmarks focus on outdated content and limited fields, facing difficulties in real-time updating and leaving new terms unexplored. To address this problem, we propose an adaptive benchmark, NewTerm, for real-time evaluation of new terms. We design a highly automated construction method to ensure high-quality benchmark construction with minimal human effort, allowing flexible updates for real-time information. Empirical results on various LLMs demonstrate over 20% performance reduction caused by new terms. Additionally, while updates to the knowledge cutoff of LLMs can cover some of the new terms, they are unable to generalize to more distant new terms. We also analyze which types of terms are more challenging and why LLMs struggle with new terms, paving the way for future research. Finally, we construct NewTerm 2022 and 2023 to evaluate the new terms updated each year and will continue updating annually. The benchmark and codes can be found at https://github.com/hexuandeng/NewTerm.