LLM-as-an-Interviewer: Beyond Static Testing Through Dynamic LLM Evaluation

TL;DR

This paper presents a dynamic, multi-turn LLM evaluation framework called LLM-as-an-Interviewer, which actively interacts with models to assess performance more realistically than static tests.

Contribution

It introduces a novel interactive evaluation paradigm for LLMs that mitigates data contamination and provides detailed insights into model capabilities.

Findings

Effective in evaluating six models on MATH and DepthQA tasks.

Addresses limitations of static evaluation methods like verbosity bias.

Provides comprehensive interview reports with model strengths and weaknesses.

Abstract

We introduce LLM-as-an-Interviewer, a novel paradigm for evaluating large language models (LLMs). This approach leverages multi-turn interactions where the LLM interviewer actively provides feedback on responses and poses follow-up questions to the evaluated LLM. At the start of the interview, the LLM interviewer dynamically modifies datasets to generate initial questions, mitigating data contamination. We apply the LLM-as-an-Interviewer framework to evaluate six models on the MATH and DepthQA tasks. Our results show that the framework effectively provides insights into LLM performance, including the quality of initial responses, adaptability to feedback, and ability to address follow-up queries like clarification or additional knowledge requests. The framework also addresses key limitations of conventional methods like LLM-as-a-Judge, including verbosity bias and inconsistency across runs. Finally, we propose the Interview Report, which aggregates insights from the interview process, providing examples and a comprehensive analysis of the LLM's strengths and weaknesses. This report offers a detailed snapshot of the model's real-world applicability. The code for our framework is publicly available at https://github.com/interview-eval/.