NLPBench: Evaluating Large Language Models on Solving NLP Problems

TL;DR

This paper introduces NLPBench, a comprehensive benchmark dataset for evaluating large language models on NLP problem-solving tasks, revealing varied performance and prompting strategy effects.

Contribution

We created NLPBench, a novel dataset with diverse NLP questions, and systematically evaluated LLMs using advanced prompting techniques to identify strengths and weaknesses.

Findings

Advanced prompts can sometimes reduce LLM performance.

Smaller models like LLAMA-2 struggle with logical reasoning.

LLMs show specific weaknesses in scientific problem-solving.

Abstract

Recent developments in large language models (LLMs) have shown promise in enhancing the capabilities of natural language processing (NLP). Despite these successes, there remains a dearth of research dedicated to the NLP problem-solving abilities of LLMs. To fill the gap in this area, we present a unique benchmarking dataset, NLPBench, comprising 378 college-level NLP questions spanning various NLP topics sourced from Yale University's prior final exams. NLPBench includes questions with context, in which multiple sub-questions share the same public information, and diverse question types, including multiple choice, short answer, and math. Our evaluation, centered on LLMs such as GPT-3.5/4, PaLM-2, and LLAMA-2, incorporates advanced prompting strategies like the chain-of-thought (CoT) and tree-of-thought (ToT). Our study reveals that the effectiveness of the advanced prompting strategies can be inconsistent, occasionally damaging LLM performance, especially in smaller models like the LLAMA-2 (13b). Furthermore, our manual assessment illuminated specific shortcomings in LLMs' scientific problem-solving skills, with weaknesses in logical decomposition and reasoning notably affecting results.