Rethinking STS and NLI in Large Language Models

TL;DR

This paper evaluates the limitations of large language models in semantic textual similarity and natural language inference tasks, especially in specialized domains, highlighting ongoing challenges in model confidence and capturing human disagreement.

Contribution

It provides a comprehensive assessment of LLMs' performance on STS and NLI in domain-specific contexts and examines their ability to reflect human consensus and confidence.

Findings

LLMs perform poorly in clinical/biomedical STS and NLI tasks.

LLMs exhibit overconfidence and struggle to capture human disagreement.

Existing challenges in STS and NLI persist despite advances in LLMs.

Abstract

Recent years have seen the rise of large language models (LLMs), where practitioners use task-specific prompts; this was shown to be effective for a variety of tasks. However, when applied to semantic textual similarity (STS) and natural language inference (NLI), the effectiveness of LLMs turns out to be limited by low-resource domain accuracy, model overconfidence, and difficulty to capture the disagreements between human judgements. With this in mind, here we try to rethink STS and NLI in the era of LLMs. We first evaluate the performance of STS and NLI in the clinical/biomedical domain, and then we assess LLMs' predictive confidence and their capability of capturing collective human opinions. We find that these old problems are still to be properly addressed in the era of LLMs.