Benchmarking zero-shot stance detection with FlanT5-XXL: Insights from training data, prompting, and decoding strategies into its near-SoTA performance

TL;DR

This paper evaluates FlanT5-XXL's zero-shot stance detection on tweets, revealing it can match or surpass fine-tuned models, with performance influenced by prompts, decoding strategies, and inherent biases.

Contribution

It provides a comprehensive analysis of zero-shot stance detection with FlanT5-XXL, highlighting factors affecting performance and ensuring no data leakage from training datasets.

Findings

Zero-shot approach matches or exceeds state-of-the-art benchmarks.

Performance varies with prompts, decoding strategies, and prompt perplexity.

Identifies a positivity bias influencing results.

Abstract

We investigate the performance of LLM-based zero-shot stance detection on tweets. Using FlanT5-XXL, an instruction-tuned open-source LLM, with the SemEval 2016 Tasks 6A, 6B, and P-Stance datasets, we study the performance and its variations under different prompts and decoding strategies, as well as the potential biases of the model. We show that the zero-shot approach can match or outperform state-of-the-art benchmarks, including fine-tuned models. We provide various insights into its performance including the sensitivity to instructions and prompts, the decoding strategies, the perplexity of the prompts, and to negations and oppositions present in prompts. Finally, we ensure that the LLM has not been trained on test datasets, and identify a positivity bias which may partially explain the performance differences across decoding strategie