Enhancing Zero-Shot Crypto Sentiment with Fine-tuned Language Model and Prompt Engineering

TL;DR

This paper enhances zero-shot cryptocurrency sentiment analysis by fine-tuning large language models and employing prompt engineering, resulting in significant accuracy improvements and insights into instruction tuning effects.

Contribution

It introduces a novel approach combining supervised and instruction-based fine-tuning for large language models to improve cryptocurrency sentiment analysis accuracy.

Findings

Fine-tuning yields a 40% average zero-shot performance gain.

Larger models achieve up to 75.16% accuracy with instruction tuning.

Short, simple instructions lead to 72.38% accuracy, outperforming complex instructions.

Abstract

Blockchain technology has revolutionized the financial landscape, with cryptocurrencies gaining widespread adoption for their decentralized and transparent nature. As the sentiment expressed on social media platforms can significantly influence cryptocurrency discussions and market movements, sentiment analysis has emerged as a crucial tool for understanding public opinion and predicting market trends. Motivated by the aim to enhance sentiment analysis accuracy in the cryptocurrency domain, this paper investigates fine-tuning techniques on large language models. This paper also investigates the efficacy of supervised fine-tuning and instruction-based fine-tuning on large language models for unseen tasks. Experimental results demonstrate a significant average zero-shot performance gain of 40% after fine-tuning, highlighting the potential of this technique in optimizing pre-trained language model efficiency. Additionally, the impact of instruction tuning on models of varying scales is examined, revealing that larger models benefit from instruction tuning, achieving the highest average accuracy score of 75.16%. In contrast, smaller-scale models may experience reduced generalization due to the complete utilization of model capacity. To gain deeper insight about how instruction works with these language models, this paper presents an experimental investigation into the response of an instruction-based model under different instruction tuning setups. The investigation demonstrates that the model achieves an average accuracy score of 72.38% for short and simple instructions. This performance significantly outperforms its accuracy under long and complex instructions by over 12%, thereby effectively highlighting the profound significance of instruction characteristics in maximizing model performance.