Large Language Models for Semantic Monitoring of Corporate Disclosures: A Case Study on Korea's Top 50 KOSPI Companies

TL;DR

This study evaluates GPT-3.5-turbo and GPT-4's effectiveness in semantically analyzing Korean corporate disclosures, highlighting GPT-4's superior accuracy and potential for automating disclosure monitoring tasks.

Contribution

It provides a comparative analysis of GPT models' performance in semantic sentiment analysis of corporate disclosures in Korea, demonstrating GPT-4's enhanced accuracy over GPT-3.5-turbo.

Findings

GPT-4 outperforms GPT-3.5-turbo in sentiment accuracy

Spearman correlation coefficient of 0.61 for GPT-4

Concordance rate of 0.82 for GPT-4

Abstract

In the rapidly advancing domain of artificial intelligence, state-of-the-art language models such as OpenAI's GPT-3.5-turbo and GPT-4 offer unprecedented opportunities for automating complex tasks. This research paper delves into the capabilities of these models for semantically analyzing corporate disclosures in the Korean context, specifically for timely disclosure. The study focuses on the top 50 publicly traded companies listed on the Korean KOSPI, based on market capitalization, and scrutinizes their monthly disclosure summaries over a period of 17 months. Each summary was assigned a sentiment rating on a scale ranging from 1(very negative) to 5(very positive). To gauge the effectiveness of the language models, their sentiment ratings were compared with those generated by human experts. Our findings reveal a notable performance disparity between GPT-3.5-turbo and GPT-4, with the latter demonstrating significant accuracy in human evaluation tests. The Spearman correlation coefficient was registered at 0.61, while the simple concordance rate was recorded at 0.82. This research contributes valuable insights into the evaluative characteristics of GPT models, thereby laying the groundwork for future innovations in the field of automated semantic monitoring.