Temporal Text Classification with Large Language Models

TL;DR

This paper systematically evaluates large language models on their ability to date texts across multiple languages, revealing that proprietary models excel especially with few-shot prompting, while fine-tuning enhances open-source models but still lags behind.

Contribution

First comprehensive assessment of LLMs on Temporal Text Classification across languages, comparing proprietary and open-source models with various prompting and fine-tuning methods.

Findings

Proprietary models perform well, especially with few-shot prompting.

Fine-tuning improves open-source models significantly.

Open-source models still lag behind proprietary LLMs in TTC performance.

Abstract

Languages change over time. Computational models can be trained to recognize such changes enabling them to estimate the publication date of texts. Despite recent advancements in Large Language Models (LLMs), their performance on automatic dating of texts, also known as Temporal Text Classification (TTC), has not been explored. This study provides the first systematic evaluation of leading proprietary (Claude 3.5, GPT-4o, Gemini 1.5) and open-source (LLaMA 3.2, Gemma 2, Mistral, Nemotron 4) LLMs on TTC using three historical corpora, two in English and one in Portuguese. We test zero-shot and few-shot prompting, and fine-tuning settings. Our results indicate that proprietary models perform well, especially with few-shot prompting. They also indicate that fine-tuning substantially improves open-source models but that they still fail to match the performance delivered by proprietary LLMs.