Still Not There: Can LLMs Outperform Smaller Task-Specific Seq2Seq Models on the Poetry-to-Prose Conversion Task?

TL;DR

This paper evaluates whether large language models can outperform smaller, specialized Sanskrit poetry-to-prose models, finding that domain-specific fine-tuning yields superior results, especially in complex, low-resource language tasks.

Contribution

It introduces a comprehensive comparison between instruction-tuned LLMs and a task-specific fine-tuned Seq2Seq model for Sanskrit poetry-to-prose conversion, highlighting the effectiveness of domain-specific fine-tuning.

Findings

Domain-specific fine-tuning of ByT5-Sanskrit outperforms instruction-driven LLMs.

Prompting strategies serve as effective alternatives when domain data is scarce.

The task-specific model generalizes well to out-of-domain data.

Abstract

Large Language Models (LLMs) are increasingly treated as universal, general-purpose solutions across NLP tasks, particularly in English. But does this assumption hold for low-resource, morphologically rich languages such as Sanskrit? We address this question by comparing instruction-tuned and in-context-prompted LLMs with smaller task-specific encoder-decoder models on the Sanskrit poetry-to-prose conversion task. This task is intrinsically challenging: Sanskrit verse exhibits free word order combined with rigid metrical constraints, and its conversion to canonical prose (anvaya) requires multi-step reasoning involving compound segmentation, dependency resolution, and syntactic linearisation. This makes it an ideal testbed to evaluate whether LLMs can surpass specialised models. For LLMs, we apply instruction fine-tuning on general-purpose models and design in-context learning templates grounded in Paninian grammar and classical commentary heuristics. For task-specific modelling, we fully fine-tune a ByT5-Sanskrit Seq2Seq model. Our experiments show that domain-specific fine-tuning of ByT5-Sanskrit significantly outperforms all instruction-driven LLM approaches. Human evaluation strongly corroborates this result, with scores exhibiting high correlation with Kendall's Tau scores. Additionally, our prompting strategies provide an alternative to fine-tuning when domain-specific verse corpora are unavailable, and the task-specific Seq2Seq model demonstrates robust generalisation on out-of-domain evaluations.