Bits of Grass: Does GPT already know how to write like Whitman?

TL;DR

This paper investigates whether GPT-3.5, GPT-3.5-turbo, and GPT-4 can generate poetry in the style of specific authors using zero-shot and few-shot prompts, finding they do not succeed without fine-tuning even with extensive examples.

Contribution

The study provides an automated evaluation of large language models' ability to imitate specific poetic styles without fine-tuning, highlighting current limitations.

Findings

Models do not replicate specific author styles without fine-tuning.

Maximum prompt length does not enable style imitation.

Even with 17 poem examples, style transfer remains ineffective.

Abstract

This study examines the ability of GPT-3.5, GPT-3.5-turbo (ChatGPT) and GPT-4 models to generate poems in the style of specific authors using zero-shot and many-shot prompts (which use the maximum context length of 8192 tokens). We assess the performance of models that are not fine-tuned for generating poetry in the style of specific authors, via automated evaluation. Our findings indicate that without fine-tuning, even when provided with the maximum number of 17 poem examples (8192 tokens) in the prompt, these models do not generate poetry in the desired style.