StringLLM: Understanding the String Processing Capability of Large Language Models

TL;DR

This paper systematically evaluates large language models' ability to process strings, introduces datasets for benchmarking, analyzes their limitations, and proposes fine-tuning methods to improve their string processing skills.

Contribution

It presents StringLLM and StringBench for benchmarking string processing in LLMs, and offers insights and methods to enhance their capabilities.

Findings

LLMs struggle with string processing compared to humans

The proposed fine-tuning approach significantly improves LLMs' string capabilities

The study provides a foundation for future research in LLM string processing

Abstract

String processing, which mainly involves the analysis and manipulation of strings, is a fundamental component of modern computing. Despite the significant advancements of large language models (LLMs) in various natural language processing (NLP) tasks, their capability in string processing remains underexplored and underdeveloped. To bridge this gap, we present a comprehensive study of LLMs' string processing capability. In particular, we first propose StringLLM, a method to construct datasets for benchmarking string processing capability of LLMs. We use StringLLM to build a series of datasets, referred to as StringBench. It encompasses a wide range of string processing tasks, allowing us to systematically evaluate LLMs' performance in this area. Our evaluations indicate that LLMs struggle with accurately processing strings compared to humans. To uncover the underlying reasons for this limitation, we conduct an in-depth analysis and subsequently propose an effective approach that significantly enhances LLMs' string processing capability via fine-tuning. This work provides a foundation for future research to understand LLMs' string processing capability. Our code and data are available at https://github.com/wxl-lxw/StringLLM.