PARALLELPROMPT: Extracting Parallelism from Large Language Model Queries

TL;DR

PARALLELPROMPT introduces a benchmark dataset and evaluation suite to measure and leverage intra-query parallelism in large language models, enabling significant speedups with minimal quality loss.

Contribution

It presents the first benchmark and dataset for intra-query parallelism, along with an evaluation suite to study structure-aware execution in LLM serving systems.

Findings

Over 75% of prompts can be parsed for parallelism

Achieved up to 5x speedups in tasks like translation and comprehension

Minimal quality degradation when using parallel execution strategies

Abstract

LLM serving systems typically treat user prompts as monolithic inputs, optimizing inference through decoding tricks or inter-query batching. However, many real-world prompts contain latent semantic parallelism--decomposable structures where subtasks can be executed independently to reduce latency while preserving meaning. We introduce PARALLELPROMPT, the first benchmark for measuring intra-query parallelism in natural user prompts. Our dataset comprises over 37,000 real-world prompts from public LLM chat logs, each annotated with a structured schema capturing task templates, shared context, and iteration inputs. These schemas are extracted using LLM-assisted prompting with rule-based multilingual validation. To evaluate the benefits of decomposition, we provide an execution suite that benchmarks serial vs. parallel strategies, measuring latency, structural adherence, and semantic fidelity. Our results show that intra-query parallelism can be successfully parsed in over 75% of curated datasets, unlocking up to 5x speedups on tasks like translation, comprehension, and comparative analysis, with minimal quality degradation. By releasing this benchmark, curation pipeline, and evaluation suite, we provide the first standardized testbed for studying structure-aware execution in LLM serving pipelines.