Designing LLM Chains by Adapting Techniques from Crowdsourcing Workflows

TL;DR

This paper explores how techniques from crowdsourcing workflows can inform the design of LLM chains, providing a structured approach to decomposing complex tasks for improved performance.

Contribution

It constructs a comprehensive design space for LLM chain development by surveying existing literature and demonstrates the application of crowdsourcing tactics to LLM chaining through three case studies.

Findings

A new design space for LLM chains is proposed.

Crowdsourcing tactics can be adapted to improve LLM chain effectiveness.

Case studies illustrate practical applications and strategies.

Abstract

LLM chains enable complex tasks by decomposing work into a sequence of subtasks. Similarly, the more established techniques of crowdsourcing workflows decompose complex tasks into smaller tasks for human crowdworkers. Chains address LLM errors analogously to the way crowdsourcing workflows address human error. To characterize opportunities for LLM chaining, we survey 107 papers across the crowdsourcing and chaining literature to construct a design space for chain development. The design space covers a designer's objectives and the tactics used to build workflows. We then surface strategies that mediate how workflows use tactics to achieve objectives. To explore how techniques from crowdsourcing may apply to chaining, we adapt crowdsourcing workflows to implement LLM chains across three case studies: creating a taxonomy, shortening text, and writing a short story. From the design space and our case studies, we identify takeaways for effective chain design and raise implications for future research and development.