Composable Effect Handling for Programming LLM-integrated Scripts

TL;DR

This paper introduces a composable effect handling approach for LLM-integrated scripts, enhancing modularity and enabling significant performance improvements through separation of workflow logic from effectful operations.

Contribution

It presents a novel method using effect handlers to decouple LLM calls and other effects from script logic, facilitating modularity and performance optimization.

Findings

Achieved up to 10× speedup in a Tree-of-Thoughts case study

Demonstrated improved modularity without sacrificing performance

Showed effectiveness of effect handling in LLM scripting environments

Abstract

Implementing LLM-integrated scripts introduces challenges in modularity and performance, as scripts are often coupled to specific LLM implementations and fail to exploit parallelization opportunities. This paper proposes using composable effect handling to separate workflow logic from effectful operations, such as LLM calls, I/O, and concurrency, enabling modularity without sacrificing the opportunity for performance optimization. By treating these operations as abstract interfaces and discharging them via effect handlers, this paper shows that scripts can achieve significant speedups (e.g., 10$\times$ in a Tree-of-Thoughts case study) without compromising modularity. This paper aims to promote composable effect handling as a programming style for LLM scripting.