Matryoshka Pilot: Learning to Drive Black-Box LLMs with LLMs

TL;DR

Matryoshka Pilot (M-Pilot) is a white-box controller that guides black-box LLMs through intermediate prompts, improving their performance on complex, multi-step tasks without needing access to their internal parameters.

Contribution

We propose M-Pilot, a lightweight white-box controller that interacts with black-box LLMs via prompts, enabling better control and self-improvement in complex tasks.

Findings

Enhances black-box LLMs' performance on complex tasks

Enables controllable multi-turn generation

Demonstrates effectiveness through empirical evaluations

Abstract

Despite the impressive generative abilities of black-box large language models (LLMs), their inherent opacity hinders further advancements in capabilities such as reasoning, planning, and personalization. Existing works aim to enhance LLM capabilities via domain-specific adaptation, which require additional training on accessible model parameters, an infeasible option for black-box LLMs. To address this challenge, we introduce Matryoshka Pilot (M-Pilot), a lightweight white-box LLM controller that guides a large-scale black-box LLM generator by decomposing complex tasks into a series of intermediate outputs. Specifically, we consider the black-box LLM as an environment, with M-Pilot serving as a policy to provide intermediate guidance through prompts for driving the black-box LLM. M-Pilot is trained to pivot the outputs of the black-box LLM aligning with preferences during iterative interaction, which enables controllable multi-turn generation and self-improvement in optimizing intermediate guidance. Empirical evaluations on diverse tasks demonstrate that our method effectively enhances the capabilities of black-box LLMs in complex, long-horizon tasks. Our code is publicly available at: https://github.com/lichangh20/Matryoshka.