RLAP: A Reinforcement Learning Enhanced Adaptive Planning Framework for Multi-step NLP Task Solving

TL;DR

RLAP introduces a reinforcement learning-based adaptive planning framework that models multi-step NLP tasks as MDPs, enabling LLMs to determine optimal subtask sequences by considering linguistic features, improving task performance.

Contribution

The paper presents a novel RL-enhanced adaptive planning framework that models NLP tasks as MDPs and trains an Actor to optimize subtask ordering based on linguistic features.

Findings

RLAP outperforms baseline methods on multiple NLP datasets.

Incorporating linguistic features improves subtask sequencing.

RLAP demonstrates robustness across different NLP tasks.

Abstract

Multi-step planning has been widely employed to enhance the performance of large language models (LLMs) on downstream natural language processing (NLP) tasks, which decomposes the original task into multiple subtasks and guide LLMs to solve them sequentially without additional training. When addressing task instances, existing methods either preset the order of steps or attempt multiple paths at each step. However, these methods overlook instances' linguistic features and rely on the intrinsic planning capabilities of LLMs to evaluate intermediate feedback and then select subtasks, resulting in suboptimal outcomes. To better solve multi-step NLP tasks with LLMs, in this paper we propose a Reinforcement Learning enhanced Adaptive Planning framework (RLAP). In our framework, we model an NLP task as a Markov decision process (MDP) and employ an LLM directly into the environment. In particular, a lightweight Actor model is trained to estimate Q-values for natural language sequences consisting of states and actions through reinforcement learning. Therefore, during sequential planning, the linguistic features of each sequence in the MDP can be taken into account, and the Actor model interacts with the LLM to determine the optimal order of subtasks for each task instance. We apply RLAP on three different types of NLP tasks and conduct extensive experiments on multiple datasets to verify RLAP's effectiveness and robustness.