LLaPa: A Vision-Language Model Framework for Counterfactual-Aware Procedural Planning

TL;DR

LLaPa is a multimodal procedural planning framework that integrates vision and language models with counterfactual reasoning modules, significantly improving the quality and correctness of generated action plans in embodied AI tasks.

Contribution

We introduce LLaPa, a novel vision-language framework with auxiliary modules for task-environment alignment and counterfactual reasoning, advancing multimodal procedural planning capabilities.

Findings

LLaPa outperforms existing models on ActPlan-1K and ALFRED benchmarks.

LLaPa achieves higher LCS scores and plan correctness.

The auxiliary modules enhance reasoning and task alignment in planning.

Abstract

While large language models (LLMs) have advanced procedural planning for embodied AI systems through strong reasoning abilities, the integration of multimodal inputs and counterfactual reasoning remains underexplored. To tackle these challenges, we introduce LLaPa, a vision-language model framework designed for multimodal procedural planning. LLaPa generates executable action sequences from textual task descriptions and visual environmental images using vision-language models (VLMs). Furthermore, we enhance LLaPa with two auxiliary modules to improve procedural planning. The first module, the Task-Environment Reranker (TER), leverages task-oriented segmentation to create a task-sensitive feature space, aligning textual descriptions with visual environments and emphasizing critical regions for procedural execution. The second module, the Counterfactual Activities Retriever (CAR), identifies and emphasizes potential counterfactual conditions, enhancing the model's reasoning capability in counterfactual scenarios. Extensive experiments on ActPlan-1K and ALFRED benchmarks demonstrate that LLaPa generates higher-quality plans with superior LCS and correctness, outperforming advanced models. The code and models are available https://github.com/sunshibo1234/LLaPa.