CogVLA: Cognition-Aligned Vision-Language-Action Model via Instruction-Driven Routing & Sparsification

TL;DR

CogVLA introduces a novel, efficient vision-language-action framework that uses instruction-driven routing and sparsification, achieving state-of-the-art results with reduced computational costs in robotic and benchmark tasks.

Contribution

The paper presents CogVLA, a new architecture that aligns cognition with vision-language-action tasks through instruction-driven routing and token sparsification, improving efficiency and performance.

Findings

Achieves 97.4% success rate on LIBERO benchmark.

Reduces training costs by 2.5 times.

Decreases inference latency by 2.8 times.

Abstract

Recent Vision-Language-Action (VLA) models built on pre-trained Vision-Language Models (VLMs) require extensive post-training, resulting in high computational overhead that limits scalability and deployment.We propose CogVLA, a Cognition-Aligned Vision-Language-Action framework that leverages instruction-driven routing and sparsification to improve both efficiency and performance. CogVLA draws inspiration from human multimodal coordination and introduces a 3-stage progressive architecture. 1) Encoder-FiLM based Aggregation Routing (EFA-Routing) injects instruction information into the vision encoder to selectively aggregate and compress dual-stream visual tokens, forming a instruction-aware latent representation. 2) Building upon this compact visual encoding, LLM-FiLM based Pruning Routing (LFP-Routing) introduces action intent into the language model by pruning instruction-irrelevant visually grounded tokens, thereby achieving token-level sparsity. 3) To ensure that compressed perception inputs can still support accurate and coherent action generation, we introduce V-L-A Coupled Attention (CAtten), which combines causal vision-language attention with bidirectional action parallel decoding. Extensive experiments on the LIBERO benchmark and real-world robotic tasks demonstrate that CogVLA achieves state-of-the-art performance with success rates of 97.4% and 70.0%, respectively, while reducing training costs by 2.5-fold and decreasing inference latency by 2.8-fold compared to OpenVLA. CogVLA is open-sourced and publicly available at https://github.com/JiuTian-VL/CogVLA.

Tables

Table 1. Table 1 : Simulation Experimental Results. Comparison of task success rates (SR) and their ranks (RK) on the LIBERO benchmark across four task types. “†” indicates our reproduced results.

Method	Spatial		Object		Goal		Long		Average
	SR $\uparrow$	RK $\downarrow$	SR $\uparrow$	RK $\downarrow$	SR $\uparrow$	RK $\downarrow$	SR $\uparrow$	RK $\downarrow$	SR $\uparrow$	RK $\downarrow$
Diffusion Policy [RSS’23] Chi et al. (2023)	78.3	13	92.5	9	68.3	13	50.5	13	72.4	13
Octo fine-tuned [RSS’24] Team et al. (2024)	78.9	12	85.7	13	84.6	10	51.1	12	75.1	12
OpenVLA [CoRL’24] Kim et al. (2024a)	84.7	10	88.4	12	79.2	11	53.7	11	76.5	11
${\pi }_0$ fine-tuned [RSS’25] Black et al.	96.8	4	98.8	1	95.8	3	85.2	6	94.2	6
${\pi }_0$-Fast [RSS’25] Pertsch et al. (2025)	96.4	5	96.8	6	88.6	7	60.2	9	85.5	7
${\pi }_{0.5}$-KI [arXiv’25] Driess et al. (2025)	98.0	2	97.8	5	95.6	4	85.8	5	94.3	4
OpenVLA-OFT [RSS’25] Kim et al. (2025)	97.6	3	98.4	3	97.9	1	94.5	2	97.1	2
SpatialVLA [RSS’25] Qu et al. (2025)	88.2	8	89.9	11	78.6	12	55.5	10	78.1	10
PD-VLA† [arXiv’25] Song et al. (2025)	95.5	6	96.7	7	94.9	6	91.7	3	94.7	3
STAR [ICML’25] Hao et al. (2025)	95.5	6	98.3	4	95.0	5	88.5	4	94.3	4
Dita [arXiv’25] Hou et al. (2025)	84.2	11	96.3	8	85.4	9	63.8	8	82.4	9
CoT-VLA [CVPR’25] Zhao et al. (2025)	87.5	9	91.6	10	87.6	8	69.0	7	83.9	8
CogVLA	98.6	1	98.8	1	96.6	2	95.4	1	97.4	1

Table 2. Table 2 : Real-world Experimental Results. Performance comparison on the Cobot Agilex ALOHA tasks. “†” indicates our reproduced results, while “*” denotes results reported in the original paper.

Method	Object Placement		Drawer Manipulation			T-shirt Folding			Average
	Cube$\to$Plate	+Toy$\to$Bowl	Open	+Place	+Close	Step 1	+Step 2	+Step 3	SR
VQ-BeT Lee et al. (2024)*	5/10	3/10	4/10	3/10	1/10	-	-	-	20.0%
QueST Mete et al. (2024)*	6/10	4/10	3/10	1/10	0/10	-	-	-	20.0%
STAR* Hao et al. (2025)	8/10	6/10	6/10	4/10	3/10	-	-	-	45.0%
PD-VLA† Song et al. (2025)	8/10	7/10	6/10	6/10	4/10	7/10	6/10	4/10	50.0%
OpenVLA-OFT† Kim et al. (2025)	8/10	7/10	8/10	6/10	5/10	7/10	7/10	5/10	56.7%
CogVLA	9/10	8/10	8/10	7/10	7/10	9/10	8/10	6/10	70.0%

Table 3. Table 3 : Efficiency Optimization Results. CogVLA maintains superior performance while achieving the highest efficiency. Ablation studies on Stage 1 and Stage 2 validate the efficiency contribution of each routing module. “†” indicates our reproduced results.

Method	Inference Time $\downarrow$	Throughput $\uparrow$	FLOPs $\downarrow$	Taining Cost $\downarrow$	LIBERO SR $\uparrow$
OpenVLA† Kim et al. (2024a)	0.254 s	3.9 Hz	8.48 T	11.7 h/10k steps	76.5%
OpenVLA-OFT† Kim et al. (2025)	0.132 s	60.6 Hz	8.45 T	12.5 h/10k steps	97.1%
PD-VLA† Song et al. (2025)	0.143 s	55.9 Hz	8.48 T	11.7 h/10k steps	94.7%
CogVLA	0.091 s	87.9 Hz	2.72 T	4.7 h/10k steps	97.4%
w/o Stage 1	0.162 s	49.4 Hz	5.38 T	8.4 h/10k steps	-
w/o Stage 2	0.117 s	68.4 Hz	3.52 T	5.3 h/10k steps	-

Table 4. Table 4 : Ablation study on model components. Pruning and TG-Pruning denote LFP-Routing w/o and w/ instruction guidance. All ablations maintain a fixed 8× overall sparsification ratio.

Stage 1		Stage 2		Stage 3	Spatial SR
Step 1	Step 2	Pruning	TG-Pruning
		✓	✓	✓	91.2 (-7.4)
	✓	✓	✓	✓	96.0 (-2.6)
✓		✓	✓	✓	95.2 (-3.4)
✓	✓			✓	92.0 (-6.6)
✓	✓	✓		✓	96.2 (-2.4)
✓	✓	✓	✓		92.0 (-6.6)
✓	✓	✓	✓	✓	98.6

Table 5. Table 7 : Multi-seed evaluation results in simulation. Task success rates (SR) are compared across four task categories on the LIBERO benchmark. “†” denotes our reproduced results. CogVLA demonstrates strong and consistent performance.

Method	Spatial	Object	Goal	Long	Average
	SR $\uparrow$	SR $\uparrow$	SR $\uparrow$	SR $\uparrow$	SR $\uparrow$	RK $\downarrow$
OpenVLA [CoRL’24] Kim et al. (2024a)	84.7 $\pm$ 0.9	88.4 $\pm$ 0.8	79.2 $\pm$ 1.0	53.7 $\pm$ 1.3	76.5 $\pm$ 0.6	5
SpatialVLA [RSS’25] Qu et al. (2025)	88.2 $\pm$ 0.5	89.9 $\pm$ 0.7	78.6 $\pm$ 0.6	55.5 $\pm$ 1.0	78.1 $\pm$ 0.7	4
STAR [ICML’25] Hao et al. (2025)	95.5 $\pm$ 0.6	98.3 $\pm$ 0.2	95.0 $\pm$ 0.7	88.5 $\pm$ 0.3	94.3 $\pm$ 0.1	2
CoT-VLA [CVPR’25] Zhao et al. (2025)	87.5 $\pm$ 1.4	91.6 $\pm$ 0.5	87.6 $\pm$ 0.6	69.0 $\pm$ 0.8	83.9 $\pm$ 0.6	3
CogVLA	98.5$\pm$ 0.5	98.8$\pm$ 0.4	96.5$\pm$ 0.6	95.2$\pm$ 1.1	97.4$\pm$ 0.4	1

Table 6. Table 8 : Extended real-world results on Tasks 4–5. Performance comparison on the Cobot Agilex ALOHA tasks. “†” indicates our reproduced results.

Method	Task 4		Task 5	Average
	Red Cube$\to$Plate	+Big Cube$\to$Bowl	Left Cube$\to$Plate	SR
PD-VLA† Song et al. (2025)	7/10	5/10	6/10	60.0%
OpenVLA-OFT† Kim et al. (2025)	7/10	6/10	6/10	63.3%
CogVLA	8/10	7/10	8/10	76.7%

Table 7. Table 9 : Supplementary ablation on sparsification ratio allocation. Spf.Ratio denotes the sparsification ratio, which can be adjusted based on the performance–efficiency trade-off. CogVLA achieves better performance when a relatively higher sparsification ratio is allocated to Stage 1 compared to Stage 2.

Stage 1	Stage 2	Spf.Ratio	Spatial SR	FLOPs
2 $\times$	2 $\times$	4 $\times$	96.4 (-2.2)	3.87 T
4 $\times$	4 $\times$	16 $\times$	93.2 (-5.4)	2.30 T
2 $\times$	4 $\times$	8 $\times$	94.6 (-4.0)	2.72 T
4 $\times$	2 $\times$	8 $\times$	98.6	2.72 T

Equations

$$\mathbf{A}=[{\mathbf{a}}_0,{\mathbf{a}}_1,\dots ,{\mathbf{a}}_{K-1}]\in {\mathbb{R}}^{K\times D}$$

$${\mathbf{a}}_i=[{\mathbf{a}}_i^{(1)},{\mathbf{a}}_i^{(2)},\cdots ,{\mathbf{a}}_i^{(D)}{]}^{\top },{\mathbf{a}}_i^{(k)}=f_{\text{AR}}([\mathbf{X},\{{\mathbf{a}}_{\tau }{\}}_{\tau <i},{\mathbf{a}}_i^{(1:k-1)}])$$

$$\tilde{\mathbf{X}}=[\mathbf{X},0_0,0_1,\dots ,0_{K-1}]\in {\mathbb{R}}^{M+T+K\times D}$$

$$\mathbf{A}=f_{\text{parallel}}(\tilde{\mathbf{X}})$$

$${\mathbf{v}}_{\text{agg}}^{(i)}={\text{Encoder-FiLM}}_i({\mathbf{I}}^{(i)},{\mathbf{v}}_{\text{agg}}^{(i)},{\mathbf{t}}_r),i=1,...,N$$

$${\mathbf{v}}_{\text{agg}}=\sum _{i=1}^M{\alpha }_i\cdot {\mathbf{v}}_{\text{agg}}^{(i)}$$

$$\mathbf{\alpha }=[{\alpha }_1,\dots ,{\alpha }_N{]}^{\top }=\text{Softmax}\left({\text{MLP}}_{\text{route}}({\mathbf{t}}_r)\right)$$

$${\mathbf{Z}}_{l+1}=\text{CAtten}(\text{LFP-Routing}({\mathbf{Z}}_l,{\mathbf{t}}_l))$$

$${\mathbf{A}}_t=f_{\text{parallel}}(\tilde{\mathbf{X}})=f_{\text{parallel}}([{\mathbf{Z}}_0,{\mathbf{t}}_0,0_0,0_1,\dots ,0_{K-1}])$$

$$f_{\text{FA}}({\mathbf{I}}^{(i)},{\mathbf{v}}_{\text{agg}}^{(i)},{\mathbf{t}}_r)$$

$${\mathbf{v}}_{\text{agg}}^{(i)}$$

$$\alpha =\text{Sigmoid}(W_2(\sigma (W_1{\mathbf{t}}_r+{\mathbf{b}}_1))+{\mathbf{b}}_2)$$

$${\mathbf{v}}_{\text{agg}}=\alpha \cdot {\mathbf{v}}_{\text{agg}}^{\text{SigLIP}}+(1-\alpha )\cdot {\mathbf{v}}_{\text{agg}}^{\text{DINOv2}}$$

$$f_{\text{FP}}({\mathbf{Z}}_l,{\mathbf{t}}_l)$$

$${\mathbf{Z}}_{l+1}$$

$$R_l^j=\text{MLP}({\mathbf{Z}}_l^j)$$

$${\mathbf{Z}}_{l+1}^j=\{\begin{array}{ll}{R}_l^j\times f_{\text{SF}}([{\mathbf{Z}}_l^j,{\mathbf{t}}_l])+{\mathbf{Z}}_l^j,& \text{if }{R}_l^j>P_l^{\beta }\\ {\mathbf{Z}}_l^j,& \text{otherwise}\end{array}$$

$$\tilde{\mathbf{X}}=[{\mathbf{Z}}_l,{\mathbf{t}}_l,{\mathbf{A}}_l]\in {\mathbb{R}}^{M+T+K\times D}$$

$${\text{Attn}}_{\text{VL}}([{\mathbf{Z}}_l,{\mathbf{t}}_l])=\text{Softmax}\left(\frac{[{\mathbf{Z}}_l,{\mathbf{t}}_l][{\mathbf{Z}}_l,{\mathbf{t}}_l{]}^{\top }}{\sqrt{d}}+{\mathbf{M}}_{\text{causal}}^{\text{VL}}\right)[{\mathbf{Z}}_l,{\mathbf{t}}_l]$$

$${\text{Attn}}_{\text{act}}({\mathbf{A}}_l)=\text{Softmax}\left(\frac{{\mathbf{A}}_l{\mathbf{A}}_l^{\top }}{\sqrt{d}}+{\mathbf{M}}_{\text{bi}}^{\text{act}}\right){\mathbf{A}}_l$$

$$\text{CAtten}(\tilde{\mathbf{X}})=\text{Softmax}\left(\frac{\tilde{\mathbf{X}}{\tilde{\mathbf{X}}}^{\top }}{\sqrt{d}}+{\mathbf{M}}_{\text{hybrid}}\right)\tilde{\mathbf{X}}$$

$${\mathbf{M}}_{\text{hybrid}}=\left[\begin{array}{ccc}{\mathbf{M}}_{\text{causal}}^{\text{VL}}& -\infty & -\infty \\ 0& 0& -\infty \\ 0& 0& {\mathbf{M}}_{\text{bi}}^{\text{act}}\end{array}\right]$$

$${\beta }_l=\frac{1}{2}\cos \frac{\pi l}{L}+\eta ,l=1,2,\cdots ,L$$

Full text

1. Encoder-FiLM based Aggregation Routing (EFA-Routing) injects instruction information into the vision encoder to selectively aggregate and compress dual-stream visual tokens, forming a instruction-aware latent representation.
2. Building upon this compact visual encoding, LLM-FiLM based Pruning Routing (LFP-Routing) introduces action intent into the language model by pruning instruction-irrelevant visually grounded tokens, thereby achieving token-level sparsity.
3. To ensure that compressed perception inputs can still support accurate and coherent action generation, we introduce V‑L‑A Coupled Attention (CAtten), which combines causal vision-language attention with bidirectional action parallel decoding. Extensive experiments on the LIBERO benchmark and real-world robotic tasks demonstrate that CogVLA achieves state-of-the-art performance with success rates of 97.4% and 70.0%, respectively, while reducing training costs by 2.5$\times$ and decreasing inference latency by 2.8$\times$ compared to OpenVLA.

1 Introduction

Vision-Language Action (VLA)Zhou et al. (2025b); Li et al. (2024a); Kim et al. (2024a); Black et al. ; Zhao et al. (2025); Zhong et al. (2025) research has advanced rapidly, fueled by the rich visual and linguistic representations provided by powerful pre-trained Vision-Language Models (VLMs)Li et al. (2023); Alayrac et al. (2022); Liu et al. (2024a); Grattafiori et al. (2024); Bai et al. ; Liu et al. (2023b). Leveraging these foundational models, the VLA paradigm is progressing toward end-to-end robotic control and embodied intelligence, enabling agents to comprehend natural language instructions, perceive complex scenes, and perform manipulation tasks with minimal task-specific engineering. Pioneering works such as RT-2 Brohan et al. (2023), Octo Team et al. (2024), OpenVLA Kim et al. (2024a), ${\pi}_{0}$ Black et al. , and ${\pi}_{0.5}$ Intelligence et al. (2025) have demonstrated the potential of this paradigm.

However, aligning the high-dimensional multimodal features output by VLMs with continuous action spaces remains computationally expensive Zhou et al. (2025a); Wu et al. (2023); Cheang et al. (2024); Bharadhwaj et al. (2024); Kim et al. (2024a). Standard fine-tuning and joint training procedures often entail substantial memory consumption, high FLOPs overhead, and extended training times, severely limiting scalability and practical deployment on resource-constrained platforms. For instance, fine-tuning a 7B VLA model Kim et al. (2025) with action chunking on a single-task dataset from the LIBERO benchmark Liu et al. (2023a) consumes over 600 GPU hours (using 80G A100 GPUs), incurring significant computational costs. Although techniques such as Mixture-of-Depths Raposo et al. (2024); Zhang et al. (2024a); Luo et al. (2024), layer skipping Zhang et al. (2025b); Wu et al. (2024), and early exit Yue et al. (2024); Del Corro et al. (2023) have been proposed to sparsify and accelerate the model training and inference, these methods primarily focus on computation optimization within language models, overlooking the semantic coupling across perception, language alignment, and action decoding. This modular optimization paradigm often leads to cross-modal semantic degradation, manifesting as follows: i) visual compression within encoders discards task-relevant fine-grained features, ii) token skipping within LLMs disrupts the contextual coherence necessary for reference resolution, and iii) action generation lacks causal reasoning over multimodal state transitions.

From a cognitive science perspective Schulze Buschoff et al. (2025b); Lake et al. (2017b), humans exhibit a highly optimized and efficient multimodal coordination mechanism during manipulation. For example, when receiving the instruction "place the red cup at the corner of the table," the human Visual and Attention System (VAS) selectively focuses Bramley et al. (2018); Zhao (2024) on the color attributes of the cup and the spatial structure of the table. Concurrently, the Supplementary Motor Area (SMA) injects task-relevant action intentions Smith et al. (2018) derived from key semantic associations (e.g., “red–cup–corner”) into the visual processing stream, while the Premotor Cortex (PMC) dynamically integrates both visual and linguistic information to plan coherent motion trajectories. This organic unification of perception, reasoning, and control results in remarkable task efficiency. Inspired by this, we propose CogVLA, a Cognition-Aligned Vision-Language-Action framework based on Instruction-Driven Routing & Sparsification, as shown in Fig. 1 (f). Unlike existing modular pipelines, CogVLA establishes a task-semantic-consistent joint optimization mechanism across vision, language, and action modalities, reinforcing cross-modal coherence while improving computational efficiency.

Specifically, CogVLA adopts a 3-stage progressive design to jointly enhance computational efficiency and task performance, as shown in Fig. 1 (d) and (e): 1) Encoder-FiLM based Aggregation Routing (EFA-Routing): To alleviate visual information redundancy and achieve VAS-like visual focus, EFA-Routing compresses visual tokens to 25% of the original input scale, guided by task-specific instructions. This process begins by dynamically encoding the instruction into modulation parameters that guide the aggregation of visual tokens within the visual encoder. Subsequently, the outputs from different encoder branches are adaptively fused to produce cross-branch representations that are semantically aligned with the given task. 2) LLM-FiLM based Pruning Routing (LFP-Routing): Building upon the aggregated visual encoding, LFP-Routing learns a novel, instruction-aware sparsity pattern to prune visual tokens within the language model. By emulating the functionality of SMA, which injects action intentions into visual features, the mechanism selectively skips attention computations over 50% of task-irrelevant tokens. As a result, it significantly reduces the computational burden of the language model and effectively minimizes latency in action generation. 3) V-L-A Coupled Attention (CAtten): To ensure that the compressed visual inputs retain the capacity to support accurate and coherent action sequence, CAtten introduces a coupled attention mechanism inspired by PMC: i) Cross-modal causal attention is applied between the V-L-A interaction layer to preserve temporal reasoning capabilities; ii) Unidirectional attention is employed within the V-L layer to ensure semantic consistency, where visual features have been pre-enhanced with task-specific language intent; iii) Bidirectional attention is utilized within the Action layer to enhance the coherence of action sequences and enable efficient parallel decoding.

We conduct comprehensive evaluations of CogVLA on the LIBERO benchmark and real-world robotic manipulation tasks. Experimental results show that CogVLA achieves state-of-the-art task success rates while reducing end-to-end computational costs significantly, as shown in Fig. 1 (g) and (h). Ablation studies further validate the complementarity and synergistic effect of the routing modules and the coupled attention mechanism. Our main contributions are summarized as follows:

• •

We propose CogVLA, a Cognition-Aligned Vision-Language-Action framework inspired by human multimodal coordination, which establishes a biomimetic 3-stage architecture: “VAS (visual information focusing) – SMA (semantic intent filtering) – PMC (action sequence planning)."

• •

We develop synergistic EFA-Routing and LFP-Routing, enabling instruction‑driven vision sparsification in perception-reasoning pipelines.

• •

We formulate CAtten ensuring cross-modal logical consistency and temporal action coherence in doubly compressed multimodal representations.

• •

Through extensive experiments on the LIBERO benchmark and real-world robotic tasks, we demonstrate the superior performance and efficiency of CogVLA.

2 Methods

2.1 Preliminary: Parallel Decoding in Action Chunk

We consider a sequence prediction setting where a Vision-Language-Action (VLA) model outputs a sequence of actions across $K$ future timesteps. Traditional autoregressive (AR) decoding predicts actions sequentially, whereas parallel decoding enables simultaneous prediction of all actions within the chunk, improving inference efficiency and supporting scalable deployment.

Action Chunk. Given the current input context $\mathbf{X}=\{\mathbf{I},\mathbf{t}\}\in\mathbb{R}^{M+T}$, which comprises the visual observation and task instruction, the model predicts a chunk of $K$ future actions:

[TABLE]

Here, $D$ denotes the dimensionality of each atomic action (e.g., $D=7$ for 3-DoF translation $\Delta{T}$, 3-DoF rotation $\Delta{R}$, and binary gripper control).

Autoregressive Decoding. In causal autoregressive decoding, the action sequence is generated incrementally. For each timestep $i\in{0,\dots,K-1}$, the atomic action vector $\mathbf{a}_{i}\in\mathbb{R}^{D}$ is produced token-by-token, with each token $\mathbf{a}_{i}^{(k)}$ conditioned on the preceding tokens and previously actions:

[TABLE]

This decoding necessitates $K\times D$ forward passes, introducing latency from token-level dependencies.

Parallel Decoding. In contrast, parallel decoding eliminates the sequential dependency. The model receives the input observation embeddings $\mathbf{X}$ along with $K$ empty placeholder embeddings:

[TABLE]

where $\mathbf{0}_{i}\in\mathbb{R}^{D}$ denotes a learnable zero-action embedding. Under a bidirectional attention scheme (instead of causal masking), the decoder jointly produces all future actions in a single pass:

[TABLE]

2.2 CogVLA: Framework

Recent methods Yue et al. (2024); Zhang et al. (2025b); Pertsch et al. (2025); Song et al. (2025); Kim et al. (2025) primarily focus on lightweight computation within isolated stages of action chunk prediction in VLA models, often leading to cross-modal semantic degradation due to modular disconnection. To address this, we propose CogVLA, a cognition-aligned framework that enhances both efficiency and performance via Instruction-Driven Routing & Sparsification.

As illustrated in Fig. 2, the framework operates through a 3-stage progressive architecture inspired by human multimodal coordination. In Stage 1, CogVLA incorporates $N$ vision encoders $\{\text{Enc}_{1},\dots,\text{Enc}_{N}\}$ that extract visual tokens from image observations $\mathbf{I}^{(i)}$. Each encoder is modulated by the instruction $\mathbf{t}_{r}$ (obtained via LLM embedding layer) through an Encoder-FiLM module:

[TABLE]

where $\mathbf{v}_{\text{agg}}^{(i)}$ denotes the aggregation token for the $i$-th encoder. These modality-specific aggregated tokens are dynamically dual-aggregated via an instruction-conditioned routing mechanism:

[TABLE]

[TABLE]

In Stage 2, the dual-aggregation tokens are injected into the LLM, where LFP-Routing selectively filters out instruction-irrelevant visual tokens. This instruction-driven sparsification enables token-level efficiency by reducing redundant attention computation, aligning the retained tokens with task-relevant semantics. The filtered representation is then processed by the proposed CAtten module to produce the task-aligned representation in Stage 3. The module operates as follows:

[TABLE]

where $\mathbf{Z}_{l}$, and $\mathbf{t}_{l}$ denote the visual and instruction input tokens at the $(l+1)$-th transformer layer, respectively, with $\mathbf{Z}_{0}=\mathbf{v}_{\text{agg}}$ as the initial state. Finally, action chunks are decoded in parallel using the compressed multimodal context combined with placeholder action tokens:

[TABLE]

Through this progressive design across Stage 1–3, CogVLA realizes instruction-driven sparsification and routing across the vision-language-action pipeline, effectively reducing computational overhead while preserving task-relevant semantics and enhancing cross-modal reasoning fidelity.

2.3 CogVLA: Cognition‑Aligned 3-Stage Progressive Design

As illustrated in Fig. 3, CogVLA adopts a 3-Stage Progressive Design, which emulates the human’s optimized coordination during manipulation tasks: EFA-Routing mimics the VAS by selectively aggregating visual tokens conditioned on task-specific instructions, thereby achieving focused perception. LFP-Routing emulates the SMA by introducing action intentions into the visual context within the language model, selectively pruning irrelevant tokens to achieve instruction-driven token sparsity. CAtten simulates the PMC by dynamically integrating compressed multimodal representations, ensuring cross-modal logical consistency and temporal coherence in action decoding.

2.3.1 Encoder-FiLM based Aggregation Routing

Step 1: Intra-encoder Aggregation. To aggregate visual information and enable instruction-guided representation learning, we introduce Encoder-FiLM, which dynamically consolidates observation tokens into aggregation tokens based on task-specific instructions. The language instruction $\mathbf{t}_{r}$ modulates visual tokens $\mathbf{I}^{(i)}$ and aggregation tokens $\mathbf{v}_{\text{agg}}^{(i)}$ within each visual encoder branch:

[TABLE]

where $\gamma_{i}$ and $\beta_{i}$ denote the FiLM-generated scale and shift vectors conditioned on $\mathbf{t}_{r}$, and $\odot$ represents element-wise multiplication. Through iterative visual encoder blocks, the aggregation token $\mathbf{v}_{\text{agg}}^{(i)}$ adaptively integrates instruction-relevant information from observation tokens while discarding redundant information. Consequently, only the final $\mathbf{v}_{\text{agg}}^{(i)}$ is retained while the image tokens $\mathbf{I}^{(i)}$ are discarded, effectively reducing the number of visual tokens to 25% of the original size.

Step 2: Cross-encoder Aggregation. To integrate the aggregated visual representations from two heterogeneous vision encoder branches (SigLIP and DINOv2), we design an instruction-conditioned aggregation routing gate that computes a fusion weight $\alpha\in(0,1)$ based on the input language instruction. Rather than statically assigning equal contributions, the fusion ratio is dynamically predicted for different tasks to reflect instruction-dependent visual preferences:

[TABLE]

where $W_{1}$, $W_{2}$ are trainable weight matrices, $\mathbf{b}_{1}$, $\mathbf{b}_{2}$ are biases, and $\sigma$ denotes the GeLU Hendrycks and Gimpel (2016) non-linearity. The final dual-aggregated visual token is computed as:

[TABLE]

This instruction-conditioned aggregation routing allows the model to adaptively balance visual features from different encoders based on the semantics of the instruction, promoting more effective cross-modal fusion, as shown in Fig. 2 and Fig. 3 (a). At the first transformer layer, we denote $\mathbf{Z}_{0}=\mathbf{v}_{\text{agg}}$ as the dual-aggregation visual tokens and $\mathbf{t}_{0}$ as the instruction tokens, which serve as the initial inputs for Stage 2.

2.3.2 LLM-FiLM based Pruning Routing

Motivated by sparse token routing techniques Zhang et al. (2024a); Luo et al. (2024); Wu et al. (2024), we recognize that EFA-Routing (Stage 1) aggregates features across all image tokens, potentially retaining redundant or semantically irrelevant visual information. To further reduce computational overhead and steer the visual representation toward the intended action semantics, we propose a lightweight LFP-Routing module prior to injecting visual context into the large language model, as illustrated in Fig. 3(b).

Given the dual-aggregation tokens and the corresponding task instruction at transformer layer $l$, denoted as $\mathbf{Z}_{l}$ and $\mathbf{t}_{l}$ respectively, LLM-FiLM performs a semantic-aware modulation as follows:

[TABLE]

where $\gamma_{\text{LLM}}(\cdot)$ and $\beta_{\text{LLM}}(\cdot)$ denote instruction-conditioned scaling and shifting functions, respectively, both implemented as lightweight MLPs. The $\text{Prune}(\cdot)$ operation selectively discards tokens with low task relevance, producing a filtered representation that maintains critical visual semantics.

We introduce a Task-Guided Pruning Router to implement the $\text{Prune}(\cdot)$ operation. This module filters tokens based on their instruction-aware relevance, preserving only those most critical to the task. At Transformer layer $l$, routing weights $R_{l}^{j}$ are computed for each visual token $\mathbf{Z}_{l}^{j}$ using an MLP:

[TABLE]

We define a token retention ratio $\beta$, and determine a relevance threshold $P_{\beta}^{l}$ as the $\beta$-th percentile of the routing weights at layer $l$. Only tokens whose scores exceed $P_{\beta}^{l}$ are preserved. Formally:

[TABLE]

where $f_{\text{SF}}(\cdot)$ represents the self-attention and feed-forward operations within the current layer. The hyperparameter $\beta\in[0,1]$ governs the sparsity level by controlling the proportion of retained tokens.

2.3.3 V-L-A Coupled Attention

To maintain semantic consistency and temporal coherence under compressed multimodal inputs, we introduce V-L-A Coupled Attention (CAtten), a biologically inspired mechanism grounded in the functional role of the PMC for planning and coordination. As shown in Fig. 3 (c), CAtten hierarchically combines causal and bidirectional attention across vision, language, and action streams. At the $l$-th transformer layer of the LLM, the input multimodal token sequence is defined as:

[TABLE]

where $\mathbf{A}_{l}=[\mathbf{a}_{0}^{l},\dots,\mathbf{a}_{K-1}^{l}]$ denotes the action chunk. CAtten operates in three consecutive stages:

Causal Vision-Language Attention. To preserve instruction-conditioned visual reasoning, causal attention is applied over the concatenated vision-language token segment:

[TABLE]

where $\mathbf{M}_{\text{causal}}^{\text{VL}}\in\mathbb{R}^{(M+T)\times(M+T)}$ is a lower-triangular mask within vision-language tokens.

Bidirectional Action Chunk Decoding. To support coherent action generation, bidirectional attention is employed within the action decoding, allowing full context integration among future action tokens:

[TABLE]

where $\mathbf{M}_{\text{bi}}^{\text{act}}\in\mathbb{R}^{(K\times D)\times(K\times D)}$ enables full continuous parallel decoding within each action chunk while maintaining causal dependencies from visual-language inputs.

Unified Hybrid Attention Mask. A global attention mask $\mathbf{M}_{\text{hybrid}}\in\mathbb{R}^{(M+T+K\times D)\times(M+T+K\times D)}$ enforces hierarchical token dependencies across vision, language, and action modalities:

[TABLE]

[TABLE]

This coupled attention structure enables CogVLA to retain fine-grained cross-modal alignment and planning consistency under significant sparsification of visual inputs, ensuring that action sequences remain both semantically relevant and temporally coherent throughout the decoding process.

3 Experiments

3.1 Experiments Setting

All experiments are conducted on 4$\times$ A800 GPUs (80GB), benefiting from CogVLA’s efficient instruction-driven sparsification. Implementation details are in Appendix A.

Simulation Benchmark. We use the LIBERO benchmark Liu et al. (2023a) to evaluate task performance and efficiency. Its long and diverse instructions (avg. 10.48 words vs. 3.34 in RLBench) reflect the model’s language understanding. LIBERO covers four suites—Spatial, Object, Goal, and Long—each with 10 tasks and 50 demonstrations.

Real-World Experiments. CogVLA is deployed on the Cobot Agilex ALOHA platform for three long-horizon tasks: Object Placement, Drawer Manipulation, and T-shirt Folding (45, 45, and 30 demonstrations). We introduce spatial and semantic variations during data collection.

Baselines. We compare CogVLA with multiple state-of-the-art methods, such as OpenVLA, SpatialVLA, STAR, and CoT-VLA. For efficiency assessment, we further evaluate OpenVLA, along with the top-performing PD-VLA and OpenVLA-OFT—an improved variant of OpenVLA that achieves higher performance and efficiency—under the same fine-tuning and inference settings as CogVLA.

3.2 Performance improvement

Simulation Experimental Results. The diverse task suites in the LIBERO benchmark reflect varying levels of instruction-following requirements from different perspectives. We conducted 500 trials for each task suite, and CogVLA achieved the highest success rate of 97.4%, as shown in Tab. 1. This strong performance stems from CogVLA’s 3-stage progressive design, which enhances instruction-driven perception throughout the manipulation process. Notably, CogVLA ranks second only in the LIBERO-Goal suite, primarily due to a deliberate trade-off between performance and efficiency—CogVLA reduces visual input by 8× compared to other VLA models in the table.

Real-world Experimental Results. We conducted real-world training and evaluation of the top four models from the LIBERO simulation benchmark on complex long-horizon tasks with rich instructions (Object Placement and Drawer Manipulation) and the representative dual-arm task T-shirt Folding. As shown in Tab. 2, CogVLA achieved the highest subtask and overall success rates. To better assess instruction-following ability in real-world settings, we collected ALOHA-based experimental data and applied data augmentation following LIBERO’s protocol, including variations in spatial arrangements, manipulated objects, and their attributes. The results demonstrate that CogVLA’s performance advantage generalizes effectively to real-world tasks.

3.3 Efficiency Optimization

As shown in Tab. 3, CogVLA achieves 2.79$\times$ faster inference time, 22.54$\times$ higher throughput, 3.12$\times$ lower FLOPs, and 2.49$\times$ reduction in training cost compared to OpenVLA. Moreover, CogVLA also outperforms state-of-the-art efficient VLA models such as OpenVLA-OFT and PD-VLA in both training and inference efficiency. These gains stem from: 1) instruction-driven vision sparsification achieved via EFA-Routing and LFP-Routing, reducing visual input by up to 8×; and 2) parallel action decoding enabled by bidirectional attention in the CAtten module.

3.4 Qualitative Analysis

As shown in Fig. 4, we compare the performance and efficiency of CogVLA and OpenVLA-OFT in both the LIBERO simulation and the ALOHA real-world setting. With strong instruction guidance and improved logical consistency, CogVLA executes manipulation tasks more accurately and avoids failures such as drawer collisions (e.g., row 3, column 5). CogVLA also demonstrates shorter action inference time, with its efficiency advantage becoming more pronounced as the task length increases.

3.5 Ablation Studies

Tab. 6 validates the effectiveness of each module within CogVLA’s 3-stage progressive design, highlighting their synergistic contributions under a unified framework. Tab. 6 presents different sparsity ratio allocations across Stage 1 and Stage 2 under a fixed 8× vision sparsification. Results show that both stages contribute to performance gains, with larger Stage 1 ratios yielding greater improvements, as Stage 2 further filters instruction-relevant tokens based on Stage 1 outputs. Tab. 6 demonstrates that the combined Stage 1+2 sparsification outperforms existing methods, as it is more instruction-driven and deeply integrated into the overall CogVLA architecture. Additional ablation studies are provided in Appendix C.3.

4 Related Work

Vision-Language Action (VLA) Models. Vision-Language Models (VLMs) Li et al. (2023); Alayrac et al. (2022); Liu et al. (2024a); Grattafiori et al. (2024); Shen et al. (2024); Bai et al. ; Liu et al. (2023b); Zhang et al. (2025a); Li et al. (2025a); Qiao et al. (2025) have propelled robotic control by providing rich multimodal representations, fostering the development of VLA models Li et al. (2025b); Shi et al. (2025); Liu et al. (2025); Wang et al. (2025); Wolf et al. (2025); Shao et al. (2025); Wen et al. (2023) that bridge perception and action generation. Early works like CLIPort Shridhar et al. (2022) and PerAct Shridhar et al. (2023) aligned visual features with language-conditioned action policies. The RT series Brohan et al. (2022, 2023); Belkhale et al. (2024) introduced action tokenization to enable scalable web-to-robot transfer. More recently, Octo Team et al. (2024) constructed a diverse multi-robot dataset to support multitask training, while OpenVLA Kim et al. (2024a) demonstrated superior generalization to household tasks compared to diffusion-based methods. The $\pi$ series Black et al. ; Intelligence et al. (2025) proposed heterogeneous co-training across robots and semantic prediction tasks to enhance open-world generalization. However, directly fine-tuning billion-parameter VLMs for action prediction remains computationally intensive, limiting scalability.

Efficient Design in VLA Models. Improving VLA efficiency has largely followed two paths: LLM-centric and vision-centric. LLM-centric approaches include Mixture-of-Depth (MoD) pruning Raposo et al. (2024); Zhang et al. (2024a); Luo et al. (2024), dynamic reasoning depth adjustment Yue et al. (2024); Zhang et al. (2025b), sparse Mixture-of-Experts (MoE) architectures Dai et al. (2024); Lin et al. (2024); Chen et al. (2024a), and lightweight models like DeeR-VLA Yue et al. (2024), RoboMamb Liu et al. (2024b), and TinyVLA Wen et al. (2025), all aiming to reduce decoding overhead. Vision-centric approaches focus on reducing the number of visual tokens passed to the LLM, employing techniques such as patch token selection based on similarity Shang et al. (2024); Li et al. (2024b), cropping-based techniques Liu et al. (2024c); Huang et al. (2024), and an additional compression module Bai et al. (2023); Li et al. (2024c); Ye et al. (2024). However, naively adapting these methods often leads to semantic inconsistency across modalities due to a lack of unified sparsification. To address this, we propose a cognition-aligned, instruction-driven sparsification framework that jointly improves efficiency and cross-modal consistency.

5 Conclusion

We presented CogVLA, a cognition-aligned and instruction-driven Vision-Language-Action framework designed to address the computational inefficiencies and semantic fragmentation in existing VLA models. By integrating EFA-Routing, LFP-Routing, and CAtten into a unified 3-stage progressive design, CogVLA achieves effective vision sparsification and coherent cross-modal reasoning. Extensive evaluations on both the LIBERO benchmark and real-world robotic tasks demonstrate that CogVLA not only achieves state-of-the-art performance but also significantly reduces computational cost and inference latency. This work highlights the importance of instruction-driven multimodal sparsification in building scalable and efficient embodied AI systems.

**CogVLA: Cognition-Aligned Vision-Language-Action Model via Instruction-Driven Routing & Sparsification

Appendix**

This appendix provides comprehensive supplementary material to support the methodology, analysis, and findings presented in the main paper.

• •

Section A describes implementation details, including model and training details.

• •

Section B outlines experimental details for both simulation and real-world settings.

• •

Section C presents extended quantitative analyses, including multi-seed evaluations, additional ablation studies, and expanded real-world results.

• •

Section D provides supplementary qualitative analyses, such as diverse task executions and instruction-to-observation attention visualizations.

• •

Section E discusses additional insights into the motivation behind CogVLA, highlights its current limitations, and reflects on the broader societal implications and potential risks.

• •

We provide third-person view videos at https://jiutian-vl.github.io/CogVLA-page , demonstrating CogVLA performing manipulation tasks in a fully autonomous mode, played at 1× speed. Due to the requirement of remote communication during each action chunk prediction, slight delays are introduced by network latency. For future deployments, we plan to run CogVLA locally on hardware with more than 20 GB of GPU memory (e.g., RTX 4090 with 24 GB) to eliminate such latency.

Appendix A Implementation Details

A.1 Model Details

EFA-Routing. In Step 1, each of the two vision encoders uses 64 aggregation tokens, thereby reducing the number of visual tokens to 25% of the original. In addition, the scale and shift vectors for FiLM, $\gamma_{i}$ and $\beta_{i}$, are derived from a linear transformation of the text embedding. In Step 2, a two-layer MLP is applied to the text embedding to produce routing weights for the two vision encoders.

LFP-Routing. In this module, we employ a shifted cosine schedule Zhang et al. (2024a) to control the proportion of visual tokens retained at each layer. The formulation is as follows:

[TABLE]

where $L$ denotes the total number of layers in the LLM, which is $L=32$ for CogVLA. The constant $\eta$ is a shift factor that vertically adjusts the cosine decay curve, providing a flexible mechanism to control the overall computational cost of the model. In our implementation, $\eta$ is set to 0.5. Specifically, we apply a clamp operation to constrain $\beta_{l}$ within the range $[0.05,0.85]$. As a result, LFP-Routing achieves approximately a 50% token pruning rate.

In addition, the instruction-conditioned scaling and shifting functions $\gamma_{\text{LLM}}(\cdot)$ and $\beta_{\text{LLM}}(\cdot)$ in LFP-Routing are both implemented using two-layer MLPs, with a hidden layer dimension of 2048, resulting in a parameter count almost identical to that of a direct linear layer.

A.2 Training Details

LIBERO Training Setup. We adopt OpenVLA Kim et al. (2024b) as the backbone model and set the action chunk size to $K=8$. Fine-tuning is performed using Low-Rank Adaptation (LoRA) with a rank of 32 and an $\alpha$ value of 64. The model is trained for 60K steps with a batch size of 64 and an initial learning rate of 5e-4. Checkpoints are evaluated every 10K steps, and the best-performing checkpoint is selected for reporting.

Real-World Training Setup. For the real-world experiments, we set the chunk size to $K=25$ and fine-tune OpenVLA using LoRA with a rank of 32 and an alpha value of 64. The model is trained with a batch size of 32 for a total of 80K steps. The initial learning rate was set to 5e-4, which is reduced to 5e-5 after 50K steps. Starting from step 60K, we evaluate checkpoints every 10K steps and report the best-performing checkpoint.

Appendix B Experimental Details

B.1 Simulation Benchmark

We evaluate CogVLA on the LIBERO simulation benchmark Liu et al. (2023a), a standardized suite of language-conditioned robotic manipulation tasks. Unlike earlier benchmarks such as RLBench James et al. (2020), LIBERO features more complex and diverse instructions, averaging 10.48 words per command compared to only 3.34 in RLBench. This makes it a more suitable testbed for assessing the model’s capacity in language grounding and multimodal reasoning. LIBERO comprises four task suites—Spatial, Object, Goal, and Long—each containing 10 tasks with 50 human-teleoperated demonstrations. These suites are designed to probe distinct reasoning capabilities:

• •

LIBERO-Spatial evaluates spatial reasoning capabilities by presenting identical objects arranged in different spatial configurations. The agent must interpret spatial relations (e.g., left/right, front/behind) described in the instruction to complete the task correctly.

• •

LIBERO-Object measures the model’s ability to generalize across object categories. While spatial layouts remain fixed, the manipulated objects vary in type, shape, or color, requiring the agent to ground object-referential language and adapt its actions accordingly.

• •

LIBERO-Goal tests task-oriented comprehension by altering the goal specification while keeping object types and spatial layouts constant. The agent must disambiguate subtle differences in instruction semantics to execute distinct manipulation outcomes.

• •

LIBERO-Long challenges the agent with multi-step, long-horizon tasks involving diverse objects and environments. Success requires not only grounded perception and instruction following, but also sequential planning.

CogVLA is trained and evaluated under the same setting as OpenVLA Kim et al. (2024a) to ensure comparability. We report results on all four suites to validate the model’s generalization, efficiency, and semantic grounding capabilities.

B.2 Real-World Setup

We deploy CogVLA on Cobot Agilex ALOHA Fu et al. (2024) manipulation platform, to validate its real-world applicability. The real-world evaluation consists of five diverse tasks involving both single-arm and coordinated dual-arm manipulation. To assess robustness and generalization, we introduce moderate data augmentation by varying object attributes (e.g., size, color) and rearranging spatial layouts.

We collect real-world training data for the Cobot Agilex ALOHA robot via human teleoperation. For Tasks 1–5, we gather 45, 45, 30, 30, and 45 expert demonstrations, respectively. We report the results of Tasks 1–3 in the main paper, and provide additional results for Tasks 4–5 in this appendix.The instructions and descriptions for Tasks 1–5 are provided below:

• •

Task 1: “Put the cube into the plate, and then put the toy into the bowl.”

A two-step pick-and-place task involving object category understanding and temporal sequencing. This is a dual-arm task consisting of two sequential subtasks: 1) “Put the cube into the plate” with the left arm, and 1) “Put the toy into the bowl” with the right arm. Task success is achieved only when both subtasks are completed successfully. We report success rates for each subtask and the overall task.

• •

Task 2: *“Open the drawer, place the toy into the drawer, and then close it.”

*A composite task requiring interaction with articulated objects and multi-stage execution. This is a dual-arm task consisting of three sequential subtasks: 1) “Open the drawer” with the left arm, 2) “Place the toy into the drawer” with the right arm, and 3) “Close the drawer” with the left arm. Task success requires all three subtasks to be completed. We report success rates for each subtask and the overall task.

• •

Task 3: *“Fold the T-shirt.”

*A soft-body manipulation task that evaluates the system’s ability to handle deformable objects. This is a dual-arm task consisting of three sequential folding steps. Task success is determined by the successful execution of all three steps. We report intermediate success rates for each step and the overall task performance.

• •

Task 4: *“Pick the red cube into the plate, and then pick the big cube into the bowl.”

*A multi-attribute grounding task requiring comprehension of both color and size references. This is a dual-arm task consisting of two sequential subtasks: 1) “Pick the red cube into the plate” with the left arm, and 2) “Pick the big cube into the bowl” with the right arm. Task success is achieved only when both subtasks are completed. We report success rates for each subtask and the overall task.

• •

Task 5: *“Pick the left cube into the plate.”

*A spatial reasoning task focusing on relative positioning and egocentric understanding. This is a single-arm task consisting of one pick-and-place action. We report the final task success rate.

Appendix C Supplementary Quantitative Analysis

C.1 Multi-Seed Evaluation

To evaluate the statistical robustness and consistency of CogVLA’s performance, we conduct multi-seed evaluations on the LIBERO benchmark. For each of the four task suites (Spatial, Object, Goal, and Long), we run experiments using three independent random seeds and report the mean success rate along with the standard deviation.

As shown in Tab. 7, CogVLA exhibits consistently high performance across different seeds, with standard deviations ranging from 0.2% to 0.6%. This indicates stable learning behavior and further validates the strong generalization capability of CogVLA’s three-stage instruction-driven architecture across diverse task types.

C.2 Extended Real-World Task Results

In addition to the results reported in the main paper, we present the performance of CogVLA on Tasks 4 and 5, as shown in Tab. 8.

• •

Task 4 (“Pick the red cube into the plate, and then pick the big cube into the bowl”) evaluates the model’s ability to ground multi-attribute language and execute sequential actions. CogVLA achieves the highest success rates across both subtasks and the overall task, demonstrating strong compositional understanding of attribute references such as color and size.

• •

Task 5 (“Pick the left cube into the plate”) focuses on egocentric spatial reasoning, requiring precise interpretation of relative spatial references from the agent’s visual perspective. CogVLA maintains a high success rate in this setting, indicating robust grounding of spatial concepts.

These results further validate CogVLA’s ability to generalize to real-world tasks that demand fine-grained language grounding and spatial understanding.

C.3 Extended Ablation Studies

We extend the sparsification analysis by evaluating additional Stage 1/Stage 2 configurations: 2×2 and 4×4, while keeping the total sparsification ratio fixed at 4× and 16×, respectively. These configurations are compared alongside the baseline 8× setting with different asymmetric allocations (e.g., 2×–4× and 4×–2×), allowing us to systematically assess how the distribution of sparsity across stages impacts downstream performance.

As shown in Tab.3, the 2×2 setting provides a favorable trade-off between performance and computational efficiency. In contrast, the 4×4 setting leads to a slight degradation in performance, suggesting that excessive sparsification across both stages may hinder the preservation of task-relevant information.

Interestingly, the asymmetric configurations, particularly the 4×–2× setup, outperform their symmetric counterparts, achieving the highest spatial success rate of 98.6. This highlights the advantage of applying a more aggressive token reduction in Stage 1 (EFA-Routing), where redundant visual tokens can be effectively compressed via instruction-guided aggregation. Subsequently, Stage 2 (LFP-Routing) performs finer-grained token pruning in a context-aware manner within the language model, allowing for better preservation of task-relevant information.

These findings support the core design principle of CogVLA: progressive sparsification with an asymmetric allocation tailored to the representational characteristics of each stage. By balancing early-stage compression and late-stage selectivity, the model achieves both computational efficiency and high task accuracy, reinforcing the importance of stage-aware sparsity scheduling in multimodal architectures.

Appendix D Supplementary Qualitative Analysis

D.1 Additional Visualizations of Simulation and Real-World Results

We present additional qualitative results from both simulation and real-world experiments to illustrate CogVLA’s generalization and execution capabilities. As shown in Fig. 8, the model consistently completes multi-step tasks across diverse environments, object configurations, and instruction variants.

In real-world tasks with varying instructions, CogVLA accurately interprets long-horizon commands and produces coherent action sequences. These examples further highlight the model’s ability to maintain cross-modal consistency and temporal reasoning, as well as its robustness in simulation-to-reality transfer. Fig. 5 illustrates the real-world manipulation workflows for Tasks 1-5. For Task 1, we provide multi-view observations from the Front Camera, Left Wrist Camera, and Right Wrist Camera. For Tasks 2-5, only Front Camera observations are shown for clarity. In Fig. 6, we present a third-person view demonstration of CogVLA performing a manipulation task in the lab. The corresponding MP4 video file is provided in the supplementary materials.

D.2 Instruction-to-Observation Attention Maps

To gain deeper insights into how CogVLA aligns language instructions with visual observations, we visualize the attention maps generated by the cross-modal attention modules. As shown in Fig. 7, the attention weights highlight task-relevant regions in the input image.

These visualizations demonstrate that CogVLA’s instruction-aware routing mechanisms effectively guide the perception module to attend to semantically meaningful areas, enabling robust visual grounding even in cluttered or ambiguous scenes.

Appendix E Discussion

E.1 Supplementary Details on the Motivation

CogVLA is motivated by the need to improve both computational efficiency and cross-modal semantic alignment in instruction-conditioned robotic systems. Its architectural design is informed by cognitive science research on how humans process language, perceive their environment, and execute actions in a coherent and goal-directed manner.

Cognitive studies suggest that humans rely on structured inductive biases—often termed "intuitive theories"—to interpret the world, including intuitive physics, causality, and theory of mind Lake et al. (2017a); Tenenbaum et al. (2011). While recent multimodal large language models exhibit partial competence in these areas, they often lack robustness in compositional reasoning and causally grounded behavior Schulze Buschoff et al. (2025a).

To address these limitations, CogVLA adopts a biologically inspired architecture that reflects the division of functional roles observed in the human brain. Specifically, we draw connections between the model’s three routing modules and key components in human multimodal cognition: the Visual Attention System (VAS), the Supplementary Motor Area (SMA), and the Premotor Cortex (PMC).

Visual Attention System (VAS) $\rightarrow$ Encoder-FiLM. The human visual attention system selectively enhances perception of task-relevant features while suppressing distractors Corbetta and Shulman (2002). Top-down signals from frontal and parietal cortices bias visual processing toward objects or regions mentioned in language or necessary for action. This selective modulation improves efficiency and semantic grounding in complex scenes. In CogVLA, the Encoder-FiLM module mimics VAS by dynamically modulating visual encoder features conditioned on instructions, focusing perception on semantically relevant regions and reducing redundancy Perez et al. (2018). This allows the model’s perception to be grounded in context, much as the brain’s attention system tunes visual processing to relevant aspects of a scene during coordinated vision-language tasks.

Supplementary Motor Area (SMA) $\rightarrow$ LLM-FiLM. The SMA plays a key role in planning and sequencing actions, even in the absence of physical movement Schwartze et al. (2012); Tanaka and Kirino (2017). It integrates multimodal information and high-level goals to shape future motor behavior, before engaging primary motor circuits. In CogVLA, the LLM-FiLM module can be seen as the “intention planner” of the model and serves a similar function: it injects task-specific intent into the language model, pruning irrelevant visual-linguistic tokens and steering the model toward generating appropriate action plans. This enables more efficient and intention-aligned reasoning, analogous to how the SMA organizes abstract motor programs before execution.

Premotor Cortex (PMC) $\rightarrow$ V-L-A Coupled Attention. The premotor cortex is involved in translating perceptual cues into executable motor plans Gallese et al. (1996); Fogassi et al. (2005). It contains visuomotor neurons that represent both the perception of object affordances and the intended grasping actions, enabling visuomotor grounding. CogVLA’s V-L-A Coupled Attention module reflects this mechanism by integrating visual, linguistic, and action representations through a unified attention mechanism. This ensures that generated actions are causally and temporally coherent with respect to both the observed scene and the given instruction.

By aligning its modular design with biologically plausible cognitive functions, CogVLA offers not only performance and efficiency gains, but also a cognitively grounded pathway for improving generalization and interpretability in embodied multimodal agents.

E.2 Limitation and Future Work

While CogVLA demonstrates strong performance across simulation and real-world tasks, several limitations remain. First, the current instruction-to-vision routing relies on predefined sparsity ratios and fixed token pruning schedules, which may not adapt optimally to varying instruction complexity or scene difficulty. Second, although the model generalizes well within the LIBERO and ALOHA settings, its performance under out-of-distribution instructions or unseen manipulation categories is yet to be thoroughly evaluated.

In future work, we aim to explore adaptive sparsification mechanisms conditioned on task semantics and environmental uncertainty. Moreover, integrating lifelong learning or online adaptation strategies may further enhance CogVLA’s robustness in open-world deployment scenarios. Lastly, extending the framework to support multimodal feedback (e.g., haptic or force sensing) could improve its applicability to fine-grained manipulation tasks.

E.3 Broader Impact and Potential Risk

CogVLA advances the efficiency and interpretability of instruction-driven robotic manipulation, offering potential benefits in applications such as assistive robotics, household automation, and industrial assembly. Its biologically inspired sparsification and routing mechanisms reduce computation cost, making it more accessible for resource-constrained platforms. However, as with any vision-language-action system, risks include misinterpretation of ambiguous instructions, failure in unpredictable environments, and bias amplification from training data. If deployed in safety-critical settings without appropriate safeguards, such failures could lead to unintended behaviors or physical harm. We encourage the community to adopt robust evaluation protocols, prioritize transparency in model behavior, and consider human-in-the-loop designs to mitigate such risks. Broader societal considerations—including data diversity, accessibility, and responsible deployment—should guide future development of systems built upon CogVLA.