EMCompress: Video-LLMs with Endomorphic Multimodal Compression

TL;DR

This paper introduces EMCompress, a novel multimodal compression method inspired by cognitive principles, to improve VideoQA by preserving essential information while reducing input complexity.

Contribution

It formulates Endomorphic Multimodal Compression (EMC) as a structurally constrained transformation that maintains answer invariance and integrates seamlessly into existing VideoQA pipelines.

Findings

EMC surpasses prior methods by 0.40 F-1 on a new benchmark.

Integrating EMC improves training efficiency by 7.33%.

Inference efficiency increases by 33.7% with EMC.

Abstract

Video-LLMs face a fundamental tension in long-video reasoning: static, sparse frame sampling either dilutes evidence across task-irrelevant segments at significant cost or misses fine-grained temporal semantics altogether. We propose a novel, cognitively-inspired task -- Endomorphic Multimodal Compression (EMC) -- as a structurally-constrained sufficient-statistic problem for VideoQA, and formulate it as an endomorphic transformation F_EMC : (V, Q) -> (v, q) that compresses the multimodal input while preserving answer invariance across reasonable downstream models. The endomorphic form keeps the compressed output in the downstream pipeline's native task space -- a structural mirror of the filter-then-reason mechanism in the cognitive literature motivating EMC -- distinguishing it from latent-code compression (IB / VIB) and making the formulation extensible to other multimodal settings. Under the Markov chain A -> (V, Q) -> (v, q), EMC realizes the classical sufficiency condition I((v, q); A) = I((V, Q); A) in its VideoQA-natural form. As a modular front-end, EMC plugs into both Video Instruction Tuning and Video Question Answering pipelines. We release the first dedicated benchmark and propose ReSimplifyIt, an EMC baseline surpassing prior methods by 0.40 F-1 with competitive query rewriting. Integrating EMC yields relative gains of 7.33% in training and 33.7% in inference for video-language understanding.

Tables

Table 1. (a) Results on video output

Method	Temporal Relational		Timepoint Indexed		Multifaceted Integrative		Average
	mIoU	F1	mIoU	F1	mIoU	F1	mIoU	F1
VTG-GPT (Xu et al., 2024b)	0.17	0.29	0.15	0.26	0.16	0.24	0.16	0.27
Zheng et al. (2024)	0.11	0.19	0.04	0.08	0.06	0.10	0.07	0.12
ReSimplifyIt (Ours)	0.23	0.37	0.98	0.99	0.47	0.64	0.56	0.67

Table 2. (a) Results on video output

Method	Temporal Relational		Timepoint Indexed		Multifaceted Integrative		Average
	mIoU	F1	mIoU	F1	mIoU	F1	mIoU	F1
VTG-GPT (Xu et al., 2024b)	0.17	0.29	0.15	0.26	0.16	0.24	0.16	0.27
Zheng et al. (2024)	0.11	0.19	0.04	0.08	0.06	0.10	0.07	0.12
ReSimplifyIt (Ours)	0.23	0.37	0.98	0.99	0.47	0.64	0.56	0.67

Table 3. (b) Results on query rewriting

Method	Temporal Relational	Timepoint Indexed	Multifaceted Integrative	Average
ReSimplifyIt (Ours)	66.8	78.5	72.8	72.7

Table 4. Table 2: Evaluation results of EMC-guided inference. Bold values indicate the better-performing result for each baseline, comparing the with EMC v.s. without EMC configurations on each benchmark.

Method	Size	ActivityNetQA		EMCompress		EgoSchema		LVBench		MLVU		Video-MME		NExT-OE
		w/emc	w/o	w/emc	w/o	w/emc	w/o	w/emc	w/o	w/emc	w/o	w/emc	w/o	w/emc	w/o
Video-ChatGPT (Maaz et al., 2024)	7B	58.5	50.5	38.9	28.91	29.2	23.0	23.5	22.9	22.5	18.6	31.1	29.5	49.8	41.5
Video-LLaVA (Lin et al., 2023)	7B	61.2	52.1	43.2	32.4	43.8	40.0	26.5	23.2	31.9	27.5	43.4	39.1	48.0	43.3
ChatUniVi (Jin et al., 2023)	7B	63.2	52.0	56.5	47.4	–	–	–	–	–	–	–	–	34.5	25.9
LLaVA-NExT (Liu et al., 2024b)	7B	67.8	62.5	58.2	48.6	38.6	38.2	31.4	23.8	34.2	28.3	43.8	40.3	52.3	43.7
InternVL3.5 (Wang et al., 2025)	8B	59.9	57.2	51.5	45.7	67.2	61.8	46.3	39.6	46.8	45.1	60.4	56.9	59.1	53.9
InternVL3.5 (Wang et al., 2025)	14B	60.1	58.9	52.1	48.8	70.6	67.6	47.8	42.9	46.5	45.2	62.2	61.1	58.2	55.6
Qwen2.5-VL (Bai et al., 2025b)	3B	57.6	55.8	47.9	40.3	61.8	57.2	44.2	36.3	45.3	41.3	59.0	57.7	54.8	51.6
Qwen2.5-VL (Bai et al., 2025b)	7B	58.0	55.2	48.0	41.1	68.2	66.4	43.0	34.8	43.1	37.6	60.0	58.5	55.2	53.4
Qwen3-VL (Bai et al., 2025a)	4B	59.9	57.7	48.0	45.6	71.2	70.2	43.2	37.5	49.0	42.8	60.5	60.6	61.1	58.2
Qwen3-VL (Bai et al., 2025a)	32B	60.9	60.5	47.8	45.2	72.8	72.6	–	–	44.9	38.7	64.0	61.2	61.4	58.9
LLaVA-OneVision (Li et al., 2024a)	4B	–	–	–	–	18.6	18.2	–	–	13.5	5.5	–	–	–	–
LLaVA-OneVision (Li et al., 2024a)	8B	32.4	27.5	–	–	38.4	32.5	–	–	14.1	12.2	51.1	51.3	–	–
VideoAgent (Wang et al., 2024b)	-	61.5	60.2	53.9	38.4	–	–	–	–	–	–	–	–	47.8	49.6
VideoTree (Wang et al., 2024c)	-	63.6	59.0	69.4	57.1	–	–	–	–	–	–	–	–	61.9	57.7
GPT-4o (OpenAI et al., 2024a)	-	75.65	72.2	73.84	63.59	72.38	71.17	46.22	35.04	48.92	44.73	66.91	61.63	62.25	54.9
GPT-4.1-mini (OpenAI et al., 2024b)	-	77.07	75.19	76.04	68.74	71.12	72.02	46.35	34.43	50.67	39.90	62.45	61.20	68.93	57.45
GPT-4-turbo (OpenAI et al., 2024b)	-	77.1	74.34	79.39	70.61	69.20	66.60	–	–	–	–	–	–	70.35	61.4

Table 5. Table 3: Evaluation results of EMC-guided training by downstream inference. On the rows, w/o emc , w/emc (gt) , and w/emc (pred) indicate training with vanilla EMCompress dataset, training with ground truth EMC labels, and training with the predicted EMC results, respectively; on the columns, w/o emc and w/emc indicate the inference mode.

Method		ActivityNetQA		EMCompress		NExT-QA		NExT-OE
		w/emc	w/o	w/emc	w/o	w/emc	w/o	w/emc	w/o
Video-ChatGPT	w/o emc	57.4	49.8	45.1	35.6	48.6	41.3	46.2	40.0
	w/ emc (gt)	62.7	54.4	49.8	38.9	51.0	45.7	49.2	43.5
	w/ emc (pred)	60.0	51.3	48.2	37.1	49.6	44.3	48.7	41.9
Video-LLaVA	w/o emc	53.9	45.1	46.3	35.9	50.2	47.8	47.0	39.1
	w/ emc (gt)	57.0	48.9	52.6	41.6	55.7	50.8	52.1	44.9
	w/ emc (pred)	54.2	46.9	48.1	37.4	51.0	49.3	50.7	41.8
ChatUniVi	w/o emc	63.2	50.0	62.4	57.4	–	–	29.0	22.5
	w/ emc (gt)	65.6	54.4	67.8	63.8	–	–	34.3	26.0
	w/ emc (pred)	64.9	52.6	62.8	61.6	–	–	32.1	24.0

Table 6. (a) ReSimplifyIt-simple

Dataset	DensAmp	OT/1%	CostRatio$\left(K\right)$
			$K=8$	$K=16$	$K=32$	$K=100$
ActivityNet-QA	25.0$\times$	3.8	0.15	0.09	0.07	0.05
EMCompress	29.7$\times$	3.9	0.13	0.08	0.06	0.04
NExT-OE	20.7$\times$	5.2	0.19	0.12	0.08	0.06
EgoSchema	6.3$\times$	6.0	0.59	0.38	0.27	0.19
LVBench	320.7$\times$	15.1	0.01	0.01	0.01	0.00
MLVU	70.6$\times$	10.8	0.06	0.04	0.02	0.02
Video-MME	51.2$\times$	8.8	0.08	0.05	0.03	0.02

Table 7. (a) ReSimplifyIt-simple

Dataset	DensAmp	OT/1%	CostRatio$\left(K\right)$
			$K=8$	$K=16$	$K=32$	$K=100$
ActivityNet-QA	25.0$\times$	3.8	0.15	0.09	0.07	0.05
EMCompress	29.7$\times$	3.9	0.13	0.08	0.06	0.04
NExT-OE	20.7$\times$	5.2	0.19	0.12	0.08	0.06
EgoSchema	6.3$\times$	6.0	0.59	0.38	0.27	0.19
LVBench	320.7$\times$	15.1	0.01	0.01	0.01	0.00
MLVU	70.6$\times$	10.8	0.06	0.04	0.02	0.02
Video-MME	51.2$\times$	8.8	0.08	0.05	0.03	0.02

Table 8. (b) ReSimplifyIt (full).

Dataset	DensAmp	OT/1%	CostRatio$\left(K\right)$
			$K=8$	$K=16$	$K=32$	$K=100$
ActivityNet-QA	65.2$\times$	39.6	0.18	0.10	0.06	0.03
EMCompress	14.7$\times$	25.5	0.70	0.38	0.23	0.12
NExT-OE	12.3$\times$	34.9	0.83	0.45	0.27	0.14
EgoSchema	25.8$\times$	77.3	0.67	0.36	0.20	0.09
LVBench	852.8$\times$	38.5	0.02	0.01	0.01	0.00
MLVU	308.5$\times$	48.9	0.06	0.03	0.02	0.01
Video-MME	114.5$\times$	43.2	0.12	0.07	0.04	0.02

Table 9. (a) more results on stage 1 video output

Method	Temporal Relational				Timepoint Indexed				Multifaceted Integrative				Average
	mIoU	Pre.	Cov.	F1	mIoU	Pre.	Cov.	F1	mIoU	Pre.	Cov.	F1	mIoU	Pre.	Cov.	F1
VTG-GPT (Xu et al., 2024b)	0.17	0.26	0.32	0.29	0.15	0.22	0.30	0.26	0.16	0.23	0.31	0.24	0.16	0.27	0.31	0.27
Zheng et al. (2024)	0.11	0.18	0.21	0.19	0.04	0.08	0.09	0.08	0.06	0.10	0.11	0.10	0.07	0.12	0.13	0.12
ReSimplifyIt (Ours)	0.23	0.36	0.39	0.37	0.98	1.0	0.98	0.99	0.47	0.60	0.69	0.64	0.56	0.65	0.69	0.67

Table 10. (a) more results on stage 1 video output

Method	Temporal Relational				Timepoint Indexed				Multifaceted Integrative				Average
	mIoU	Pre.	Cov.	F1	mIoU	Pre.	Cov.	F1	mIoU	Pre.	Cov.	F1	mIoU	Pre.	Cov.	F1
VTG-GPT (Xu et al., 2024b)	0.17	0.26	0.32	0.29	0.15	0.22	0.30	0.26	0.16	0.23	0.31	0.24	0.16	0.27	0.31	0.27
Zheng et al. (2024)	0.11	0.18	0.21	0.19	0.04	0.08	0.09	0.08	0.06	0.10	0.11	0.10	0.07	0.12	0.13	0.12
ReSimplifyIt (Ours)	0.23	0.36	0.39	0.37	0.98	1.0	0.98	0.99	0.47	0.60	0.69	0.64	0.56	0.65	0.69	0.67

Table 11. Table 6: Evaluation results of EMC-guided inference on four benchmark: ActivityNetQA, EMCompress, NExT-QA, and NExT-OE, with the sub-categories presented in each benchmark.

Method	ActivityNetQA
	w/emc	w/o
Video-ChatGPT (Maaz et al., 2024)	58.5	50.5
Video-LLaVA (Lin et al., 2023)	61.2	52.1
ChatUniVi (Jin et al., 2023)	63.2	52.0
LLaVA-NExT (Liu et al., 2024b)	67.8	62.5
VideoAgent (Wang et al., 2024b)	61.5	60.2
VideoTree (Wang et al., 2024c)	59.0	63.6

Table 12. (a) Results on video output.

Method	Temporal Relational		Timepoint Indexed		Multifaceted Integrative		Average
	mIoU	F1	mIoU	F1	mIoU	F1	mIoU	F1
ReSimplifyIt (Ours)	0.23	0.37	0.98	0.99	0.47	0.64	0.56	0.67
ReSimplifyIt-simple (Ours)	0.24	0.37	0.98	0.99	0.42	0.57	0.55	0.64
ReSimplifyIt-blind (Ours)	0.12	0.20	0.97	0.99	0.38	0.55	0.49	0.58

Table 13. (a) Results on video output.

Method	Temporal Relational		Timepoint Indexed		Multifaceted Integrative		Average
	mIoU	F1	mIoU	F1	mIoU	F1	mIoU	F1
ReSimplifyIt (Ours)	0.23	0.37	0.98	0.99	0.47	0.64	0.56	0.67
ReSimplifyIt-simple (Ours)	0.24	0.37	0.98	0.99	0.42	0.57	0.55	0.64
ReSimplifyIt-blind (Ours)	0.12	0.20	0.97	0.99	0.38	0.55	0.49	0.58

Table 14. (b) Results on query rewriting.

Method	Temporal Relational	Timepoint Indexed	Multifaceted Integrative	Average
ReSimplifyIt (Ours)	66.8	78.5	72.8	72.7
ReSimplifyIt-simple (Ours)	66.2	81.9	68.1	72.0
ReSimplifyIt-blind (Ours)	65.0	80.5	70.7	72.1

Table 15. Table 8: Per-dataset cost drivers and compression effectiveness for ReSimplifyIt-simple.

Dataset	#TotalCap	#Tool	#LLM	#OutTok	DurAll	DurScrn	Compress%
ActivityNet-QA	21.4	1.9	3.2	341	9.8%	9.0%	99.2%
EMCompress	21.9	1.9	3.3	366	6.4%	6.4%	100.0%
NExT-OE	23.4	3.9	5.0	433	20.2%	16.7%	95.7%
EgoSchema	22.1	2.1	3.5	446	26.0%	26.0%	100.0%
LVBench	27.0	7.0	9.6	1482	2.2%	1.9%	99.7%
MLVU	24.9	4.9	6.9	1122	12.2%	7.0%	94.4%
Video-MME	23.9	3.9	5.7	806	8.3%	8.0%	99.7%

Table 16. Table 9: Per-dataset cost drivers and compression effectiveness for the full ReSimplifyIt. VwTl denotes total Viewer tool calls per sample.

Dataset	#TotalCap	V$\to$Vw	VwTl	DurAll	DurScrn	Compress%
ActivityNet-QA	67.5	2.7	6.8	51.7%	16.2%	57.6%
EMCompress	73.9	2.4	5.8	6.8%	6.8%	97.9%
NExT-OE	18.3	0.4	1.3	86.9%	32.8%	19.5%
EgoSchema	50.6	2.3	4.5	75.0%	25.6%	33.6%
LVBench	83.8	2.6	11.3	57.5%	4.8%	44.6%
MLVU	64.5	1.7	5.2	81.4%	7.2%	20.1%
Video-MME	76.2	2.4	6.7	64.0%	10.4%	40.1%

Table 17. Table 10: Full per-source caption breakdown for ReSimplifyIt. #loc abbreviates #localize. Passive captions are pre-loaded into agent prompts; active captions are fetched during tool execution.

Dataset	Pasv	VaIn	LoIn	Actv	#scan	#loc	#get_cap
ActivityNet-QA	34.0	17.4	16.6	33.5	2.6	1.7	2.4
EMCompress	38.2	22.8	15.4	35.7	3.0	1.5	1.1
NExT-OE	5.3	2.7	2.6	13.0	0.4	0.3	0.4
EgoSchema	24.5	13.6	10.8	26.1	2.5	1.1	0.8
LVBench	48.9	17.5	31.3	94.9	4.8	3.1	3.0
MLVU	22.1	9.8	12.3	42.4	2.1	1.2	1.6
Video-MME	32.8	15.0	17.8	43.4	3.3	1.8	1.5

Equations

$${\mathcal{F}}_{\text{EMC}}:(V,Q)\to (v,q),$$

$$A\to (V,Q)\to (v,q),$$

$$I((v,q);A)\le I((V,Q);A),$$

$${\mathcal{F}}_{\text{EMC}}:(V,Q)⟼(v,q)$$

$$v={\bigcup }_{i=1}^n\{F_{s_i},\dots ,F_{e_i}\}\subseteq V,$$

$$\mathcal{M}(q,v)=\mathcal{M}(Q,V).$$

$$(v^{\ast },q^{\ast })=(\underset{\begin{array}{c}v:\exists q,\\ (v,q)\in \mathcal{A}\end{array}}{\mathrm{argmin}}\mathrm{Size}(v),\underset{q:(v^{\ast },q)\in \mathcal{A}}{\mathrm{argmin}}\mathrm{Infer}(q)).$$

$$response=VideoQA(v,q)$$

$$v,q={\mathcal{F}}_{\text{EMC}}(V,Q)$$

$$p\left({\mathbf{X}}_{\mathrm{A}}\mid {\mathbf{X}}_{\mathrm{V}},{\mathbf{X}}_{\mathrm{Q}}\right)=\prod _{i=1}^L{p}_{\theta }\left({\mathbf{X}}_{\mathrm{A}}^{[i]}\mid {\mathbf{X}}_{\mathrm{V}},{\mathbf{X}}_{\mathrm{Q}},{\mathbf{X}}_{\mathrm{A}}^{[1:i-1]}\right)$$

$${\mathbf{X}}_{\mathrm{A}},{\mathbf{X}}_{\mathrm{Q}},{\mathbf{X}}_{\mathrm{V}}=f_t(a),f_t(q),f_v(v)$$

$$v,q={\mathcal{F}}_{\text{EMC}}(V,Q)$$

$$\mathrm{DensAmp}$$

$$\mathrm{EquivFr}(K)$$

$$\mathrm{CostRatio}(K)$$

$$OutTok/1\%Red=\frac{\mathbb{E}[\#\mathrm{OutTok}]}{(1-\mathbb{E}[\mathrm{DurScrn}])\cdot 100}.$$

$$(t_{start}^{\prime },t_{end}^{\prime })$$

$$(t_{start},t_{end})$$

$$\text{overlap\_ratio}=\frac{|\min (e_1,e_2)-\max (s_1,s_2)|}{\max (e_1,e_2)-\min (s_1,s_2)}$$

$$s_2>s_1,e_2>e_1,\text{and}\text{overlap\_ratio}\le \theta$$

$$\text{train: }1926,\text{val: }270,\text{test: }558$$

Full text

EMCompress: Video-LLMs with Endomorphic Multimodal Compression

Zheyu Fan††thanks: Work done during internship at UIUC.1,2, Jiateng Liu1, Yuji Zhang1, Zihan Wang2,

Yi R. (May) Fung1, Manling Li2, Heng Ji1

1University of Illinois Urbana-Champaign     2Northwestern University

zheyufan@u.northwestern.edu     hengji@illinois.edu

1 Introduction

Human cognition optimizes information processing via a two-stage control mechanism: a fast, pre-attentive filtering to prune redundancy, followed by focused reasoning on a compacted, goal-aligned representation (Treisman and Gelade, 1980; Wolfe, 1994; Zacks et al., 2007; Hochstein and Ahissar, 2002). Behaviorally, this manifests as scrubbing the seek bar before detailed viewing (Wu and Xie, 2024), minimizing Extraneous Load to purify Germane Load (Sweller, 1988; Mayer and Moreno, 2003; Paas et al., 2003)—reshaping the problem before engaging the expensive reasoning stage.

This filter-before-reason economy is forced by VideoQA: a minute of video contains thousands of frames of which only a narrow band bears on any given query, yet current Video-LLMs (Chen et al., 2023; Li et al., 2023; Xu et al., 2024a; Maaz et al., 2024; Li et al., 2024b; Ma et al., 2024) typically fit the entire stream through a bounded visual-token budget via query-agnostic uniform sampling. This dilutes semantic signal at critical segments and injects noise, weakening supervision during training (Lin et al., 2024; Zhang et al., 2025) and reducing task-oriented focus at inference.

Existing work such as traditional grounding and grounded QA (Xiao et al., 2024; Chen et al., 2024a) attempts the obvious remedy of retrieving query-relevant frames, but operates at the perceptual level—depictive similarity between query tokens and visual evidence. Complex queries entangle multi-hop temporal, relational, and causal dependencies with sparse, weakly localized cues, so retrieving depicted frames without adapting the query leaves a multimodal semantic mismatch that misleads the downstream model.

This calls for a novel mechanism that jointly reshapes reasoning structure and cognitive load while treating input modalities as a whole. The two-stage structure of human cognition (Treisman and Gelade, 1980; Wolfe, 1994) suggests a clean architectural response: decouple filtering from reasoning so one mechanism serves arbitrary downstream reasoners. We design a front-end adapter that reshapes the raw instance symmetrically over all modalities before it reaches any Video-LLM.

The cited cognitive mechanisms share a structural property: the filter stage’s output stays within the substrate the reasoning stage natively consumes—Feature Integration masks the visual field in place (Treisman and Gelade, 1980), Guided Search overlays priority on the original perceptual map (Wolfe, 1994), event segmentation carves segments out of the ongoing temporal stream (Zacks et al., 2007); none transcode to an alien representation. Taking this commitment seriously forces the computational filter into the downstream reasoner’s native task space—that is, to be endomorphic:

[TABLE]

mapping a multimodal task instance back into the same task space. This distinguishes EMC from latent-code compression (Information Bottleneck / VIB), whose learned code is not natively consumable by existing pipelines. Model-agnostic answer invariance—formalized as C2 in §2.1—is a separate behavioral requirement parallel to endomorphism, not derived from it. The endomorphic form additionally makes the formulation extensible to any multimodal task admitting a uniform agent signature; VideoQA is one instance.

Taking a generative view—where answer $A$ is a latent fact and $(V,Q)$ is an observation drawn from it—this compression forms the Markov chain $A\to(V,Q)\to(v,q)$, and Endomorphic Multimodal Compression (EMC) is precisely a structurally-constrained sufficient-statistic problem: find the most compact admissible $(v,q)$ preserving all $A$-relevant information. This identification places EMC in the Information Bottleneck lineage (Tishby et al., 1999; Tishby and Zaslavsky, 2015), instantiated over the original multimodal input space rather than a learned latent code. $A$ is conserved throughout, aligning with the fixity of the Goal State in Problem Space Theory (Newell and Simon, 1972): EMC reshapes “what to ask”, not “what to answer”.

EMC yields four concrete benefits: (i) Training alignment: removing off-target supervision boosts gradient signal-to-noise and reduces spurious correlations under fixed budgets; (ii) Inference robustness: offloading non-informative perceptual burdens shortens the implicit reasoning program, unleashing reasoning capabilities; (iii) Interpretability: the compact $(v,q)$ is a controllable, faithful artifact exposing model bottlenecks; (iv) Generalizability: as a model-agnostic front-end, EMC benefits diverse VideoQA agents.

We present ReSimplifyIt, a plug-and-play multi-agent framework that ReSiliently improvises and qualifies proposals Iteratively. Each compression round runs a language-only Launcher that hypothesizes a trimming instruction and rewritten query (using the intrinsic cross-modal relation as a prior), a Validator that executes and critiques these plans through a Viewer module, and memory trackers for self-correction—operationalizing a competence-from-consequence (Brooks, 1991) principle: trial execution provides constructive feedback tightening the coupling between hypothesized reasoning structure and available evidence.

We further construct EMCompress, the first benchmark dedicated to evaluating EMC as a joint transformation over video and query. Benchmarking shows EMC is a non-trivial task far from solved, revealing a key bottleneck for downstream VideoQA reasoning and deserving independent study.

To our knowledge, ours is the first work to formally recognize that grounding and trimming a video creates a semantic mismatch with complex, context-dependent queries, to formalize the task on this premise, and to extensively explore temporal filtering within the Video-LLM framework. Our results show that EMC not only improves inference-time performance but also enables more structured and effective training through finer-grained multimodal alignment, paving the way for scalable, interpretable, cognitively-inspired video-language understanding systems.

The main contributions of this work are summarized below:

• •

We propose endomorphic multimodal compression (EMC), a novel cognitively-inspired, information-theoretically framed task that guides task-aware reasoning structure filtering and addresses a key bottleneck in multi-modal alignment. We also release EMCompress, a benchmark with 238.2 hours of video and 2,754 questions.

• •

We introduce ReSimplifyIt, the first baseline for the EMC task: a model-agnostic plug-in compatible with any existing videoQA or instruction-tuning pipeline.

• •

We quantitatively evaluate EMC in VideoQA and Video Instruction Tuning, where our method delivers over $10\%$ and $5\%$ absolute gains at inference and training, respectively, across multiple strong baselines and benchmarks.

2 Formulation of Endomorphic Multimodal Compression (EMC)

2.1 Task Definition

We formulate Endomorphic Multimodal Compression (EMC) as a structurally-constrained compression problem grounded in a generative view of VideoQA. We treat the ground-truth answer $A$ as a latent fact about the world—e.g., a procedural truth such as “the step after cutting mushrooms is mixing vegetables”—with the video $V$ and query $Q$ jointly observed as $(V,Q)\sim p(V,Q\mid A)$. A video records an instantiation of the underlying fact; a natural-language query points to an aspect thereof. This mirrors the measurement-invariance principle in the physical sciences: observations access a fact rather than constitute it.

Under this view, any compression $(V,Q)\to(v,q)$ forms the Markov chain

[TABLE]

and the Data Processing Inequality (Cover and Thomas, 2006) bounds the mutual information $I(\cdot\,;\cdot)$ as

[TABLE]

with equality iff $(v,q)$ is an $A$-sufficient statistic of $(V,Q)$.222See Appendix I.4 for discussion. EMC is the problem of finding such a sufficient statistic under additional structural and VideoQA-specific constraints: temporal continuity on $v$, answer invariance at instance level across reasonable downstream models, and joint minimality on both modalities. This situates EMC within the Information Bottleneck lineage (Tishby et al., 1999; Tishby and Zaslavsky, 2015; Alemi et al., 2017), instantiated over the original multimodal input space.

Formally, EMC defines a transformation

[TABLE]

over the original multimodal task space (Figure 1), where $(v,q)$ is required to satisfy two admissibility conditions and minimize two complementary complexity measures.

Admissibility conditions.

• •

(C1) Structural Continuity. $v$ is the concatenation of $n\geq 1$ non-overlapping contiguous subsegments of $V$:

[TABLE]

with $1\leq s_{1}\leq e_{n}\leq T$, preserving low-level temporal integrity for motion, appearance, and scene dynamics.

• •

(C2) Answer Sufficiency. For any reasonable VideoQA agent $\mathcal{M}$,

[TABLE]

This is the VideoQA-natural form of the sufficient-statistic identity $I((v,q);A)=I((V,Q);A)$: VideoQA has no distribution-averaged notion of correctness—every instance has a specific ground-truth answer that the user expects the model to return, so sufficiency must hold at instance level across reasonable downstream models rather than merely on average. Under this instantiation, (C2) implies the MI identity by preserving the full distribution of model outputs,333See Appendix I.5 for discussion. locating EMC in the sufficient-statistic regime while specifying its VideoQA-specific form.

Let $\mathcal{A}$ denote the set of pairs $(v,q)$ satisfying both (C1) and (C2).

Minimality objectives.

• •

(O1) Video-side minimality. Minimize $\mathrm{Size}(v)$, the content retained from $V$ (e.g., total duration or frame count).

• •

(O2) Query-side minimality. Minimize $\mathrm{Infer}(q)$, the number of reasoning steps required to derive $A$ from $(v,q)$. For example, reasoning on “within 5 s after event A” incurs three steps: localizing event A, relational reasoning on “after”, and temporal reasoning on “5 s”.

Rather than tracing a Pareto frontier, the two objectives admit a unique solution via video-priority lexicographic resolution444A mechanism-level account—based on constraint re-allocation between modalities—of why the two objectives collapse to a single operating point rather than a Pareto continuum is provided in Appendix I.3.: video compression drives the transformation and query adaptation is its downstream consequence. Formally,

[TABLE]

2.2 Integrating EMC into Video-LLM Workflows

Because $\mathcal{F}_{\text{EMC}}$ is endomorphic, any existing Video-LLM training or inference pipeline continues to operate unchanged, receiving $(v,q)$ in place of $(V,Q)$; see Appendix H for details.

3 Method

3.1 ReSimplifyIt Framework

Drawing inspiration from dynamic and interactive attention deployment (Treisman and Gelade, 1980; Wolfe, 1994) and humans’ information compression process by dragging the video’s progress bar, we propose our multi-agent ReSimplifyIt framework. The framework is composed of three main agentic components: the Launcher, the Validator, and the Viewer, accompanied by two extra helper modules as memory trackers: the Failure History and the Success History. For implementation details, please refer to Section 4. We denote $\{(v_{r},q_{r})|1\leq r\leq R\}$ as the whole EMC process, where $v_{r},q_{r}$ represent the output video clip and question of the $r$th round, respectively, and $R$ sets the maximum number of rounds. Refer to Algorithm 2 for the complete algorithm of our EMC process performed by our ReSimplifyIt framework, and Figure 2 for a snapshot demonstration.

Initiating an EMC Round: Launcher drafting a video trimming instruction

The EMC process unfolds over iterative rounds, akin to how humans drag the progress bar multiple times based solely on empirical surmise before actually viewing the video content. In each round $r$, a Launcher module generates a trimming instruction $i_{r}$ and a revised query $q_{r}$ as a trial, where the trimming instruction may only contain high-level semantics yielding a declarative goal (e.g. “keep the clip after event X” instead of “clip([5s,10s])”), based solely on the previous query $q_{r-1}$, without vision access. This trial-based methodology strikes an effective balance between performance and efficiency as the inherent semantic relevance between the query and video input can thereby serve as a prior to enabling plausible instruction generation without video access.

Akin to a human landing on undesired segments after dragging the progress bar, we equip the Launcher module with self-correction based adaptive refinement mechanisms to tackle rare failure cases and ensure correctness. Specifically, we maintain two memory tracker modules: Success History (SH) and Failure History (FH). A successful trial is added to SH and terminates the round; otherwise, it is recorded in FH and retriggers the Launcher with updated feedback.

We formalize the Launcher’s behavior as:

$(i_{r},q_{r})=\texttt{Launcher}(q_{r-1},FH,SH,t)$

where $t$ denotes the task prompt.

Our feedback-driven formulation achieves iterative self-correction, a key distinction from prior work (Wang et al., 2024b; Shang et al., 2024). It decomposes the task into progressive rounds to enable iterative self-correction and foster structured reasoning, which enhances multi-hop robustness while minimizing correction costs. The lightweight, video-free Launcher further boosts efficiency by performing abductive reasoning on the language query without sacrificing fidelity.

Validating a trial: Validator executing instruction

The Validator module receives the high-level instruction $i_{r}$ from the Launcher and determines its success through execution. The module performs two tightly coupled tasks: (i) assessing the feasibility of the instruction (i.e., whether it is able to succeed), and (ii) executing the instruction to obtain results or feedback. Unlike the Launcher, the Validator has indirect access to visual semantics by interacting with the Viewer module while not taking visual inputs or frame captions itself.

It returns a tuple $(d_{r},m_{r})$, where:

• •

$d_{r}$ indicates whether the instruction is deemed succeeded (feasible) or failed (infeasible),

• •

$m_{r}$ contains either the resulting trimmed video (in the form of timestamp ranges) if successfully executed the instruction, or an explanation if failed.

$(d_{r},m_{r})=\texttt{Validator}(i_{r},v_{r-1},t)$

Here, $\texttt{Validator}()$ denotes the Validator module, $v_{r-1}$ is the previous video state, and $t$ is the task prompt.

The Validator may also decide to call the Viewer module before returning to Launcher. In this scenario, $d_{r}$ and $m_{r}$ stand for this decision symbol and the message to the Viewer, respectively.

Our unified Validator assesses plan viability via direct execution rather than handcrafted rules, embodying a “competence from consequence” philosophy (Brooks, 1991). This integration simplifies control flow and reduces inter-module communication. Crucially, failures are not terminal but provide constructive feedback for the Launcher to refine instructions, transforming execution into a mechanism for both validation and continual adaptation.

Scanning the video: Viewer scanning and localizing

Inspired by the Bottom-Up (stimulus-driven) and Top-Down (goal-driven) Activation schema proposed by GS2.0 (Wolfe, 1994), which also guides humans’ scrubbing the seek bar when watching videos, we design the Viewer module to explicitly incorporate two complementary tasks:

(i) Scanning: summarize the content of a video snippet given a timestamp range;

(ii) Localizing: retrieve a timestamp range given a text summarization of a video snippet.

With this complementarily symmetric design, the Viewer module achieves elegant and efficient bidirectional navigation by providing both top-down (content-driven) and bottom-up (time-driven) exploration, making the Viewer module highly flexible.

To support both tasks, we first perform a two-stage keyframe extraction and captioning process: (1) extract I-frames using MPEG-4 compression to capture key visual content, and (2) apply a dynamic frame clustering algorithm to select the final keyframes, which adaptively adjusts the number of clusters without supervision. Compared to frame clustering techniques in previous work (Wang et al., 2024c), this approach demonstrates stronger generalizability and robustness. For the algorithm of the keyframe extraction process, please refer to algorithm 1. This pre-processing is denoted as:

$C=prep(start,end).$

Scanning executes by summarizing the snippet content via LLM reasoning over $C$, optionally querying additional frames, and is denoted as:

$Cap=Scanner(prep(start,end),t).$

Localizing follows a lightweight three-stage search: (1) locate top-k candidate timestamps, (2) select the best, and (3) expand it into a full range. The LLM may optionally query extra frames for confirmation, ensuring minimal frame access while maintaining accuracy. This process is denoted as:

$(t_{start},t_{end})=Localizer(prep(0,d),q).$

This lightweight yet effective three-stage design achieves fine-grained temporal grounding with minimal overhead, showcasing the Viewer’s adaptability and plug-and-play potential.

Algorithms of the keyframe extraction and localization stages are provided in appendix A.

3.2 EMC-guided inference

Attributed to its endomorphic property, EMC can serve as a plug-and-play adapter process for any VideoQA pipeline, including both inference and training stages. Given the input question $Q$ and video $V$, EMC-guided inference is conducted as:

[TABLE]

[TABLE]

where $VideoQA$ can be any VideoQA pipeline, including video-LLMs, LLM-assisted pipelines, or any others alternatives, and $\mathcal{F}_{\text{EMC}}$ stands for the Endomorphic Multimodal Compression process.

3.3 EMC-guided training

EMC can also be applied to training stage to purify supervision signal. Given input question $Q$, input video $V$, and ground truth answer $a$, we compute the likelihood during the EMC-guided training as:

[TABLE]

[TABLE]

[TABLE]

where $f_{t}$, $f_{v}$ are the text and visual tokenizers.

4 Experiment Settings

4.1 EMCompress

As EMC is a new task, few benchmarks are capable of performing the evaluation. While some studies (Lei et al., 2018; Chen et al., 2024b; Lei et al., 2020) have explored grounded VideoQA which may be mistakenly seen as equivalent, they mainly focus on improving visual evidence. See Appendix F for details. The synchronous update of video and query inputs in EMC naturally resolves any potential semantic mismatch, maximally enhancing generalizability and adaptability of our work.

To bridge this gap, we introduce the EMCompress benchmark, which provides both EMC and standard VideoQA labels. Built upon the YouCookII dataset (Zhou et al., 2018), it comprises 2,754 datapoints split into training, validation, and test sets (roughly 7:1:2). This dual-task benchmark enables unified evaluation of both EMC and VideoQA. Refer to Appendix B for more details.

4.2 Datasets and Benchmarks

We conduct the evaluation from three aspects.

Firstly, we evaluate the quality of the EMC process on EMCompress (Section 4.1). Next, we examine the impact of EMC on downstream VideoQA performance across the following benchmarks: EMCompress, which supports evaluation of both the EMC process and VideoQA performance; ActivityNet-QA (Yu et al., 2019), NExT-OE (Xiao et al., 2021), Video-MME (Fu et al., 2024), MLVU (Zhou et al., 2024a), LVBench (Wang et al., 2024a), and EgoSchema (Mangalam et al., 2023). Lastly, we investigate the role of EMC in video instruction tuning by fine-tuning Video-LLM baselines on EMCompress, and comparing their downstream performance across various benchmarks. For NExT-QA (Xiao et al., 2021), we conduct open-ended generation during inference and map predictions to MCQ format using GPT-3.5, ensuring consistent evaluation across baselines and benchmarks (see the Appendix for details).

4.3 Implementation Details

We provide implementation details in Appendix D.

4.4 Baselines

Endomorphic Multimodal Compression

Considering the lack of baselines of the new EMC task, we adopt two baselines from the training-free temporal localization task which are considered to be robust and generalizable to unseen datasets, to make a side-by-side comparison of the video output alone of EMC. Specifically, we adopt VTG-GPT (Xu et al., 2024b), a proposal-based method which made one of the first attempts to training-free video temporal grounding, and Zheng et al. (2024)’s work which comprehend candidate proposals based on both static and dynamic matching scores. For the query output of the EMC process, we report open-ended evaluation result of our proposed framework.

EMC-guided inference

Integrating EMC as a front-end module into existing VideoQA agents yields a novel VideoQA framework. We adopt several Video-LLMs—Video-ChatGPT (Maaz et al., 2024), Video-LLaVA (Lin et al., 2023), ChatUniVi (Jin et al., 2023), LLaVA-NeXT (Liu et al., 2024b), InternVL3.5 (Wang et al., 2025), Qwen2.5-VL (Bai et al., 2025b), Qwen3-VL (Bai et al., 2025a), LLaVA-OneVision (Li et al., 2024a)—as representative baselines. We also examine LLM-assisted frameworks such as VideoTree (Wang et al., 2024c) and VideoAgent (Wang et al., 2024b), which, unlike the static encoding approaches of Video-LLMs, dynamically extract video frames based on the textual query.

EMC-guided training

Following Section 3.3, we performed video instruction tuning on Video-ChatGPT (Maaz et al., 2024), Video-LLaVA (Lin et al., 2023), and ChatUniVi (Jin et al., 2023) on EMCompress. We kept the LLM backbone frozen, and only tuned their multimodal projectors.

5 Evaluation Results

The EMC task addresses a core bottleneck in VideoQA: enabling models to focus on semantically aligned moments in long videos by abstracting task-level semantics and guiding modality-aware information filtering, which in turn enhances multi-modal alignment. We validate EMC’s importance through two complementary tests: (1) ground-truth EMC significantly boosts performance (e.g., +7.3% in Table 3), confirming it as a key supervision signal; (2) current methods fail to solve EMC effectively (Table 1), motivating our ReSimplifyIt as a plug-and-play approach.

Endomorphic Multimodal Compression

Our ReSimplifyIt framework outperforms both baselines significantly, on every metric and subset of our EMCompress benchmark. On the average performance of the whole test set of EMCompress, ReSimplifyIt achieved an mIoU score higher than 300% of the baselines’ performance. For query rewriting, our proposed framework also achieved notably good performance. Refer to Table 1 and Table 5 for results. We provide the prompt in Appendix E.

Ablation Study of ReSimplifyIt Framework

We conduct an ablation study on the ReSimplifyIt framework to evaluate the effectiveness of its design. More details are provided in Appendix C.

EMC guided inference

As shown in Table 2 and Table 6, nearly all baselines—ranging from open source models of different sizes and architectures to proprietary models—saw solid absolute performance gain, suggesting that Video-LLMs largely suffer from superfluous cognition noise. The consistent performance gains across all benchmarks underscore the universality of this general reasoning load optimization problem across diverse tasks and scenarios. In contrast, both LLM-assisted reasoning frameworks saw smaller gains. We hypothesize that the design and implementation of these frameworks have intrinsically incorporated the EMC process, by leveraging strong reasoning ability of external LLMs or multi-turn interaction.

EMC guided training

All checkpoints trained on ground truth EMC labels achieved stable performance gain over their counterparts trained on the vanilla EMCompress benchmark, while the performance gain of the checkpoints trained on the predicted EMC output of ReSimplifyIt was weaker. We argue that EMC-guided training benefits Video-LLMs better, particularly when faced with untrimmed videos and complex text instructions requiring multi-hop reasoning. Refer to Table 3 for further details.

6 Efficiency Analysis

We now quantify the practical overhead of the EMC compression loop against the savings it yields downstream. For comparability under variable conditions, we report hardware-agnostic proxies—LLM/tool/caption call counts, output tokens, duration reduction, and compression success rate—together with closed-loop metrics under realistic frame budgets.

Cost drivers and compression effectiveness.

For each sample we record the number of external LLM turns, tool invocations, total captions, and output tokens, together with the duration ratios $\mathrm{DurAll}=\mathbb{E}[|v|/|V|]$ and $\mathrm{DurScrn}$ (restricted to successfully compressed samples), and $\mathrm{Compress\%}$, the fraction of samples for which $\mathcal{F}_{\mathrm{EMC}}(V,Q)\neq(V,Q)$. See Appendix G (Tables 8, 9, and 10) for per-dataset values and the full per-source caption breakdown. ReSimplifyIt-simple averages only ${\sim}22$ captions, $3$–$10$ LLM calls, and $300$–$1{,}500$ output tokens per sample, succeeding on $94.4\%$–$100\%$ of samples across all seven benchmarks. The full multi-agent ReSimplifyIt uses more resources due to its iterative refinement, with roughly half of its captions pre-loaded in agent prompts and the rest fetched via the Viewer’s tools. Both variants reduce videos to a small fraction of their original duration on successfully compressed samples—down to $1.9\%$ (simple) and $4.8\%$ (full) on LVBench—with compression strongest on the longest benchmarks.

Closing the loop under fixed frame budgets.

A naive comparison via the count of downstream frames is misleading: a fixed budget of $K$ frames sampled uniformly over a long video is extremely sparse and may under-cover the evidence, while the same $K$ frames concentrated on the compressed segment yield much denser coverage. To make this precise, we define

[TABLE]

Here $\mathrm{DensAmp}$ captures how much denser downstream sampling becomes after compression: $K$ post-compression frames are equivalent to $\mathrm{EquivFr}(K)$ frames uniformly sampled over the original video. $\mathrm{CostRatio}(K)$ then expresses the total visual sampling cost ($K+\mathrm{TotalCap}$) as a fraction of the dense-sampling cost needed to reach the same evidence density. A lower $\mathrm{CostRatio}$ therefore means EMC achieves the same effective evidence density at a smaller fraction of the dense-sampling cost: $\mathrm{CostRatio}{=}0.1$ means the no-EMC baseline would incur $10\times$ the frame-sampling cost to reach the same evidence density. We also report the output-token cost of each $1\%$ reduction in video duration,

[TABLE]

Density amplification and end-to-end cost.

Table 4 quantifies how the compression overhead is repaid by concentrated downstream sampling. ReSimplifyIt-simple achieves $6.3\times$–$320.7\times$ density amplification; on LVBench, $8$ frames sampled from the compressed segment match the temporal density of $2{,}566$ frames sampled uniformly over the full video. The full ReSimplifyIt reaches $12.3\times$–$852.8\times$ on successfully compressed samples, with output-token costs of $3.8$–$15.1$ tokens per $1\%$ duration reduction (simple) and $25.5$–$77.3$ (full). At $K{=}8$, the simple variant’s $\mathrm{CostRatio}\leq 0.19$ on $6$ of $7$ datasets (LVBench reaching $0.01$, a ${\sim}100\times$ reduction); the full variant reports $\mathrm{CostRatio}\in[0.02,0.18]$ on the long-video benchmarks where it is most effective. As $K$ grows to $16,32,100$, $\mathrm{CostRatio}$ decreases monotonically across all datasets, indicating that EMC becomes increasingly cost-effective as downstream Video-LLMs adopt denser frame sampling. Combined with the accuracy gains of the previous section, EMC is a practically deployable front-end whose three instantiations (Appendix C) further offer an explicit performance–cost spectrum.

7 Related Work

We provide more details in appendix F.

Video-LLMs for VideoQA

Video-LLMs have spurred a wave of models aimed at enhancing video understanding (Lin et al., 2023; Ma et al., 2024; Li et al., 2024b, c; Liu et al., 2024b; Xu et al., 2024a), while the sparsity and query-invariant nature of their encoding limits efficacy in capturing fine-grained spatial-temporal details.

LLM-assisted Agentic Reasoning for VideoQA

Some other work proposing robust VideoQA baselines opt to explore pure-text LLM assisted frameworks or multi-agent systems for VideoQA (Wang et al., 2024c; Shang et al., 2024; Wang et al., 2024b). These methods adopt LLM-based methods to serve as a scheduler, which implicitly fulfills the EMC objective to a significant extent.

8 Conclusion

We introduce Endomorphic Multimodal Compression (EMC), a cognitively inspired task that reconstructs each VideoQA instance $(V,Q)$ into an answer-preserving compact pair $(v,q)$ to purify cognition load. We realize EMC via ReSimplifyIt, a plug-and-play, model-agnostic multi-agent framework, and release EMCompress to benchmark compression-centric reasoning. Across models and datasets, EMC consistently strengthens both training-time alignment and inference-time robustness, yielding notable accuracy gains and exposing bottlenecks via compact, auditable rationales, positioning information compression as a key direction for advancing video-language understanding.

Acknowledgments

This research is based upon work supported by U.S. DARPA ECOLE Program No. #HR00112390060. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of DARPA, or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for governmental purposes notwithstanding any copyright annotation therein.

Limitations

While endomorphic multimodal compression brings considerable performance gains, limitations remain and leave room for future work. Firstly, endomorphic multimodal compression is applied only on the temporal axis and cannot filter redundant spatial visual information. We regard spatial visual compression as a natural next frontier, complementary to the temporal compression studied here and opening an orthogonal axis of the broader information compression agenda. Secondly, while our plug-and-play framework adapts to any VideoQA agent, its end-to-end counterpart—Video-LLMs with inter-frame reasoning or query-adaptive frame sampling built into the encoder—is left unexplored. Both external adapters and end-to-end integration can advance information compression.

Appendix A Viewer Implementation Details

Keyframe Extraction and Captioning.

We utilize a two-stage keyframe extraction process combined with frame captioning. In the first stage, we implement the MPEG-4 compression technique (Le Gall, 1991) to extract all I-frames as candidate keyframes. I-frames typically contain rich visual content and clarity, or represent scene transitions.

In the second stage, we apply a modified Isodata clustering algorithm to the visual features of these I-frames, selecting cluster centers as final keyframes. This algorithm adaptively determines the number of clusters, unlike KNN-based clustering (Wang et al., 2024c; Zhou et al., 2024b), which requires a pre-defined number of clusters or external supervision. Our approach ensures generalizability and robustness for diverse video types.

After clustering, we utilize an off-the-shelf vision-language model (VLM) to generate frame captions for the selected keyframes. The overall process is denoted by:

$C=prep(start,end)$

where $C$ is the set of frame captions between the given timestamps.

Scanning.

Given a timestamp range $(start,end)$, the Viewer calls an external LLM with $C$ to produce an overview caption. It may query additional frames via a captioning tool to refine its summary, mimicking how users drag to specific timestamps for clarification. The process is:

$Cap=Scanner(prep(start,end),t)$

where $t$ is the task prompt.

Localizing.

To identify the timestamp range matching a textual description, we propose a lightweight three-stage search process:

Stage 1: We feed an external LLM the set of keyframe captions to let it acquire an overall capture of the video content. At the same time, we instruct the LLM to output five most possible timestamps that depicts the text query. The LLM is able to call frame caption tool to acquire extra captions at arbitrary timestamps, before it’s confident enough to output the answer. Formally:

$P=stage_{1}(prep(0,d),e,t)$

where $e$ are extra captions and $t$ is the prompt. $P$ gives the resulting list of five timestamps which depicts the language query the best.

Stage 2: We initialize the conversation and feed it the frame captions at these five timestamps, while instructing it to pick the one timestamp that best depicts the language query, out of the five candidates. Also, the LLM is able to call frame caption tool to acquire extra captions at arbitrary timestamps before it’s confident enough to output the answer.

$t_{best}=stage_{2}(P,e,t)$

where $P$ is the output of the previous step, and $t$ is the task prompt.

Stage 3: Last, we initialize the conversation again, and instruct the external LLM to expand the single timestamp $t_{best}$ from the last step to a timestamp range. The frame caption at $t_{best}$ is provided, and the LLM is still able to acquire more captions by tool calling to confirm the boundary. Considering the difficulty in dealing with dynamic transitions of events, as explored by some previous work (Zheng et al., 2024), we simply apply a hard value of 5 seconds on the output. Formally:

[TABLE]

This three-stage process balances precision and efficiency by minimizing frame access while ensuring robust temporal grounding. For visual clarity of the Isodata clustering, refer to Algorithm 1 for details.

Appendix B Details on the EMCompress benchmark

We constructed a synthetic question-answering dataset named EMCompress, based on the YouCookII dataset, to support fine-grained temporal and semantic understanding of cooking videos.

B.1 Source Data Preparation

The original YouCookII dataset (Zhou et al., 2018) contains temporally annotated instructional videos. Each annotation includes a segment $[s_{i},e_{i}]$ representing start and end times (in seconds), along with a natural language description of the cooking step.

To ensure video consistency and avoid duplications, we have verified that each video clip name is unique across the dataset source.

B.2 Annotation Grouping via Temporal Connectivity

We define a temporal connectivity criterion to group sequential cooking steps into higher-level event triplets. Given two segments $[s_{1},e_{1}]$ and $[s_{2},e_{2}]$, we define their overlap ratio as:

[TABLE]

Two segments are considered connectable if:

[TABLE]

We set $\theta=0.1$ in our experiments. Using this rule, we perform a greedy grouping of annotations into connected segments, and extract all valid length-3 subsequences (triplets) from each group.

B.3 Triplet-Based Question Generation

Each triplet $T=\{t_{1},t_{2},t_{3}\}$ consists of three temporally ordered steps. For each $T$, we generate nine different types of question-answer pairs by instantiating predefined templates. The question types are categorized into three groups:

Temporal Relational Reasoning (TRR)

• •

trr1: What is the cooking step after of $[description]$?

• •

trr2: What is the cooking step before $[description]$?

• •

trr3: What is the cooking step between $[description]$ and $[description]$?

Timepoint Indexed Reasoning (TIR)

• •

tir1: What is the step between timestamps $s_{2}$ and $e_{2}$?

• •

tir2: What is the step between frame indices $f_{s_{2}}=s_{2}\cdot r$ and $f_{e_{2}}=e_{2}\cdot r$?

• •

tir3: What step appears within $f_{d}=(e_{2}-s_{2})\cdot r$ frames after $s_{2}$ seconds?

Here, $r$ denotes the video frame rate.

Multifaceted Integrative Reasoning (MIR)

• •

mir1: What is the first step after timestamp $s_{2}$?

• •

mir2: What is the last step before timestamp $e_{2}$?

• •

mir3: Within $s_{1}$ and $e_{3}$, what is (are) the cooking step(s) apart from $[description]$ and $[description]$?

Template instantiation is performed by replacing placeholders with actual sentences and timestamps (framestamps) from the triplet.

B.4 Data Structuring and Metadata

Each generated data point is stored with the following fields:

• •

vid_name, vid_fname: Video ID and filename.

• •

vid_duration, vid_frame_rate: Metadata from video parsing.

• •

type: One of the nine QA types.

• •

question: Instantiated natural language query.

• •

answer: Corresponding ground-truth step description, serving as the ground-truth label for the VideoQA task.

• •

gt_timestamp: Temporal segment(s) serving as the ground-truth label for our EMC task on video trimming.

• •

gt_rewritten_query: Natural language query serving as the ground-truth label for our EMC task on query re-writing.

We generated a total of $N=2754$ QA samples, covering all types evenly.

B.5 Dataset Splitting

To support evaluation, we partition the dataset into training, validation, and test splits. For each question type, we allocated:

[TABLE]

This roughly follows 7:1:2.

More details about the dataset statistics can be found in Figure 4.

Appendix C Ablation Study

C.1 Experiment Settings

We first ablate the modular design—responsible for facilitating structured reasoning—by proposing the ReSimplifyIt-simple variant. In this setting, all underlying interfaces for handling textual and visual inputs, such as ISODATA-clustering (Algorithm 1) and frame-caption querying, are preserved. However, the entire reasoning pipeline is collapsed into a single, universal agent. Specifically, we initialize an external agentic LLM with task descriptions and operational instructions, and expose the aforementioned interfaces either through conversational context or tool invocation. This unified agent is then responsible for all reasoning procedures—functionally covering the roles of the Launcher, Validator, and Viewer modules, as well as memory tracking—in the original ReSimplifyIt framework, and ultimately produces the final output.

Then, we further ablate the video access completely, i.e., no access to the video content is provided throughout the entire reasoning process of the external agent. As the multi-turn interactions are essentially references to video information for refining text input, disabling video access also renders multi-turn interactions unnecessary. To reflect this, we introduce the ReSimplifyIt-blind variant, in which a tool-calling LLM generates a rewritten query and a fixed sequence of tool invocations within a single-turn conversation. This sequence is subsequently executed by a separate executor module to produce the video output.

Refer to Figure 5 for an illustration of these frameworks.

C.2 Evaluation Results

Evaluation results are presented in Table 7. Although the removal of modular design yields comparable video outputs, the modular architecture still demonstrates advantages—particularly on the Multifaceted Integrative subset, which involves more complex multi-hop reasoning, demonstrating the effectiveness of modular design and structured reasoning in more complex and intricate reasoning scenarios. In comparison, the ablation of video access brings notable performance drop, underscoring the critical role of cooperative, feedback-driven multi-turn interaction. This aligns with the interdependency of the vision and text modality, which lies at the core of our EMC task formulation, reflecting the essence of the task initiative.

A performance–cost spectrum of EMC instantiations.

Taken together, ReSimplifyIt, ReSimplifyIt-simple, and ReSimplifyIt-blind expose an explicit performance–cost spectrum of EMC rather than a single operating point. The full ReSimplifyIt prioritizes robustness and exploratory reasoning through iterative multi-agent interaction, at the cost of more tool calls, captions, and LLM turns. ReSimplifyIt-simple flattens the interaction structure into a single unified agent to substantially reduce overhead, at a small accuracy cost. ReSimplifyIt-blind further removes all video access during compression, representing the lowest-cost extreme in which the trimming plan is produced entirely from the language query. This spectrum demonstrates that EMC can be configured either for efficiency-oriented deployment (Simple/Blind) or for maximal reasoning capability (Full) depending on application requirements, and that the observed gains arise from the compression mechanism itself rather than from any fixed amount of inference compute. Section 6 quantifies the cost characteristics of each variant across benchmarks.

C.3 Full list of tools for ReSimplifyIt-blind framework

⬇

1. get_duration():

   Return the duration of the video as a floating point value.

2. get_resolution():

   Return the resolution of the video, as a tuple.

3. get_total_frame_num():

   Return total number of the frames of the video, as an integer.

4. grounding_select(obj_name, concerned_indices_input):

   Return, in the form of a list of integers, the indices of all frames

   containing the object given by obj_name, after taking the intersection

   of indices provided by concerned_indices_input. If None is passed,

   selects all frames.

5. indices_list_intersect(list1, list2):

   Return the intersection of two lists of indices.

6. indices_list_union(list1, list2):

   Return the union of two lists of indices.

7. indices_concat_and_fill(list1, list2):

   Return the sorted union of list1 and list2, then fill in missing

   values to make the sequence continuous.

8. indices_concat(list1, list2):

   Return the concatenation of the two lists.

9. timestamp_to_single_index(timestamp):

   Return the frame index corresponding to the given timestamp (in seconds).

10. single_timestamp_to_index_range(timestamp):

    Return indices of 60 consecutive frames centered at the timestamp.

11. range_timestamp_to_index_range(start, end):

    Return all frame indices between the start and end timestamps.

Appendix D Implementation details of ReSimplifyIt framework

The Launcher module is based on single-turn conversation with GPT-4o. The Validator and Viewer module, including the scanner and localizer, are primarily based on multi-turn conversation with GPT-4o. For ISODATA frame clustering, we use CLIP (Radford et al., 2021) to obtain the visual features of the I-frames. We adopt the off-the-shelf LLaVA-1.5 (Liu et al., 2024a) for frame captioning.

Appendix E Prompts

E.1 Prompt used for evaluation of query rewriting of EMC

⬇

You are a helpful assistant to evaluate the quality of an output of a special type of sentence compression, which sentence is in the form of a question.

You will be given an output of such compression process and the ground truth answer, and the original input question sentence before the compression. Please evaluate the output based on the following three criterias:

1. Relevance: to minimize unwanted information

- in this criteria, a candidate output gets full mark if it doesn’t contain any information (phrases, concepts, etc.) out of the scope of the original input question.

- the more information it contains that is not mentioned in the original input question, the more marks are deducted.

2. Simplicity: to minimize tangential information

- in this criteria, a candidate output gets full mark if doesn’t contain any information (phrases, concepts, etc.) that is included in the original input question but not included in the ground truth compressed question. In other words, whether the question sentence is fully compressed.

- the more information it contains that is included in the original input question but not included in the ground truth question, the more marks will be deducted.

3. Completeness: to minimize over-compression

- in this criteria, a candidate output gets full mark if it contains all information included in the ground truth compressed question.

- the more information contained in the ground truth output question is found missing in the output compressed question, the more marks will be deducted.

Here is the original input question: [original_question_flag], and

here is the ground truth compressed question: [ground_truth_compressed_question_flag], and

here is the output compressed question: [output_compressed_question_flag].

Please rate the output compressed question on a scale from 0 to 100, with 0 being the worst and 100 being the best (full mark).

Now, rate the quality of the output compressed question based on all the information above.

Return your answer in this json format: {"score": [your score, from 0 to 100]}.

E.2 Prompts used for ReSimplifyIt framework

Prompt for the Launcher module:

⬇

Given a video question answering case, i.e. a video, a question, and an answer, sometimes it is possible to cut the video by only keeping a sub-clip or several sub-clips (concatenate if so) to be the video output and simultaneously modify the text question to be the paired text output while keeping the answer consistent and unchanged, so that the answer is still completely compatible to the modified question and the obtained video clip.

We define this action as "compression". By doing compression, the video becomes shorter, and therefore introduces lower cost to the downstream video question answering model. When modifying the question, note that the downstream video question answering model will only be seeing the sub-clip and think that the shown sub-clip is the whole video, so make sure the modified question is perfectly paired and aligned with and adapted to the modification.

[In context examples flag 1]

Sometimes, such compression can be repeatedly applied sequentially by several rounds, where the input in each round is the output video clip and the output modified question of its preceeding round. As mentioned, the answer consistency should be always ensured throughout the whole process, and is always completely compatible to the resulting video clip and modified question of every round. If succeeded, each round would make the video shorter and making the situation closer to the optimal case.

[In context examples flag 2]

Your duty is to help complete the compression. Due to some limit, you are only provided with the text question but not the video input.

Your task is to initiate an immediate plan for the compression operation, including a natural language instruction telling which video clip(s) to keep (similar to examples above) and the paired modified text question. If you think the compression may compose more than one round, you only need to perform one round next.

As you cannot see the actual video, your plan might fail, if the downstream video editting agent taking and executing your instruction on video clipping finds it infeasible. Therefore, your plan is refered to as a ’trial’.

In your current situation, some rounds or trials of the compression process might have already been conducted, and privided as following information:

1, the ’failure history’: it is about the history of failed previous trials of the compression of your current round. Here is the failure history for you (an empty indicates that you are making the first trial of this round): [failure_history_flag];

2, the ’success history’: it is about the history of the one success trial of all previous round. Here is the failure history for you (an empty indicates that you are at the first round): [success_history_flag].

Now, here is the original question of this round for video question answering: "[quesion_flag]"

Again, please tell your plan which describes what part of the video should be kept. Also, give the modified question under the assumption that the video is processed smoothly according to your plan.

If you feel that there is no room to make such compression (e.g. when the question is being general like "what is this video about") so you feel that you shouldn’t make any plan, you should decide to terminate the process.

Hints:

1. Before processing, remember to take a look in the ’failure history’ and ’success history’ information;

2. If you find that the success history contains a case whose modified_question and the description both significantly overlap with the ones you are about to make, then you should avoid making the same plan again. In this case, you should switch to a clear reasonable sensible alternative plan, or make a decision to terminate the modification process if you can’t confidently find one;

3. If you find that the failure history contains a case whose modified_question and the description both significantly overlap with the ones you are about to make, or that any of the "reason" in the failure history records is going to make your attempt fail, then you should avoid making the same plan again. In this case, you should switch to a clear reasonable sensible alternative plan, or make a decision to terminate the modification process if you can’t confidently find one.

Return your plan in this json format (keep in mind here, that your response should be in json format):

{"decision": [your decision, either "process" or "terminate"], "modified_question": [the modified question, or "N/A" if your previous decision is "terminate"], "description": [Description of what part of the video should be kept as wanted sub-clip. The description will be passed to downstream processer to validate. Return "N/A" if your decision is "terminate".]}

"""

Prompt for the Validator module:

⬇

You will be given a natural language instruction telling you to trim a video, which instruction itself might be infeasible. The reason it might be infeasible is because the agent who gave the instruction had no access to the actual video content, so it might be infeasible if take the actual video contenet into consideration.

Therefore, I need you to be a helpful assistant to confirm if the trimming plan is feasible.

Specifically, your job is to act as a validator to validate whether it is feasible (whether the video content really supports the plan).

If it is feasible, you will need to implement the plan and return the resulting sub-clip of the plan in the form of a two-layer list. [In context example flag 1]

If it is not feasible, you will need to tell the reason.

Here are some examples and explanations for infeasible trimming instructions:

[start of examples]

[In context examples flag 2]

[end of examples]

You can invoke a viewer multiple times to acquire the video content (partial content each time), which viewer is a downstream module prepared to assist you.

Note that the viewer is capable to deal with two types of questions:

1. snippet rough scanning, e.g. "what is the video about from xxx second to xxx second"?

2. localizing, e.g. "which segment contains xxx (event/object)?"

Therefore, if [your decision] is "view", [your message] should follow one of the above two example templates.

Here is the trimming instruction you need to validate: [plan_flag].

The video length is [video_length_flag] seconds, and its frame rate is [frame_rate_flag] fps.

Now, in each following turn of this conversation, you need to give your response in this json format: {"decision": [your decision], "message": [your message]}. This works as follows:

[your decision]: either "succeeded", "failed", or "view". "succeeded" means that the plan is successfully implemented, "failed" means that it is not, and by "view" you invoke the viewer to provide partial video content for you. If you choose "view", the viewer will take your message and return as you requested, and the conversation will continue. If you choose "succeeded" or "failed", it will be your final decision and the conversation will end.

[your message]: if you choose "succeeded" as your decision, then it should be the two-layer list as mentioned before, as the video edit result of the plan. If you choose "failed", this should be a brief reason on the failure (e.g. requested timestamp exceeds video length, video doesn’t have the object/event needed, etc.). If you choose "view", this should be the question to ask the viewer about the video content.

Hint: it is not always necessary to invoke the viewer. [In context example flags 3]

For your ease of decision, here are some initial frame captions and their timestamps for you (in the form of key value pairs, where the value is the frame caption and the key is its corresponding timestamp, in the unit of second): [initial_captions_flag].

[sys_usr_split]

Now let’s start!

Prompt for Viewer module:

⬇

You are a helpful and smart assistant that can respond to an upstream request about a video by invoking tools. The length of this video is [video_length_flag].

Here is the content of the request: [validator_request_flag].

Here are the tools you can access (you might access them multiple times if you want):

1. scan(start, end): Return the overall caption of the video snippet (clip) between start and end timestamp, which are the parameters with the unit of second.

2. localize(query): Return the video location, in the form of a timestamp range given by the start and end timestamp, which contains the visual content of the query (which query might be an object, event, etc.)

3. get_image_cap(timestamp): parameter "timestamp" is an integer in the unit of second. Return the caption of the video frame at the given timestamp (regard the frame as an image).

Now, in each turn of the following conversation, your response should be in the following json format: {"decision": [your decision], "message": [your message]}. This works as follows:

[your decision]: either "tool" (if you wish to call tool in this round) or "respond" (if you feel that you are able to respond to the upstream module’s request by comprehending your current knowledge acquired about the video.).

[your message]: if your decision is "tool", then this should only contain tool calling following given format, e.g. ’get_image_cap(10)’, ’scan(21, 35)’, ’localize("kicking the ball")’. If your decision is "respond", then this should be your response to the upstream request.

[sys_usr_split]

Now let’s start!

E.3 Prompt used for ReSimplifyIt-simple framework

⬇

You are a assistant for the video question answering process, in which a candidate is presented with a video and a question for them to answer.\

Your objective is to help the candidate so that they will be able to give the answer with watching the shortest posible sub-clip(s) of the video. \

Your task is to cut the video to acquire this sub-clip(s) and also to modify the question, so that the candidate directly answering your modified question with presented only this sub-clip(s) of the video would be equivalent to answering the original question with presented the original whole uncut video. \

[In context examples flag 1]

You will need to cut the video in the form of providing me the timestamps, which is a list of [start, end] unit clips in the unit of second. \

A tool (python function) will be helping you to get the frame caption of at a certain timestamp (in the unit of second). Whenever you need to call this tool, send a message in this json format: {"decision": "tool", "parameter": [timestamp you need]}. [In context example flag 2] \

Whenever you think you are confident enough to provide the timestamp, return {"decision": "end", "timestamps": [your result timestamps], "revised_question": [your revised question]}. [In context example flag 3].

Before we formally begin, here is a set of original captions with their timestamps provided for you to have an overall rough understanding of the video: [initial_captions_flag].

Also, the frame rate of this video is [frame_rate_flag] frames per second, and the total duration is [duration_flag].

[sys_usr_split]Now let’s begin! and the original question is "[original_question_flag]".

E.4 Prompt used for ReSimplifyIt-blind framework

⬇

You are a assistant for the video question answering process, in which a candidate is presented with a video and a question for them to answer.\

Your objective is to help the candidate so thattheywill be able to give the answer with watching the shortest posible sub-clip(s) of the video. \

Your task is to cut the video to acquire this sub-clip(s) and also to modify the question, so that the candidate directly answering your modified question with presented only this sub-clip(s) of the video would be equivalent to answering the original question with presented the original whole uncut video. \

[In context examples flag 1]

You will be provided with a list of tools to process the video, and the original question to be answered by the candidate based on which to select the frames. Here is the list of tools you have access to, with the description (content in the brackets are the arguments needed):

[1]: get_duration(): return the duration of the video as a floating point value.

[2]: get_resolution(): return the resolution of the video, as a tuple.

[3]: get_total_frame_num(): return total number of the frames of the video, as an integer.

[4]: grounding_select(obj_name, concerned_indices_input): return, in the form of a list of integers, the indices of all frames containing the object given by obj_name, after taking the intersection of indices provided by the argument ’concerned_indices_input’. ’concerned_indices_input’ is also a list of indices, and will be set to indices of all frames in the video if ’None’ is passed.

[5]: indices_list_intersect(frame_indices_list_1, frame_indices_list_2): return, in the form of a list of integers, the intersection of the two arguments as list. Both argument ’frame_indices_list_1’ and ’frame_indices_list_2’ are a list of indices.

[6]: indices_list_union(frame_indices_list_1, frame_indices_list_2): return, in the form of a list of integers, the union of the two arguments as list. Both argument ’frame_indices_list_1’ and ’frame_indices_list_2’ are a list of indices.

[7]: indices_concat_and_fill(frame_indices_list_1, frame_indices_list_2): first take the sorted union of the two lists given by the arguments, and then fill in all the missing values so that every two adjacent element only differ by 1. Both argument ’frame_indices_list_1’ and ’frame_indices_list_2’ are a list of indices.

[8]: indices_concat(frame_indices_list_1, frame_indices_list_2): return, in the form of a list, the concatenation of the two lists provided by the arguments.

[9]: timestamp_to_single_index(timestamp): return a list with a single integer, which integer is the index of the frame at the given timestamp. The argument timestamp is a floating point value, whose unit is second.

[10]: single_timestamp_to_index_range(timestamp): return, in the form of a list, the indices of 60 consecutive frames, the midpoint of which is at the given timestamp. The argument timestamp is a floating point value, whose unit is second.

[11]: range_timestamp_to_index_range(start, end): return, in the form of a list, the indices of all frames which are between the two timestamps which are provided by the arguments. The argument start and end are both floating point values, whose unit are both second.

Above are all the tools to have access to. Please note that selecting frames out of all the frames of the original video is being cut and clipped, therefore you will also need to modify the aforementioned prompt, to make it align well with the reduced video frames.

[sys_usr_split]

Now the original question is: [question_flag]. Having access to the information of all the tools mentioned above, provide me the python code which could achieve the selection of frames. You may define variables to store intermediate result, and determine the value of some arguments when necessary, but you should not require the downstream task operator to replace any of your assumption on arguments, as no more information but the original video is provided to the downstream task. Please use the variable name ’final_frames’ to store your final list of frame index. Please only provide the code and revised question in this format: {"Code:[your whole paragraph of code] Revised question:[your revised prompt]"}, where [your whole paragraph of code] should be an empty string if you think no tools need to be called and the whole original video should be passed to the downstream task.

Appendix F More on Related Work

Video-LLMs for VideoQA

Video-LLMs have spurred a wave of models aimed at enhancing video understanding by leveraging the language capabilities of large language models (LLMs) (Lin et al., 2023; Ma et al., 2024; Li et al., 2024b, c; Liu et al., 2024b; Xu et al., 2024a; Wang et al., 2025; Bai et al., 2025b, a; Li et al., 2024a). Some approaches (Chen et al., 2023; Li et al., 2024b, 2023) utilize dedicated video encoders such as video transformers (Bertasius et al., 2021) or convolution networks. However, these designs often demand large-scale annotated video-text data and significant computational resources. To address this, alternative methods adapt pre-trained image-domain MLLMs to video inputs (Liu et al., 2024c; Maaz et al., 2024), offering improved practicality. Nonetheless, the sparsity and query-invariant nature of video encoding in existing models limits their ability to capture fine-grained spatial-temporal details effectively, especially under token budget constraints. This work addresses such inefficiencies by introducing a query-adaptive processing mechanism inspired by the principle of information compression, aiming to reconcile the trade-off between token efficiency and representational fidelity.

LLM-assisted Agentic Reasoning for VideoQA

Another line of research for video question answering lies in building pure-text LLM assisted frameworks or multi-agent systems for VideoQA (Wang et al., 2024c; Shang et al., 2024; Wang et al., 2024b). Compared to Video-LLMs, which represents end-to-end single-pass approaches, these methods adopt traditional or LLM-based methods to proactively sample relevant video frames. VideoAgent (Wang et al., 2024b) and TravLER (Shang et al., 2024) utilized LLM’s planning ability to conduct iterative keyframe searching, while VideoTree (Wang et al., 2024c) presented query-adaptive hierarchical tree-based keyframe selection.

Temporal Sentence Grounding for Videos

Temporal sentence grounding (TSG) aims to localize the video segment best matching a language query. Early sliding-window methods (e.g., TALL (Gao et al., 2017), MCN (Hendricks et al., 2017)) were costly, while later proposal-based (Xu et al., 2019; Chen and Jiang, 2019) and proposal-free methods (Yuan et al., 2019; Zhang et al., 2020) improved efficiency via query-guided proposals or direct boundary prediction. All these methods take a video and a query as input and predict a matching temporal span. In contrast, our proposed EMC generalizes beyond TSG by supporting inter-frame reasoning and joint adaptation over both video and query, rather than retrieving only frames directly mentioned in the query.

Grounded videoQA

Another seemingly similar line of research lies in integrating grounding techniques into VideoQA pipelines (Xiao et al., 2024; Chen et al., 2024a). Grounded VideoQA seeks to enhance model faithfulness by explicitly linking a model’s textual output to “visual cues” or “evidence” within the video. This approach is effective at mitigating hallucinations for descriptive queries by enforcing perceptual alignment. While effective for descriptive tasks, the approach suffers from a fundamental conceptual incongruity with the nature of complex video reasoning. The semantics of a query and its corresponding video segment are often holistically entangled; for example, a high-level question about intent, causality, procedure, or even a simple temporal relational reasoning does not map to an atomic piece of visual evidence but is inferred from a continuous temporal context. The attempt to impose a discrete justification framework onto this intertwined semantic space leads to an inherently brittle and ill-defined notion of a “visual cue”.

EMC naturally sidesteps this ambiguity by fundamentally reframing the objective, turning from “what to answer” to “what to ask”. More fundamentally, EMC introduces a more principled paradigm that respects this intrinsic entanglement. Rather than seeking to atomize evidence for a given answer, EMC reformulates the task itself through a priori contextual simplification. It isolates the minimal, self-contained $(video,query)$ sub-problem through an endomorphic transformation that keeps with the original task space unchanged and preserves the necessary holistic context for reasoning. This approach is not merely a circumvention of the definitional challenge; it establishes a more fundamentally sound and cognitively aligned task. By first reducing the problem space to a manageable and semantically coherent unit—mirroring the human strategy of simplifying a problem before attempting to solve it—EMC offers a more robust foundation for complex video-language understanding.

Information Bottleneck

The Information Bottleneck principle (Tishby et al., 1999; Tishby and Zaslavsky, 2015) formalizes representation learning as a sufficiency–compression trade-off, extended by the variational IB (Alemi et al., 2017) to deep models. EMC shares the same sufficient-statistic substrate but operates in a different regime along two axes: it compresses over the original multimodal input space rather than a learned latent code, and it realizes sufficiency as a pointwise, model-agnostic condition on task behavior (C2) rather than as a distribution-level regularizer, yielding the interpretable discrete artifact $(v,q)$.

Appendix G Cost Driver Details

This section provides per-dataset cost-driver statistics for both EMC instantiations, complementing the headline CostRatio analysis of Section 6. Table 8 reports the high-level cost drivers (caption count, tool calls, LLM turns, output tokens) and compression effectiveness (DurAll, DurScrn, Compress%) for ReSimplifyIt-simple. Table 9 reports the same high-level statistics for the full ReSimplifyIt. Table 10 further decomposes the captions of the full ReSimplifyIt into passive (pre-loaded) versus active (tool-fetched) sources; column definitions follow the list below.

• •

Pasv: passive captions per sample ($=\text{VaIn}+\text{LoIn}$), embedded in prompts without explicit tool calls.

• •

VaIn: Validator initial captions, uniformly sampled frames embedded in the Validator’s prompt ($\sim$10 per trial).

• •

LoIn: Localize initial captions; each localize() call samples $k{=}10$ uniform frames for its sub-agent prompt.

• •

Actv: active captions per sample ($=\text{\#TotalCap}-\text{Pasv}$), fetched during tool execution via scan, localize, and get_image_cap.

• •

V$\rightarrow$Vw: Viewer sessions spawned per sample (number of times the Validator invokes the Viewer as a tool).

• •

#scan, #localize, #get_cap: the breakdown of total Viewer tool calls (VwTl) into its three constituent tool types.

Appendix H EMC Integration into Video-LLM Workflows

We expand the deployment modes summarized in Section 2.2. Because $\mathcal{F}_{\text{EMC}}$ is endomorphic, both modes slot into existing pipelines with no architectural change.

EMC for Inference-Time Simplification in Video-LLMs.

At inference, EMC pre-processes $(V,Q)$ into $(v,q)$ (see Section 3), reducing the temporal and semantic complexity the downstream model must handle and concentrating it on compact, task-relevant content for fine-grained temporal reasoning and direct visual grounding.

EMC for Improving Training-Time Visual-Language Alignment.

Video-LLM supervision tuples $(V,Q,a)$ often couple abstract or multi-hop queries with unfocused video. EMC reshapes them into compact triples $(v,q,a)$, where $v$ is a high-utility segment and $q$ is a grounded reformulation of $Q$, offloading off-target supervision to upstream modules so the model specializes in two core competencies: temporal understanding and multimodal alignment.

Appendix I Additional Discussion

I.1 EMC vs. Test-Time Reasoning Strategies

EMC and test-time reasoning strategies such as Chain-of-Thought (Wei et al., 2022), Tree-of-Thoughts (Yao et al., 2023), and Graph-of-Thoughts (Besta et al., 2024) address different bottlenecks and are not substitutes for one another. Test-time reasoning strategies primarily modify how an LLM explores or restructures its thought process given a fixed input; they do not directly address the sparsity of evidence in long videos, where the dominant failure mode is that a fixed downstream frame budget under-covers the relevant segment to begin with. By contrast, EMC is a front-end endomorphic problem transformation that reshapes the evidence distribution itself: by compressing $(V,Q)\to(v,q)$ under answer invariance, it concentrates the same downstream frame budget onto a short relevant segment, yielding the density gains quantified by our $\mathrm{DensAmp}$ and $\mathrm{CostRatio}$ metrics in Section 6.

These two directions are therefore orthogonal and complementary: adding reasoning steps does not automatically reproduce EMC’s benefit unless it is coupled with explicit evidence selection at the sampling stage, while EMC can be composed with any downstream reasoning strategy. Our framework further exposes a performance–cost spectrum rather than a single operating point (ReSimplifyIt, ReSimplifyIt-simple, and ReSimplifyIt-blind; Appendix C), with progressively lower amounts of inference compute. The fact that compression gains persist even in the flattened and blind variants supports that EMC’s benefit stems from the evidence-compression mechanism rather than from the sheer amount of LLM compute invested.

I.2 Minimality as a Desideratum and Boundary Expansion

The minimality condition in our formulation (Section 2.1) describes a target property of the task formalization—that the output segment $v$ should contain the minimal sufficient visual evidence for answering $q$—rather than a strict guarantee enforced by any particular implementation. In practice, event boundaries in real-world videos are often ambiguous or gradual, and strict minimal cropping is a known failure mode: small boundary errors can easily exclude critical visual evidence and lead to disproportionately large downstream reasoning failures. Our Localizer (Appendix A) therefore applies a conservative temporal expansion of $\pm p$ seconds around the predicted boundary as a safety margin. Condition (C2) is a hard admissibility requirement, not a trade-off dimension; but the Localizer’s boundary predictions are stochastic, and overly aggressive crops can drop frames whose removal would in fact violate (C2) on some downstream model. The $\pm p$ margin is therefore a robustness buffer for enforcing (C2) under localization uncertainty: smaller $p$ risks genuine (C2) violations (under-inclusion of $A$-relevant evidence), while larger $p$ merely loosens the minimality objectives (O1–O2) by retaining redundant content without affecting admissibility. The chosen $p{=}5$ is the operating point at which (C2) is reliably enforced without excessive redundancy.

Sensitivity to the expansion margin.

To validate that $p{=}5$ is not arbitrary, we vary $p\in\{1,3,5,7\}$ and report the resulting average mIoU on EMCompress:

[TABLE]

We observe a clear trade-off: too small a margin (1–3s) risks missing event boundaries and lowers overlap with the ground truth, while too large a margin (7s) unnecessarily enlarges the segment and reduces localization precision. A moderate margin of $5$ seconds achieves the best balance, and we adopt it as the default in all reported results. An adaptive, confidence-aware or boundary-aware expansion strategy would be an interesting direction for future work.

I.3 Constraint Re-allocation and the Collapse of the Pareto Frontier

A priori, the video-side minimality (O1) and query-side minimality (O2) in Section 2.1 admit a Pareto trade-off, so the optimum could span an entire frontier rather than a unique point. Note that this concerns the O1–O2 minimization only; admissibility (C1–C2) remains a hard constraint throughout and is not part of the Pareto structure. In EMC, however, the two minimality objectives are not independent—they are two faces of a single underlying compression act. Each reasoning step removable from $q$ corresponds to a constraint in the original query that can be structurally pre-satisfied by trimming $V$: rather than verifying the constraint textually, the transformation excises the non-compliant portion of the video. Consequently, every reduction in $\mathrm{Infer}(q)$ is paid for by an equivalent reduction in the span of $V$ that must be retained to remain $A$-sufficient, and vice versa. The two objectives descend in lockstep along the admissible region, and the Pareto frontier collapses to essentially a single operating point.

We emphasize that this re-allocation is a mechanism-level account of why a unique optimum exists; it does not redefine the task. The answer $A$ and the answer space are preserved throughout—no constraint is discarded, each is merely conserved across modalities—so EMC remains a joint multimodal transformation, distinct from unimodal query relaxation or evidence-retrieval paradigms (Xiao et al., 2024; Chen et al., 2024a).

This mechanism further justifies the video-priority lexicographic resolution adopted in Section 2.1: video compression is the driver of the transformation and query adaptation its downstream consequence—once $v^{*}$ is fixed, the remaining freedom in $q$ is exhausted by rewriting to the shortest form compatible with $v^{*}$.

I.4 On Measurability of the Information-Theoretic View

The mutual-information terms in $I((v,q);A)\leq I((V,Q);A)$ articulate EMC’s membership in the sufficient-statistic family; they are a theoretical characterization of the task, not a computable training signal to be estimated numerically. In practice, sufficiency is realized through the VideoQA-natural condition (C2), and minimality is tracked through the computable proxies $\mathrm{Size}(v)$ and $\mathrm{Infer}(q)$. The information-theoretic view thus provides the language of the formulation, while the concrete conditions (C1–C2) and objectives (O1–O2) provide its operational content.

I.5 On the Link between Mutual Information and Answer Invariance

The use of mutual information to capture answer preservation rests on the classical sufficient-statistic property: $I((v,q);A)=I((V,Q);A)$ if and only if there exists a decision rule on $(v,q)$ achieving the same Bayes-optimal performance on $A$ as any decision rule on $(V,Q)$ (Cover and Thomas, 2006, Ch. 2.9). MI equality is therefore a population-level characterization—the existence of a sufficient-capacity model whose predictions are invariant across the compression. Condition (C2) is the VideoQA-natural instantiation of this same sufficiency property: because VideoQA has no distribution-averaged notion of correctness, sufficiency must hold pointwise across all reasonable $\mathcal{M}$, not merely in expectation. The pointwise form implies the MI equality by preserving the full distribution of model outputs, so (C2) is simultaneously a concrete behavioral condition and a realization of the sufficient-statistic regime in the VideoQA setting.