Commander-GPT: Dividing and Routing for Multimodal Sarcasm Detection

TL;DR

Commander-GPT is a modular framework that orchestrates specialized LLM agents for multimodal sarcasm detection, significantly improving accuracy over existing methods.

Contribution

It introduces a decision routing framework with multiple types of commanders to coordinate specialized LLM agents for sarcasm detection.

Findings

Achieves 4.4% and 11.7% F1 score improvements over SoTA baselines.

Demonstrates effectiveness across MMSD and MMSD 2.0 benchmarks.

Utilizes various commander models, including lightweight and large LLMs.

Abstract

Multimodal sarcasm understanding is a high-order cognitive task. Although large language models (LLMs) have shown impressive performance on many downstream NLP tasks, growing evidence suggests that they struggle with sarcasm understanding. In this paper, we propose Commander-GPT, a modular decision routing framework inspired by military command theory. Rather than relying on a single LLM's capability, Commander-GPT orchestrates a team of specialized LLM agents where each agent will be selectively assigned to a focused sub-task such as keyword extraction, sentiment analysis, etc. Their outputs are then routed back to the commander, which integrates the information and performs the final sarcasm judgment. To coordinate these agents, we introduce three types of centralized commanders: (1) a trained lightweight encoder-based commander (e.g., multi-modal BERT); (2) four small autoregressive language models, serving as moderately capable commanders (e.g., DeepSeek-VL); (3) two large LLM-based commander (Gemini Pro and GPT-4o) that performs task routing, output aggregation, and sarcasm decision-making in a zero-shot fashion. We evaluate Commander-GPT on the MMSD and MMSD 2.0 benchmarks, comparing five prompting strategies. Experimental results show that our framework achieves 4.4% and 11.7% improvement in F1 score over state-of-the-art (SoTA) baselines on average, demonstrating its effectiveness.