COGNITION: From Evaluation to Defense against Multimodal LLM CAPTCHA Solvers

TL;DR

This paper evaluates how multimodal large language models can break visual CAPTCHA security, analyzes their strengths and weaknesses, and proposes design guidelines to improve CAPTCHA robustness against such AI-driven attacks.

Contribution

It provides a comprehensive evaluation of MLLMs on various CAPTCHA types, analyzes their reasoning mechanisms, and offers practical guidelines for designing more secure CAPTCHAs.

Findings

MLLMs can reliably solve recognition-oriented and low-interaction CAPTCHAs at human-like cost.

Tasks requiring localization or multi-step reasoning remain challenging for current models.

Hardening CAPTCHA with localization and counting reduces success rate from over 95% to 0%.

Abstract

This paper studies how multimodal large language models (MLLMs) undermine the security guarantees of visual CAPTCHA. We identify the attack surface where an adversary can cheaply automate CAPTCHA solving using off-the-shelf models. We evaluate 7 leading commercial and open-source MLLMs across 18 real-world CAPTCHA task types, measuring single-shot accuracy, success under limited retries, end-to-end latency, and per-solve cost. We further analyze the impact of task-specific prompt engineering and few-shot demonstrations on solver effectiveness. We reveal that MLLMs can reliably solve recognition-oriented and low-interaction CAPTCHA tasks at human-like cost and latency, whereas tasks requiring fine-grained localization, multi-step spatial reasoning, or cross-frame consistency remain significantly harder for current models. By examining the reasoning traces of such MLLMs, we investigate the underlying mechanisms of why models succeed/fail on specific CAPTCHA puzzles and use these insights to derive defense-oriented guidelines for selecting and strengthening CAPTCHA tasks. To validate these principles, we perform a case study by hardening a vulnerable CAPTCHA type using our guidelines. We demonstrate that incorporating fine-grained localization and implicit counting reduces the success rate of state-of-the-art MLLMs from over 95% to 0%, confirming that structural changes can effectively mitigate the threat. We conclude by discussing the implications for platform operators who deploy CAPTCHA as part of their abuse-mitigation pipeline.Code Availability (https://anonymous.4open.science/r/Captcha-465E/).