Weakest Link in the Chain: Security Vulnerabilities in Advanced Reasoning Models

TL;DR

This paper systematically evaluates the security vulnerabilities of advanced reasoning language models, revealing nuanced differences in robustness compared to non-reasoning models across various attack types.

Contribution

It provides a comprehensive comparison of adversarial prompt vulnerabilities between reasoning and non-reasoning models, highlighting category-specific strengths and weaknesses.

Findings

Reasoning models are slightly more robust overall (42.51% vs 45.53% attack success rate).

Certain attack categories show reasoning models are significantly more vulnerable.

Other categories demonstrate reasoning models are substantially more robust.

Abstract

The introduction of advanced reasoning capabilities have improved the problem-solving performance of large language models, particularly on math and coding benchmarks. However, it remains unclear whether these reasoning models are more or less vulnerable to adversarial prompt attacks than their non-reasoning counterparts. In this work, we present a systematic evaluation of weaknesses in advanced reasoning models compared to similar non-reasoning models across a diverse set of prompt-based attack categories. Using experimental data, we find that on average the reasoning-augmented models are \emph{slightly more robust} than non-reasoning models (42.51\% vs 45.53\% attack success rate, lower is better). However, this overall trend masks significant category-specific differences: for certain attack types the reasoning models are substantially \emph{more vulnerable} (e.g., up to 32 percentage points worse on a tree-of-attacks prompt), while for others they are markedly \emph{more robust} (e.g., 29.8 points better on cross-site scripting injection). Our findings highlight the nuanced security implications of advanced reasoning in language models and emphasize the importance of stress-testing safety across diverse adversarial techniques.