CARO: Chain-of-Analogy Reasoning Optimization for Robust Content Moderation

TL;DR

CARO is a novel two-stage training framework that enhances large language models' ability to perform robust analogical reasoning for content moderation, significantly reducing harmful decision shortcuts.

Contribution

Introduces a two-stage training method combining retrieval-augmented generation and preference optimization to improve LLM reasoning in moderation tasks.

Findings

Outperforms state-of-the-art reasoning and moderation models.

Achieves 24.9% higher F1 score on ambiguous moderation benchmarks.

Effectively mitigates harmful decision shortcuts in LLMs.

Abstract

Current large language models (LLMs), even those explicitly trained for reasoning, often struggle with ambiguous content moderation cases due to misleading "decision shortcuts" embedded in context. Inspired by cognitive psychology insights into expert moderation, we introduce \caro (Chain-of-Analogy Reasoning Optimization), a novel two-stage training framework to induce robust analogical reasoning in LLMs. First, \caro bootstraps analogical reasoning chains via retrieval-augmented generation (RAG) on moderation data and performs supervised fine-tuning (SFT). Second, we propose a customized direct preference optimization (DPO) approach to reinforce analogical reasoning behaviors explicitly. Unlike static retrieval methods, \caro dynamically generates tailored analogical references during inference, effectively mitigating harmful decision shortcuts. Extensive experiments demonstrate that \caro substantially outperforms state-of-the-art reasoning models (DeepSeek R1, QwQ), specialized moderation models (LLaMA Guard), and advanced fine-tuning and retrieval-augmented methods, achieving an average F1 score improvement of 24.9\% on challenging ambiguous moderation benchmarks.