Don't Command, Cultivate: An Exploratory Study of System-2 Alignment

TL;DR

This study explores how System-2 thinking patterns influence AI model safety, showing that deliberate reasoning can improve safety but still faces vulnerabilities, and proposes methods to enhance alignment.

Contribution

It investigates the impact of System-2 reasoning on model safety and introduces prompt engineering and supervision techniques to improve safety alignment.

Findings

o1 models show improved safety but remain vulnerable to jailbreak attacks

Prompt engineering helps models scrutinize user requests more carefully

Proposed process supervision could further enhance safety alignment

Abstract

The o1 system card identifies the o1 models as the most robust within OpenAI, with their defining characteristic being the progression from rapid, intuitive thinking to slower, more deliberate reasoning. This observation motivated us to investigate the influence of System-2 thinking patterns on model safety. In our preliminary research, we conducted safety evaluations of the o1 model, including complex jailbreak attack scenarios using adversarial natural language prompts and mathematical encoding prompts. Our findings indicate that the o1 model demonstrates relatively improved safety performance; however, it still exhibits vulnerabilities, particularly against jailbreak attacks employing mathematical encoding. Through detailed case analysis, we identified specific patterns in the o1 model's responses. We also explored the alignment of System-2 safety in open-source models using prompt engineering and supervised fine-tuning techniques. Experimental results show that some simple methods to encourage the model to carefully scrutinize user requests are beneficial for model safety. Additionally, we proposed a implementation plan for process supervision to enhance safety alignment. The implementation details and experimental results will be provided in future versions.