Mitigating Sycophancy in Decoder-Only Transformer Architectures: Synthetic Data Intervention

TL;DR

This paper introduces a synthetic data intervention method to reduce sycophancy in decoder-only transformers, demonstrating improved accuracy and decreased sycophantic behavior through experiments with GPT-4.

Contribution

The study presents a novel synthetic data intervention approach specifically designed to mitigate sycophancy in large language models, filling a research gap in addressing model bias.

Findings

Reduced sycophancy rate in the model

Improved accuracy on true/false questions

Effective in diversifying model responses

Abstract

To address the sycophancy problem caused by reinforcement learning from human feedback in large language models, this research applies synthetic data intervention technology to the decoder-only transformer architecture. Based on the research gaps in the existing literature, the researcher designed an experimental process to reduce the tendency of models to cater by generating diversified data, and used GPT4o as an experimental tool for verification. The experiment used 100 true and false questions, and compared the performance of the model trained with synthetic data intervention and the original untrained model on multiple indicators. The results show that the SDI training model supports the technology in terms of accuracy rate and sycophancy rate and has significant effectiveness in reducing sycophancy phenomena.