Extracting and Understanding the Superficial Knowledge in Alignment

TL;DR

This paper investigates whether alignment in large language models is mainly superficial, introducing a method to extract and analyze superficial knowledge, revealing its significant but incomplete role in model alignment.

Contribution

The paper formalizes the concept of superficial knowledge, proposes a method to extract it, and demonstrates its transferability and recoverability, highlighting its role in model alignment.

Findings

Superficial knowledge accounts for a large part of alignment in safety tasks.

Deep reasoning tasks rely more on underlying causal knowledge.

Extracted superficial knowledge can be transferred and recovered across models.

Abstract

Alignment of large language models (LLMs) with human values and preferences, often achieved through fine-tuning based on human feedback, is essential for ensuring safe and responsible AI behaviors. However, the process typically requires substantial data and computation resources. Recent studies have revealed that alignment might be attainable at lower costs through simpler methods, such as in-context learning. This leads to the question: Is alignment predominantly superficial? In this paper, we delve into this question and provide a quantitative analysis. We formalize the concept of superficial knowledge, defining it as knowledge that can be acquired through easily token restyling, without affecting the model's ability to capture underlying causal relationships between tokens. We propose a method to extract and isolate superficial knowledge from aligned models, focusing on the shallow modifications to the final token selection process. By comparing models augmented only with superficial knowledge to fully aligned models, we quantify the superficial portion of alignment. Our findings reveal that while superficial knowledge constitutes a significant portion of alignment, particularly in safety and detoxification tasks, it is not the whole story. Tasks requiring reasoning and contextual understanding still rely on deeper knowledge. Additionally, we demonstrate two practical advantages of isolated superficial knowledge: (1) it can be transferred between models, enabling efficient offsite alignment of larger models using extracted superficial knowledge from smaller models, and (2) it is recoverable, allowing for the restoration of alignment in compromised models without sacrificing performance.