From Instructions to Intrinsic Human Values -- A Survey of Alignment Goals for Big Models

TL;DR

This survey reviews various alignment goals for large language models, highlighting the evolution from basic abilities to intrinsic human values, and discusses challenges and resources for future research.

Contribution

It provides a comprehensive analysis of alignment goal evolution, emphasizing the importance of intrinsic human values for better alignment of big models.

Findings

Alignment goals evolve from fundamental abilities to value orientation.

Intrinsic human values are identified as a promising alignment goal.

Resources for future research on value alignment are compiled.

Abstract

Big models, exemplified by Large Language Models (LLMs), are models typically pre-trained on massive data and comprised of enormous parameters, which not only obtain significantly improved performance across diverse tasks but also present emergent capabilities absent in smaller models. However, the growing intertwining of big models with everyday human lives poses potential risks and might cause serious social harm. Therefore, many efforts have been made to align LLMs with humans to make them better follow user instructions and satisfy human preferences. Nevertheless, `what to align with' has not been fully discussed, and inappropriate alignment goals might even backfire. In this paper, we conduct a comprehensive survey of different alignment goals in existing work and trace their evolution paths to help identify the most essential goal. Particularly, we investigate related works from two perspectives: the definition of alignment goals and alignment evaluation. Our analysis encompasses three distinct levels of alignment goals and reveals a goal transformation from fundamental abilities to value orientation, indicating the potential of intrinsic human values as the alignment goal for enhanced LLMs. Based on such results, we further discuss the challenges of achieving such intrinsic value alignment and provide a collection of available resources for future research on the alignment of big models.