H3Fusion: Helpful, Harmless, Honest Fusion of Aligned LLMs

TL;DR

H3Fusion introduces a mixture-of-experts fusion mechanism for aligned LLMs, improving helpfulness, harmlessness, and honesty by balancing alignment properties through a controllable subspace drift and dual objectives.

Contribution

The paper presents a novel MoE-based fusion method that models alignment as a controllable drift, enhancing the integration of helpful, harmless, and honest properties in LLMs.

Findings

Outperforms individually aligned models by 11.37%.

Achieves 13.77% stronger robustness than state-of-the-art ensemble methods.

Provides 6.18% improvement over model-merging approaches.

Abstract

The alignment of pre-trained LLMs continues to draw significant attention from both industry and academia, aiming to ensure responses that are helpful, harmless, and honest. However, identifying a point in the model's representation subspace that simultaneously satisfies all these properties remains challenging. H3Fusion addresses this challenge by introducing a mixture-of-experts (MoE)-based fusion mechanism that models alignment as a controllable drift within the subspace, guided by a drift-regularization loss to balance competing alignment dimensions. Furthermore, we formulate the alignment by finding a dual objective of harnessing the distance of generated embeddings and alignment embeddings, and introduce a gating loss by canalizing the activations on the contributing experts. Extensive evaluations of three benchmark datasets show that H3Fusion is more helpful, less harmful, and more honest in three aspects: it outperforms each individually aligned model by 11.37%, and provides stronger robustness compared to the state-of-the-art LLM ensemble approaches by 13.77% and model-merging approaches by 6.18%. Code is available at https://github.com/git-disl/h3fusion.