An Exploration of Self-Supervised Mutual Information Alignment for Multi-Task Settings

TL;DR

This paper introduces SAMI, a self-supervised mutual information alignment method for multi-task language model tuning, demonstrating competitive performance against DPO and insights into its scaling behavior for mathematical tasks.

Contribution

The paper presents SAMI, a novel self-supervised alignment approach using mutual information, and evaluates its effectiveness across multi-task benchmarks and mathematical accuracy scenarios.

Findings

SAMI achieves a 57% win rate against DPO on MT-Bench.

SAMI increases zero-shot math accuracy by 1.1%.

Scaling SAMI with multiple attempts improves performance.

Abstract

There is a growing need for pluralistic alignment methods that can steer language models towards individual attributes and preferences. One such method, Self-Supervised Alignment with Mutual Information (SAMI), uses conditional mutual information to encourage the connection between behavioral preferences and model responses. We conduct two experiments exploring SAMI in multi-task settings. First, we compare SAMI to Direct Preference Optimization (DPO) on a multi-task benchmark (MT-Bench), using a stronger model to generate training data for a weaker one across diverse categories (humanities, STEM, extraction, coding, math, reasoning, and roleplay). Our results indicate that one iteration of SAMI has a 57% win rate against DPO, with significant variation in performance between task categories. Second, we examine SAMI's impact on mathematical accuracy (GSM-8K) relative to supervised fine-tuning (SFT). While SAMI increases zero-shot performance by 1.1%, SFT is more effective with a 3.2% boost. However, SAMI shows interesting scaling trends. When given 10 attempts, SAMI improves accuracy by 3.9%, while SFT achieves a 10.1% increase. Combining SAMI with SFT yields an additional improvement of 1.3% in multi-attempt settings, though single-attempt accuracy remains unchanged.