MOSAIC: Multi-Objective Slice-Aware Iterative Curation for Alignment

TL;DR

MOSAIC is a multi-objective framework for optimizing data curation in model fine-tuning, balancing safety, over-refusal, and instruction following within a fixed budget, leading to improved model performance and generalization.

Contribution

The paper introduces MOSAIC, a novel slice-aware iterative curation method that effectively balances multiple objectives during model fine-tuning using structured failure profiles.

Findings

MOSAIC improves safety and instruction following metrics under a fixed budget.

Structured failure diagnosis enhances model robustness and generalization.

The approach outperforms static baseline methods on multiple evaluation benchmarks.

Abstract

We study how to allocate a fixed supervised fine-tuning budget when three objectives must be balanced at once: multi-turn safety alignment, low over-refusal on benign boundary queries, and instruction following under verifiable constraints. We propose MOSAIC (Multi-Objective Slice-Aware Iterative Curation for Alignment), a multi-objective framework for closed-loop data mixture search built on a unified L1-L3 evaluation interface. MOSAIC turns slice-level failure profiles into executable data actions, including dataset-level mixture ratios, bucket-level weights, and focus criteria. Under a fixed 1M-token budget and five rounds of independent fine-tuning from the same base model, MOSAIC improves internal XGuard from 2.76 to 4.67 while keeping OrBench at 4.41 and IFEval at 3.65. The final Pareto solution also generalizes better than a random static LoRA baseline on independent attack, over-refusal, and capability tests, suggesting that structured failure diagnosis can serve as a practical control signal for budgeted data construction. Code is available at https://github.com/douyipu/mosaic.