Can Safety Emerge from Weak Supervision? A Systematic Analysis of Small Language Models

TL;DR

This paper presents Self-MOA, an automated framework that uses weak supervision to align small language models for safety and helpfulness, reducing reliance on costly human annotations.

Contribution

Introduction of Self-MOA, a fully automated, weak supervision-based method for aligning small language models with improved safety and maintained helpfulness.

Findings

12.41% safety improvement across benchmarks

Uses 11 times less data than human supervision

Effective in resource-constrained settings

Abstract

Safety alignment is critical for deploying large language models (LLMs) in real-world applications, yet most existing approaches rely on large human-annotated datasets and static red-teaming benchmarks that are costly, difficult to scale, and slow to adapt to evolving model behaviors. Moreover, overly conservative safety mechanisms can reduce model usefulness by rejecting sensitive but legitimate queries. We introduce Self-MOA (Self Multi-Objective Alignment), a fully automated framework for aligning small language models using weak supervision from automated evaluator models. Self-MOA operates as a closed loop that dynamically generates model-specific red team prompts, constructs preference data from model-generated responses, and aligns models via multi-objective preference optimization to jointly optimize for safety and helpfulness. Across multiple small language models and safety benchmarks, Self-MOA achieves a 12.41\% improvement in safety while preserving helpfulness, using as little as 11 times less training data than human-supervised alignment baselines. These results demonstrate that adaptive, automated alignment can reduce the dependence on static, human-curated safety pipelines in resource-constrained settings.