Token-level Accept or Reject: A Micro Alignment Approach for Large Language Models

TL;DR

This paper introduces MARA, a token-level alignment method for LLMs that improves alignment accuracy and efficiency without requiring model fine-tuning, by classifying tokens as accepted or rejected.

Contribution

MARA is a novel, model-independent approach that simplifies alignment by decomposing sentence preferences into token-level decisions using a lightweight classifier.

Findings

MARA outperforms existing methods in alignment accuracy across multiple LLMs.

MARA reduces computational costs compared to traditional fine-tuning methods.

MARA is effective across diverse datasets and models.

Abstract

With the rapid development of Large Language Models (LLMs), aligning these models with human preferences and values is critical to ensuring ethical and safe applications. However, existing alignment techniques such as RLHF or DPO often require direct fine-tuning on LLMs with billions of parameters, resulting in substantial computational costs and inefficiencies. To address this, we propose Micro token-level Accept-Reject Aligning (MARA) approach designed to operate independently of the language models. MARA simplifies the alignment process by decomposing sentence-level preference learning into token-level binary classification, where a compact three-layer fully-connected network determines whether candidate tokens are "Accepted" or "Rejected" as part of the response. Extensive experiments across seven different LLMs and three open-source datasets show that MARA achieves significant improvements in alignment performance while reducing computational costs. The source code and implementation details are publicly available at https://github.com/IAAR-Shanghai/MARA, and the trained models are released at https://huggingface.co/IAAR-Shanghai/MARA_AGENTS.