CAMO: A Class-Aware Minority-Optimized Ensemble for Robust Language Model Evaluation on Imbalanced Data

TL;DR

CAMO is a novel ensemble method designed to improve minority class performance in imbalanced language model evaluation, demonstrating superior results across multiple benchmarks and models.

Contribution

Introduces CAMO, a class-aware, minority-optimized ensemble technique that enhances minority class detection in imbalanced datasets for language models.

Findings

CAMO achieves the highest macro F1-score on unbalanced benchmarks.

It consistently outperforms seven ensemble algorithms across various models.

CAMO's benefits are amplified when combined with model adaptation.

Abstract

Real-world categorization is severely hampered by class imbalance because traditional ensembles favor majority classes, which lowers minority performance and overall F1-score. We provide a unique ensemble technique for imbalanced problems called CAMO (Class-Aware Minority-Optimized).Through a hierarchical procedure that incorporates vote distributions, confidence calibration, and inter model uncertainty, CAMO dynamically boosts underrepresented classes while preserving and amplifying minority forecasts. We verify CAMO on two highly unbalanced, domain-specific benchmarks: the DIAR-AI/Emotion dataset and the ternary BEA 2025 dataset. We benchmark against seven proven ensemble algorithms using eight different language models (three LLMs and five SLMs) under zero-shot and fine-tuned settings .With refined models, CAMO consistently earns the greatest strict macro F1-score, setting a new benchmark. Its benefit works in concert with model adaptation, showing that the best ensemble choice depends on model properties .This proves that CAMO is a reliable, domain-neutral framework for unbalanced categorization.