SemCovNet: Towards Fair and Semantic Coverage-Aware Learning for Underrepresented Visual Concepts

TL;DR

This paper introduces SemCovNet, a novel model designed to address semantic coverage imbalance in vision models, improving fairness and interpretability by explicitly learning and correcting semantic disparities.

Contribution

SemCovNet is the first model to explicitly learn and mitigate semantic coverage imbalance, incorporating a semantic descriptor map, attention modulation, and alignment loss for fairer visual concept learning.

Findings

SemCovNet reduces Coverage Disparity Index across datasets.

The model improves fairness and interpretability in visual concept recognition.

Semantic coverage imbalance is a measurable and correctable bias.

Abstract

Modern vision models increasingly rely on rich semantic representations that extend beyond class labels to include descriptive concepts and contextual attributes. However, existing datasets exhibit Semantic Coverage Imbalance (SCI), a previously overlooked bias arising from the long-tailed semantic representations. Unlike class imbalance, SCI occurs at the semantic level, affecting how models learn and reason about rare yet meaningful semantics. To mitigate SCI, we propose Semantic Coverage-Aware Network (SemCovNet), a novel model that explicitly learns to correct semantic coverage disparities. SemCovNet integrates a Semantic Descriptor Map (SDM) for learning semantic representations, a Descriptor Attention Modulation (DAM) module that dynamically weights visual and concept features, and a Descriptor-Visual Alignment (DVA) loss that aligns visual features with descriptor semantics. We quantify semantic fairness using a Coverage Disparity Index (CDI), which measures the alignment between coverage and error. Extensive experiments across multiple datasets demonstrate that SemCovNet enhances model reliability and substantially reduces CDI, achieving fairer and more equitable performance. This work establishes SCI as a measurable and correctable bias, providing a foundation for advancing semantic fairness and interpretable vision learning.