Granulon: Awakening Pixel-Level Visual Encoders with Adaptive Multi-Granularity Semantics for MLLM

TL;DR

Granulon is a novel multimodal large language model that dynamically adjusts visual granularity for improved fine-grained understanding and reasoning, outperforming existing models in accuracy and hallucination reduction.

Contribution

It introduces a text-conditioned granularity controller and adaptive token aggregation to enable unified multi-granularity visual reasoning within a single model.

Findings

Improves accuracy by approximately 30%.

Reduces hallucination by about 20%.

Outperforms all visual encoders under identical settings.

Abstract

Recent advances in multimodal large language models largely rely on CLIP-based visual encoders, which emphasize global semantic alignment but struggle with fine-grained visual understanding. In contrast, DINOv3 provides strong pixel-level perception yet lacks coarse-grained semantic abstraction, leading to limited multi-granularity reasoning. To address this gap, we propose Granulon, a novel DINOv3-based MLLM with adaptive granularity augmentation. Granulon introduces a text-conditioned granularity Controller that dynamically adjusts the visual abstraction level according to the semantic scope of the textual input, and an Adaptive Token Aggregation module that performs granularity-guided pooling and relation-aware clustering to produce compact, semantically rich visual tokens. This design enables unified "pixel-to-fine-to-coarse" reasoning within a single forward pass. Extensive and interpretable experiments demonstrate that Granulon improves accuracy by ~30% and reduces hallucination by ~20%, outperforming all visual encoders under identical settings.