MammothModa: Multi-Modal Large Language Model

TL;DR

MammothModa is a multi-modal large language model that integrates advanced visual capabilities, extends context windows for high-resolution images, and uses high-quality bilingual datasets to achieve state-of-the-art performance in visual language tasks.

Contribution

The paper introduces MammothModa, a novel MLLM that combines visual attention, extended context handling, and curated datasets to outperform existing models.

Findings

Outperforms LLaVA-series on main benchmarks

Effectively handles high-resolution images with extended context

Reduces visual hallucinations with curated datasets

Abstract

In this report, we introduce MammothModa, yet another multi-modal large language model (MLLM) designed to achieve state-of-the-art performance starting from an elementary baseline. We focus on three key design insights: (i) Integrating Visual Capabilities while Maintaining Complex Language Understanding: In addition to the vision encoder, we incorporated the Visual Attention Experts into the LLM to enhance its visual capabilities. (ii) Extending Context Window for High-Resolution and Long-Duration Visual Feature: We explore the Visual Merger Module to effectively reduce the token number of high-resolution images and incorporated frame position ids to avoid position interpolation. (iii) High-Quality Bilingual Datasets: We meticulously curated and filtered a high-quality bilingual multimodal dataset to reduce visual hallucinations. With above recipe we build MammothModa that consistently outperforms the state-of-the-art models, e.g., LLaVA-series, across main real-world visual language benchmarks without bells and whistles.