MammothModa: Multi-Modal Large Language Model
Qi She, Junwen Pan, Xin Wan, Rui Zhang, Dawei Lu, Kai, Huang

TL;DR
MammothModa is a multi-modal large language model that integrates advanced visual capabilities, extends context windows for high-resolution images, and uses high-quality bilingual datasets to achieve state-of-the-art performance in visual language tasks.
Contribution
The paper introduces MammothModa, a novel MLLM that combines visual attention, extended context handling, and curated datasets to outperform existing models.
Findings
Outperforms LLaVA-series on main benchmarks
Effectively handles high-resolution images with extended context
Reduces visual hallucinations with curated datasets
Abstract
In this report, we introduce MammothModa, yet another multi-modal large language model (MLLM) designed to achieve state-of-the-art performance starting from an elementary baseline. We focus on three key design insights: (i) Integrating Visual Capabilities while Maintaining Complex Language Understanding: In addition to the vision encoder, we incorporated the Visual Attention Experts into the LLM to enhance its visual capabilities. (ii) Extending Context Window for High-Resolution and Long-Duration Visual Feature: We explore the Visual Merger Module to effectively reduce the token number of high-resolution images and incorporated frame position ids to avoid position interpolation. (iii) High-Quality Bilingual Datasets: We meticulously curated and filtered a high-quality bilingual multimodal dataset to reduce visual hallucinations. With above recipe we build MammothModa that consistently…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsAI in cancer detection
MethodsSoftmax · Attention Is All You Need · Focus
