Enhancing Vision Models for Text-Heavy Content Understanding and Interaction

TL;DR

This paper improves vision models' ability to understand complex, text-heavy images like textbooks and research papers by fine-tuning with instructional data and integrating multimodal components, achieving high accuracy.

Contribution

It introduces a novel approach combining dataset preprocessing, instructional fine-tuning, and multimodal integration to enhance vision models for complex textual visual content understanding.

Findings

Achieved 96.71% accuracy in understanding complex visual textual data.

Developed a visual chat application integrating CLIP and text embedding models.

Enhanced multimodal AI capabilities for interpreting interconnected visual and textual information.

Abstract

Interacting and understanding with text heavy visual content with multiple images is a major challenge for traditional vision models. This paper is on enhancing vision models' capability to comprehend or understand and learn from images containing a huge amount of textual information from the likes of textbooks and research papers which contain multiple images like graphs, etc and tables in them with different types of axes and scales. The approach involves dataset preprocessing, fine tuning which is by using instructional oriented data and evaluation. We also built a visual chat application integrating CLIP for image encoding and a model from the Massive Text Embedding Benchmark which is developed to consider both textual and visual inputs. An accuracy of 96.71% was obtained. The aim of the project is to increase and also enhance the advance vision models' capabilities in understanding complex visual textual data interconnected data, contributing to multimodal AI.