Lumos : Empowering Multimodal LLMs with Scene Text Recognition

TL;DR

Lumos is a novel multimodal question-answering system that integrates scene text recognition to enhance text understanding from images, addressing challenges in accuracy, latency, and inference efficiency.

Contribution

This work introduces Lumos, the first end-to-end system combining scene text recognition with multimodal large language models for improved image-based text understanding.

Findings

High accuracy in scene text recognition from first-person images

Reduced latency in multimodal question-answering tasks

Effective integration of text extraction with language modeling

Abstract

We introduce Lumos, the first end-to-end multimodal question-answering system with text understanding capabilities. At the core of Lumos is a Scene Text Recognition (STR) component that extracts text from first person point-of-view images, the output of which is used to augment input to a Multimodal Large Language Model (MM-LLM). While building Lumos, we encountered numerous challenges related to STR quality, overall latency, and model inference. In this paper, we delve into those challenges, and discuss the system architecture, design choices, and modeling techniques employed to overcome these obstacles. We also provide a comprehensive evaluation for each component, showcasing high quality and efficiency.