TalkMosaic: Interactive PhotoMosaic with Multi-modal LLM Q&A Interactions

TL;DR

TalkMosaic introduces an interactive photomosaic system combining multimodal LLMs and image interaction to enhance environmental awareness and provide detailed car-related information efficiently.

Contribution

The paper presents a novel multimodal GPT model, TalkMosaic, integrating car images and knowledge, with optimized inference techniques for interactive photo-mosaic applications.

Findings

Effective interactive photo-mosaic with click-and-display functionality.

Enhanced inference speed using sparse attention and quantization techniques.

Prototype demonstrates feasibility and practical utility.

Abstract

We use images of cars of a wide range of varieties to compose an image of an animal such as a bird or a lion for the theme of environmental protection to maximize the information about cars in a single composed image and to raise the awareness about environmental challenges. We present a novel way of image interaction with an artistically-composed photomosaic image, in which a simple operation of "click and display" is used to demonstrate the interactive switch between a tile image in a photomosaic image and the corresponding original car image, which will be automatically saved on the Desktop. We build a multimodal custom GPT named TalkMosaic by incorporating car images information and the related knowledge to ChatGPT. By uploading the original car image to TalkMosaic, we can ask questions about the given car image and get the corresponding answers efficiently and effectively such as where to buy the tire in the car image that satisfies high environmental standards. We give an in-depth analysis on how to speed up the inference of multimodal LLM using sparse attention and quantization techniques with presented probabilistic FlashAttention (PrFlashAttention) and Staircase Adaptive Quantization (SAQ) methods. The implemented prototype demonstrates the feasibility and effectiveness of the presented approach.