Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic

TL;DR

Shikra is a simple, natural language-based multimodal LLM that can perform referential dialogue involving spatial regions, filling a key gap in current vision-language models and enabling diverse spatial tasks.

Contribution

This paper introduces Shikra, a straightforward multimodal LLM capable of handling spatial referential dialogue using natural language inputs and outputs, without extra modules or vocabularies.

Findings

Shikra achieves promising performance on spatial and vision-language tasks.

It can handle location-based tasks like REC and PointQA.

Enables applications like object coordinate reasoning and region similarity comparison.

Abstract

In human conversations, individuals can indicate relevant regions within a scene while addressing others. In turn, the other person can then respond by referring to specific regions if necessary. This natural referential ability in dialogue remains absent in current Multimodal Large Language Models (MLLMs). To fill this gap, this paper proposes an MLLM called Shikra, which can handle spatial coordinate inputs and outputs in natural language. Its architecture consists of a vision encoder, an alignment layer, and a LLM. It is designed to be straightforward and simple, without the need for extra vocabularies, position encoder, pre-/post-detection modules, or external plug-in models. All inputs and outputs are in natural language form. Referential dialogue is a superset of various vision-language (VL) tasks. Shikra can naturally handle location-related tasks like REC and PointQA, as well as conventional VL tasks such as Image Captioning and VQA. Experimental results showcase Shikra's promising performance. Furthermore, it enables numerous exciting applications, like providing mentioned objects' coordinates in chains of thoughts and comparing user-pointed regions similarities. Our code, model and dataset are accessed at https://github.com/shikras/shikra.