HotelMatch-LLM: Joint Multi-Task Training of Small and Large Language Models for Efficient Multimodal Hotel Retrieval

TL;DR

HotelMatch-LLM is a multimodal dense retrieval model tailored for the travel domain, enabling natural language hotel searches with improved accuracy and efficiency through multi-task training and a dual-model architecture.

Contribution

The paper introduces HotelMatch-LLM, a novel multi-task optimized multimodal retrieval framework combining small and large language models for efficient hotel search.

Findings

Outperforms state-of-the-art models like VISTA and MARVEL in hotel retrieval accuracy.

Achieves a 0.681 retrieval score on main query type, surpassing baseline MARVEL's 0.603.

Demonstrates scalability and generalizability across different LLM architectures.

Abstract

We present HotelMatch-LLM, a multimodal dense retrieval model for the travel domain that enables natural language property search, addressing the limitations of traditional travel search engines which require users to start with a destination and editing search parameters. HotelMatch-LLM features three key innovations: (1) Domain-specific multi-task optimization with three novel retrieval, visual, and language modeling objectives; (2) Asymmetrical dense retrieval architecture combining a small language model (SLM) for efficient online query processing and a large language model (LLM) for embedding hotel data; and (3) Extensive image processing to handle all property image galleries. Experiments on four diverse test sets show HotelMatch-LLM significantly outperforms state-of-the-art models, including VISTA and MARVEL. Specifically, on the test set -- main query type -- we achieve 0.681 for HotelMatch-LLM compared to 0.603 for the most effective baseline, MARVEL. Our analysis highlights the impact of our multi-task optimization, the generalizability of HotelMatch-LLM across LLM architectures, and its scalability for processing large image galleries.