LifeIR at the NTCIR-18 Lifelog-6 Task

TL;DR

This paper presents a multi-stage pipeline for retrieving relevant images from large-scale lifelogs using textual queries, incorporating filtering, query rewriting, event-based extension, and reranking with multimodal large language models, demonstrating improved retrieval effectiveness.

Contribution

It introduces a novel multi-stage pipeline for lifelog image retrieval that combines filtering, query rewriting, event-based candidate extension, and reranking with advanced multimodal models.

Findings

Each stage improves retrieval accuracy

The pipeline outperforms baseline methods

Multimodal reranking enhances relevance judgment

Abstract

In recent years, sharing lifelogs recorded through wearable devices such as sports watches and GoPros, has gained significant popularity. Lifelogs involve various types of information, including images, videos, and GPS data, revealing users' lifestyles, dietary patterns, and physical activities. The Lifelog Semantic Access Task(LSAT) in the NTCIR-18 Lifelog-6 Challenge focuses on retrieving relevant images from a large scale of users' lifelogs based on textual queries describing an action or event. It serves users' need to find images about a scenario in the historical moments of their lifelogs. We propose a multi-stage pipeline for this task of searching images with texts, addressing various challenges in lifelog retrieval. Our pipeline includes: filtering blurred images, rewriting queries to make intents clearer, extending the candidate set based on events to include images with temporal connections, and reranking results using a multimodal large language model(MLLM) with stronger relevance judgment capabilities. The evaluation results of our submissions have shown the effectiveness of each stage and the entire pipeline.