An Information Retrieval Approach to Short Text Conversation

TL;DR

This paper explores using information retrieval techniques to improve short text conversation systems by leveraging social media data, demonstrating that IR-based models can produce more intelligent responses.

Contribution

It formalizes short text conversation as a search problem and evaluates the effectiveness of IR methods in generating responses from social media data.

Findings

IR approach enhances response quality in short text conversations

Large social media datasets improve retrieval-based response relevance

IR methods show limitations but can be combined with data for better performance

Abstract

Human computer conversation is regarded as one of the most difficult problems in artificial intelligence. In this paper, we address one of its key sub-problems, referred to as short text conversation, in which given a message from human, the computer returns a reasonable response to the message. We leverage the vast amount of short conversation data available on social media to study the issue. We propose formalizing short text conversation as a search problem at the first step, and employing state-of-the-art information retrieval (IR) techniques to carry out the task. We investigate the significance as well as the limitation of the IR approach. Our experiments demonstrate that the retrieval-based model can make the system behave rather "intelligently", when combined with a huge repository of conversation data from social media.