Adapting Visual Question Answering Models for Enhancing Multimodal Community Q&A Platforms

TL;DR

This paper extends visual question answering models to multimodal community Q&A platforms, improving question categorization and expert retrieval by leveraging image data, and introduces novel attention augmentations for better performance.

Contribution

It is the first to adapt VQA models for multimodal CQA tasks, addressing the challenge of integrating images into community question answering systems.

Findings

Model outperforms text-only baselines in classification and retrieval.

Augmented attention methods improve grounding of visual information.

First application of VQA models to real-world multimodal CQA data.

Abstract

Question categorization and expert retrieval methods have been crucial for information organization and accessibility in community question & answering (CQA) platforms. Research in this area, however, has dealt with only the text modality. With the increasing multimodal nature of web content, we focus on extending these methods for CQA questions accompanied by images. Specifically, we leverage the success of representation learning for text and images in the visual question answering (VQA) domain, and adapt the underlying concept and architecture for automated category classification and expert retrieval on image-based questions posted on Yahoo! Chiebukuro, the Japanese counterpart of Yahoo! Answers. To the best of our knowledge, this is the first work to tackle the multimodality challenge in CQA, and to adapt VQA models for tasks on a more ecologically valid source of visual questions. Our analysis of the differences between visual QA and community QA data drives our proposal of novel augmentations of an attention method tailored for CQA, and use of auxiliary tasks for learning better grounding features. Our final model markedly outperforms the text-only and VQA model baselines for both tasks of classification and expert retrieval on real-world multimodal CQA data.