Automatic Photo to Ideophone Manga Matching

TL;DR

This paper introduces a novel system that automatically recommends and positions ideophone annotations on photos, enhancing user engagement and sharing, by combining visual object detection with semantic embedding.

Contribution

It presents a new method for automatic ideophone annotation on photos, integrating visual detection with semantic mapping to improve photo annotation expressiveness.

Findings

Annotations increase user enjoyment and sharing.

System outperforms traditional object-based annotation.

Preferred in Japanese photo sharing contexts.

Abstract

Photo applications offer tools for annotation via text and stickers. Ideophones, mimetic and onomatopoeic words, which are common in graphic novels, have yet to be explored for photo annotation use. We present a method for automatic ideophone recommendation and positioning of the text on photos. These annotations are accomplished by obtaining a list of ideophones with English definitions and applying a suite of visual object detectors to the image. Next, a semantic embedding maps the visual objects to the possible relevant ideophones. Our system stands in contrast to traditional computer vision-based annotation systems, which stop at recommending object and scene-level annotation, by providing annotations that are communicative, fun, and engaging. We test these annotations in Japanese and find they carry a strong preference and increase enjoyment and sharing likelihood when compared to unannotated and object-based annotated photos.