Emergent communication for AR

TL;DR

This paper introduces an emergent semantic communication framework for mobile augmented reality that enables efficient, low-data communication between agents, improving experience quality amidst latency and noise challenges.

Contribution

It presents a novel emergent communication protocol learned via a modified Lewis signaling game, tailored for MAR to reduce data size and handle noisy channels.

Findings

Better generalization on unseen objects compared to traditional recognition methods.

Enhanced communication efficiency with small-size messages.

Effective handling of channel noise in communication protocols.

Abstract

Mobile augmented reality (MAR) is widely acknowledged as one of the ubiquitous interfaces to the digital twin and Metaverse, demanding unparalleled levels of latency, computational power, and energy efficiency. The existing solutions for realizing MAR combine multiple technologies like edge, cloud computing, and fifth-generation (5G) networks. However, the inherent communication latency of visual data imposes apparent limitations on the quality of experience (QoE). To address the challenge, we propose an emergent semantic communication framework to learn the communication protocols in MAR. Specifically, we train two agents through a modified Lewis signaling game to emerge a discrete communication protocol spontaneously. Based on this protocol, two agents can communicate about the abstract idea of visual data through messages with extremely small data sizes in a noisy channel, which leads to message errors. To better simulate real-world scenarios, we incorporate channel uncertainty into our training process. Experiments have shown that the proposed scheme has better generalization on unseen objects than traditional object recognition used in MAR and can effectively enhance communication efficiency through the utilization of small-size messages.