Goal-Oriented Semantic Communication for Wireless Visual Question Answering

TL;DR

This paper introduces a goal-oriented semantic communication framework for wireless visual question answering that enhances accuracy and reduces latency by transmitting only the most relevant semantic information, addressing bandwidth and noise challenges.

Contribution

It proposes a novel semantic extraction and ranking method, incorporating scene graphs, to improve VQA performance over wireless channels with limited resources.

Findings

Improves answering accuracy by up to 59% under Rayleigh channels.

Reduces total latency by up to 65%.

Enhances robustness against channel noise and multipath effects.

Abstract

The rapid progress of artificial intelligence (AI) and computer vision (CV) has facilitated the development of computation-intensive applications like Visual Question Answering (VQA), which integrates visual perception and natural language processing to generate answers. To overcome the limitations of traditional VQA constrained by local computation resources, edge computing has been incorporated to provide extra computation capability at the edge side. Meanwhile, this brings new communication challenges between the local and edge, including limited bandwidth, channel noise, and multipath effects, which degrade VQA performance and user quality of experience (QoE), particularly during the transmission of large high-resolution images. To overcome these bottlenecks, we propose a goal-oriented semantic communication (GSC) framework that focuses on effectively extracting and transmitting semantic information most relevant to the VQA goals, improving the answering accuracy and enhancing the effectiveness and efficiency. The objective is to maximize the answering accuracy, and we propose a bounding box (BBox)-based image semantic extraction and ranking approach to prioritize the semantic information based on the goal of questions. We then extend it by incorporating a scene graphs (SG)-based approach to handle questions with complex relationships. Experimental results demonstrate that our GSC framework improves answering accuracy by up to 49% under AWGN channels and 59% under Rayleigh channels while reducing total latency by up to 65% compared to traditional bit-oriented transmission.