DSFlash: Comprehensive Panoptic Scene Graph Generation in Realtime

TL;DR

DSFlash is a fast, resource-efficient panoptic scene graph generation model capable of processing video streams at 56 fps, providing comprehensive scene understanding suitable for real-time applications on limited hardware.

Contribution

The paper introduces DSFlash, a low-latency, resource-efficient model that generates comprehensive scene graphs in real-time, addressing deployment challenges on edge devices.

Findings

Processes video at 56 fps on RTX 3090

Requires less than 24 hours to train on GTX 1080

Generates richer, comprehensive scene graphs

Abstract

Scene Graph Generation (SGG) aims to extract a detailed graph structure from an image, a representation that holds significant promise as a robust intermediate step for complex downstream tasks like reasoning for embodied agents. However, practical deployment in real-world applications - especially on resource constrained edge devices - requires speed and resource efficiency, challenges that have received limited attention in existing research. To bridge this gap, we introduce DSFlash, a low-latency model for panoptic scene graph generation designed to overcome these limitations. DSFlash can process a video stream at 56 frames per second on a standard RTX 3090 GPU, without compromising performance against existing state-of-the-art methods. Crucially, unlike prior approaches that often restrict themselves to salient relationships, DSFlash computes comprehensive scene graphs, offering richer contextual information while maintaining its superior latency. Furthermore, DSFlash is light on resources, requiring less than 24 hours to train on a single, nine-year-old GTX 1080 GPU. This accessibility makes DSFlash particularly well-suited for researchers and practitioners operating with limited computational resources, empowering them to adapt and fine-tune SGG models for specialized applications.