Deep Perm-Set Net: Learn to predict sets with unknown permutation and cardinality using deep neural networks

TL;DR

This paper introduces a deep neural network approach for predicting sets with unknown size and order, effectively handling permutation invariance and demonstrating superior performance in object detection and CAPTCHA tasks.

Contribution

It proposes a novel formulation that models permutation as an unobservable variable and estimates its distribution during training, advancing set prediction in neural networks.

Findings

Outperforms state-of-the-art object detectors like Faster R-CNN and YOLO.

Successfully mimics arithmetic operations without explicit rules.

Validates the approach on vision and CAPTCHA problems.

Abstract

Many real-world problems, e.g. object detection, have outputs that are naturally expressed as sets of entities. This creates a challenge for traditional deep neural networks which naturally deal with structured outputs such as vectors, matrices or tensors. We present a novel approach for learning to predict sets with unknown permutation and cardinality using deep neural networks. Specifically, in our formulation we incorporate the permutation as unobservable variable and estimate its distribution during the learning process using alternating optimization. We demonstrate the validity of this new formulation on two relevant vision problems: object detection, for which our formulation outperforms state-of-the-art detectors such as Faster R-CNN and YOLO, and a complex CAPTCHA test, where we observe that, surprisingly, our set based network acquired the ability of mimicking arithmetics without any rules being coded.