Monotonic Differentiable Sorting Networks

TL;DR

This paper introduces a novel family of sigmoid functions to create monotonic differentiable sorting networks, improving gradient correctness and performance over previous non-monotonic methods.

Contribution

It proposes a new relaxation of conditional swap operations ensuring monotonicity in differentiable sorting networks, with theoretical proof and empirical improvements.

Findings

Monotonic differentiable sorting networks guarantee correct gradient signs.

The proposed sigmoid functions produce effective monotonic sorting networks.

Experiments show improved performance over previous methods.

Abstract

Differentiable sorting algorithms allow training with sorting and ranking supervision, where only the ordering or ranking of samples is known. Various methods have been proposed to address this challenge, ranging from optimal transport-based differentiable Sinkhorn sorting algorithms to making classic sorting networks differentiable. One problem of current differentiable sorting methods is that they are non-monotonic. To address this issue, we propose a novel relaxation of conditional swap operations that guarantees monotonicity in differentiable sorting networks. We introduce a family of sigmoid functions and prove that they produce differentiable sorting networks that are monotonic. Monotonicity ensures that the gradients always have the correct sign, which is an advantage in gradient-based optimization. We demonstrate that monotonic differentiable sorting networks improve upon previous differentiable sorting methods.