Improving Local Identifiability in Probabilistic Box Embeddings

TL;DR

This paper introduces a novel approach to improve local identifiability in probabilistic box embeddings by modeling box parameters with Gumbel distributions, enhancing learning capabilities.

Contribution

It proposes using Gumbel distributions for box parameters to maintain closure under intersection, addressing local identifiability issues in probabilistic embeddings.

Findings

Model with Gumbel distributions improves learning performance.

Expected intersection volume calculation enhances model accuracy.

Method maintains closure under intersection for probabilistic boxes.

Abstract

Geometric embeddings have recently received attention for their natural ability to represent transitive asymmetric relations via containment. Box embeddings, where objects are represented by n-dimensional hyperrectangles, are a particularly promising example of such an embedding as they are closed under intersection and their volume can be calculated easily, allowing them to naturally represent calibrated probability distributions. The benefits of geometric embeddings also introduce a problem of local identifiability, however, where whole neighborhoods of parameters result in equivalent loss which impedes learning. Prior work addressed some of these issues by using an approximation to Gaussian convolution over the box parameters, however, this intersection operation also increases the sparsity of the gradient. In this work, we model the box parameters with min and max Gumbel distributions, which were chosen such that space is still closed under the operation of the intersection. The calculation of the expected intersection volume involves all parameters, and we demonstrate experimentally that this drastically improves the ability of such models to learn.