Learning Aggregation Functions

TL;DR

This paper introduces a learnable aggregation function (LAF) for set-based learning that can approximate various aggregation methods and improve performance over existing architectures like DeepSets and Principal Neighborhood Aggregation.

Contribution

The paper proposes a novel learnable aggregation function (LAF) that overcomes limitations of fixed aggregators and can approximate multiple functions for set representations.

Findings

LAF outperforms state-of-the-art set aggregation architectures.

LAF can approximate diverse aggregation functions including variance and skewness.

LAF is effective when combined with attention-based architectures.

Abstract

Learning on sets is increasingly gaining attention in the machine learning community, due to its widespread applicability. Typically, representations over sets are computed by using fixed aggregation functions such as sum or maximum. However, recent results showed that universal function representation by sum- (or max-) decomposition requires either highly discontinuous (and thus poorly learnable) mappings, or a latent dimension equal to the maximum number of elements in the set. To mitigate this problem, we introduce a learnable aggregation function (LAF) for sets of arbitrary cardinality. LAF can approximate several extensively used aggregators (such as average, sum, maximum) as well as more complex functions (e.g., variance and skewness). We report experiments on semi-synthetic and real data showing that LAF outperforms state-of-the-art sum- (max-) decomposition architectures such as DeepSets and library-based architectures like Principal Neighborhood Aggregation, and can be effectively combined with attention-based architectures.