Densely Connected Attention Propagation for Reading Comprehension

TL;DR

This paper introduces DecaProp, a densely connected neural network with attention-based links for reading comprehension, achieving state-of-the-art results across multiple benchmarks.

Contribution

DecaProp uniquely combines dense layer connections with attention-based links, enhancing passage-query relationship modeling in reading comprehension.

Findings

Achieves state-of-the-art results on four RC benchmarks.

Outperforms existing models by up to 14.2% in F1 score.

Demonstrates effectiveness of attention-based dense connections.

Abstract

We propose DecaProp (Densely Connected Attention Propagation), a new densely connected neural architecture for reading comprehension (RC). There are two distinct characteristics of our model. Firstly, our model densely connects all pairwise layers of the network, modeling relationships between passage and query across all hierarchical levels. Secondly, the dense connectors in our network are learned via attention instead of standard residual skip-connectors. To this end, we propose novel Bidirectional Attention Connectors (BAC) for efficiently forging connections throughout the network. We conduct extensive experiments on four challenging RC benchmarks. Our proposed approach achieves state-of-the-art results on all four, outperforming existing baselines by up to $2.6\%-14.2\%$ in absolute F1 score.