Consistency of Compositional Generalization across Multiple Levels

TL;DR

This paper introduces a meta-learning framework to ensure models generalize consistently across multiple levels of compositional complexity, demonstrated on visual question answering and video grounding tasks.

Contribution

It proposes a novel multi-level optimization approach with meta-weight-nets to improve consistency in compositional generalization across different composition levels.

Findings

Effective in achieving consistent compositional generalization

Improves performance on GQA-CCG dataset

Applicable to visual question answering and video grounding

Abstract

Compositional generalization is the capability of a model to understand novel compositions composed of seen concepts. There are multiple levels of novel compositions including phrase-phrase level, phrase-word level, and word-word level. Existing methods achieve promising compositional generalization, but the consistency of compositional generalization across multiple levels of novel compositions remains unexplored. The consistency refers to that a model should generalize to a phrase-phrase level novel composition, and phrase-word/word-word level novel compositions that can be derived from it simultaneously. In this paper, we propose a meta-learning based framework, for achieving consistent compositional generalization across multiple levels. The basic idea is to progressively learn compositions from simple to complex for consistency. Specifically, we divide the original training set into multiple validation sets based on compositional complexity, and introduce multiple meta-weight-nets to generate sample weights for samples in different validation sets. To fit the validation sets in order of increasing compositional complexity, we optimize the parameters of each meta-weight-net independently and sequentially in a multilevel optimization manner. We build a GQA-CCG dataset to quantitatively evaluate the consistency. Experimental results on visual question answering and temporal video grounding, demonstrate the effectiveness of the proposed framework. We release GQA-CCG at https://github.com/NeverMoreLCH/CCG.