Towards Zero-shot Commonsense Reasoning with Self-supervised Refinement of Language Models

TL;DR

This paper explores a self-supervised refinement method to enhance pre-trained language models for zero-shot commonsense reasoning, specifically targeting the Winograd Schema Challenge, without relying on annotated datasets.

Contribution

It introduces a novel self-supervised learning approach that refines language models through linguistic perturbations, enabling zero-shot reasoning capabilities.

Findings

Demonstrates viability of zero-shot commonsense reasoning on multiple benchmarks

Shows that loss landscape refinement improves reasoning without fine-tuning

Validates approach on Winograd Schema Challenge and related tasks

Abstract

Can we get existing language models and refine them for zero-shot commonsense reasoning? This paper presents an initial study exploring the feasibility of zero-shot commonsense reasoning for the Winograd Schema Challenge by formulating the task as self-supervised refinement of a pre-trained language model. In contrast to previous studies that rely on fine-tuning annotated datasets, we seek to boost conceptualization via loss landscape refinement. To this end, we propose a novel self-supervised learning approach that refines the language model utilizing a set of linguistic perturbations of similar concept relationships. Empirical analysis of our conceptually simple framework demonstrates the viability of zero-shot commonsense reasoning on multiple benchmarks.