PHALM: Building a Knowledge Graph from Scratch by Prompting Humans and a Language Model

TL;DR

This paper introduces PHALM, a novel approach for constructing a Japanese event knowledge graph from scratch by combining human prompts and large language models, enabling effective commonsense reasoning.

Contribution

The paper presents a new method for building knowledge graphs from scratch using both crowdworkers and LLMs, specifically applied to Japanese commonsense knowledge.

Findings

Built a Japanese event knowledge graph with high acceptability.

Trained Japanese commonsense generation models with the graph.

Demonstrated differences between human and LLM prompting methods.

Abstract

Despite the remarkable progress in natural language understanding with pretrained Transformers, neural language models often do not handle commonsense knowledge well. Toward commonsense-aware models, there have been attempts to obtain knowledge, ranging from automatic acquisition to crowdsourcing. However, it is difficult to obtain a high-quality knowledge base at a low cost, especially from scratch. In this paper, we propose PHALM, a method of building a knowledge graph from scratch, by prompting both crowdworkers and a large language model (LLM). We used this method to build a Japanese event knowledge graph and trained Japanese commonsense generation models. Experimental results revealed the acceptability of the built graph and inferences generated by the trained models. We also report the difference in prompting humans and an LLM. Our code, data, and models are available at github.com/nlp-waseda/comet-atomic-ja.