Concept-Oriented Deep Learning with Large Language Models

TL;DR

This paper explores how large language models can be used for concept-oriented deep learning, emphasizing their ability to understand and extract concepts from text and images, and discussing their applications in AI chatbots.

Contribution

It analyzes the role of LLMs in concept understanding and extraction, highlighting the potential of multimodal LLMs for comprehensive concept learning.

Findings

Multimodal LLMs can represent both symbolic and sensory knowledge.

LLMs are effective in concept extraction from text and images.

They enhance AI chatbot capabilities through improved conceptual understanding.

Abstract

Large Language Models (LLMs) have been successfully used in many natural-language tasks and applications including text generation and AI chatbots. They also are a promising new technology for concept-oriented deep learning (CODL). However, the prerequisite is that LLMs understand concepts and ensure conceptual consistency. We discuss these in this paper, as well as major uses of LLMs for CODL including concept extraction from text, concept graph extraction from text, and concept learning. Human knowledge consists of both symbolic (conceptual) knowledge and embodied (sensory) knowledge. Text-only LLMs, however, can represent only symbolic (conceptual) knowledge. Multimodal LLMs, on the other hand, are capable of representing the full range (conceptual and sensory) of human knowledge. We discuss conceptual understanding in visual-language LLMs, the most important multimodal LLMs, and major uses of them for CODL including concept extraction from image, concept graph extraction from image, and concept learning. While uses of LLMs for CODL are valuable standalone, they are particularly valuable as part of LLM applications such as AI chatbots.