Testing the Depth of ChatGPT's Comprehension via Cross-Modal Tasks Based on ASCII-Art: GPT3.5's Abilities in Regard to Recognizing and Generating ASCII-Art Are Not Totally Lacking

TL;DR

This paper evaluates GPT3.5's ability to understand and generate ASCII-art, testing its cross-modal comprehension skills beyond natural language, revealing its partial capabilities in visual tasks.

Contribution

It introduces a novel approach to assess GPT3.5's visual understanding through ASCII-art tasks, expanding the evaluation beyond traditional text-based benchmarks.

Findings

GPT3.5 shows partial recognition of ASCII-art images.

The model's performance varies with different visual transformations.

GPT3.5 can generate ASCII-art but with limitations.

Abstract

Over the eight months since its release, ChatGPT and its underlying model, GPT3.5, have garnered massive attention, due to their potent mix of capability and accessibility. While a niche-industry of papers have emerged examining the scope of capabilities these models possess, the information fed to and extracted from these networks has been either natural language text or stylized, code-like language. Drawing inspiration from the prowess we expect a truly human-level intelligent agent to have across multiple signal modalities, in this work we examine GPT3.5's aptitude for visual tasks, where the inputs feature content provided as ASCII-art without overt distillation into a lingual summary. We conduct experiments analyzing the model's performance on image recognition tasks after various transforms typical in visual settings, trials investigating knowledge of image parts, and tasks covering image generation.