SketchBetween: Video-to-Video Synthesis for Sprite Animation via Sketches

TL;DR

SketchBetween is a novel video-to-video synthesis model designed specifically for sprite animation, translating keyframes and sketches into animated sprites, addressing the unique stylistic movement in 2D animation workflows.

Contribution

The paper introduces a new problem formulation aligned with animation workflows and a model that effectively maps keyframes and sketches to sprite animations, outperforming existing methods.

Findings

Model outperforms existing methods in sprite animation tasks.

Problem formulation effectively captures animation workflow requirements.

Demonstrates the feasibility of automated stylized sprite animation synthesis.

Abstract

2D animation is a common factor in game development, used for characters, effects and background art. It involves work that takes both skill and time, but parts of which are repetitive and tedious. Automated animation approaches exist, but are designed without animators in mind. The focus is heavily on real-life video, which follows strict laws of how objects move, and does not account for the stylistic movement often present in 2D animation. We propose a problem formulation that more closely adheres to the standard workflow of animation. We also demonstrate a model, SketchBetween, which learns to map between keyframes and sketched in-betweens to rendered sprite animations. We demonstrate that our problem formulation provides the required information for the task and that our model outperforms an existing method.