ToonCrafter: Fill In the Gaps Between Your Cartoon Frames

You've drawn two keyframes of an animation. The character starts in one pose, ends in another, and everything in between is... nothing. Traditionally, you'd sit down and hand-draw every in-between frame, or spend hours nudging keyframes in software until the motion looks right. ToonCrafter is a research project that takes a different approach: give it two cartoon images, and it generates the frames connecting them.

What It Does

ToonCrafter is a generative cartoon interpolation tool from researchers at CUHK and Tencent AI Lab, presented at SIGGRAPH Asia 2024 (Journal Track). It takes two cartoon images—a starting frame and an ending frame—and generates the intermediate motion between them, producing a short video. The core idea is that it leverages pre-trained image-to-video diffusion priors rather than training a model from scratch on cartoon data alone.

That matters because cartoon animation has its own set of problems that general video interpolation models handle poorly. Cartoon motion is often exaggerated, non-rigid, and stylized in ways that don't follow the physics of real-world footage. By building on image-to-video diffusion models, ToonCrafter can produce plausible in-between motion while staying faithful to the cartoon aesthetic.

The project also supports sparse sketch guidance. In addition to the starting and ending frames, you can supply a sketch that constrains how the motion should unfold, giving you a degree of directorial control over the generated interpolation.

Why It's Cool

• It tackles a genuinely annoying problem. In-betweening is tedious, and automated tools for it tend to work best on live-action footage. Cartoons break those tools. ToonCrafter is aimed squarely at the animation use case.

• Sketch guidance is the interesting part. Being able to nudge the interpolation with a sparse sketch means you're not just rolling the dice and hoping the model guesses your intent. You can steer the motion without redrawing every frame.

• It's built on existing priors. Rather than training a video model from zero, the project reuses pre-trained image-to-video diffusion priors. That's a practical design choice—it means the model inherits a general understanding of motion and can focus its learning on the cartoon-specific quirks.

• The output resolution is honest about its scope. The showcases are at 512x320. This isn't a production pipeline for a feature film; it's a research tool for exploring what generative interpolation can do with stylized content.

• There are multiple ways to try it without setting up anything locally. The README links to a Hugging Face demo, a Replicate demo, and a Colab notebook. If you just want to see what it does, you don't need to touch a terminal.

One thing worth noting: the README includes an explicit warning that the team has not set up any official profit-making projects or web applications, and to be cautious. If you see a paid service claiming to be ToonCrafter, it isn't from the original authors.

How to Try It

The fastest route is one of the hosted demos linked from the repository:

1. Hugging Face Space — the official demo, linked in the README badges.
2. Replicate — a hosted version by a third party (fofr/tooncrafter).
3. Colab notebook — a Jupyter-based demo by camenduru if you want a bit more control without a full local install.

If you'd rather run it yourself, head to the repository and follow the setup instructions there:

• Repository: https://github.com/doubiiu/tooncrafter
• Project page: https://doubiiu.github.io/projects/ToonCrafter/
• Paper: https://arxiv.org/abs/2405.17933
• Video overview: https://www.youtube.com/watch?v=u3F35do93_8

The README's showcase section is worth a look before you dive in—it shows input starting frames, ending frames, and the generated videos side by side, plus examples of the sparse sketch guidance workflow.

Final Thoughts

ToonCrafter is a research project, and it reads like one. The showcases are small, the scope is specific, and the README spends more time pointing at papers and demos than at installation minutiae. That's fine—it's not trying to be a product. If you're an animator curious about where generative interpolation is heading, or a developer interested in how diffusion priors transfer to stylized domains, it's worth an afternoon. The hosted demos make it easy to form your own opinion before committing to a local setup. Just don't expect it to replace your in-betweening workflow tomorrow; expect it to show you what that workflow might look like in a year or two.

Follow @githubprojects for more developer tools and open source projects.