Meet Fay: A Fully Offline, Open Source Digital Human Framework

You know how most digital human / AI avatar projects need an internet connection, cloud APIs, or some proprietary service? They also tend to lock you into one LLM or one character model. Not fun if you want to experiment or keep everything local.

Fay is different. It’s a fully offline, open source digital human framework that lets you plug in any LLM and any 3D or 2D digital human model. No cloud, no API keys, no vendor lock-in. Just you, your hardware, and a lot of possibilities.

What It Does

Fay is a Python-based toolkit that turns a text or voice input into a lifelike digital human interaction. At its core:

• Speech-to-Text: Converts your voice to text (local ASR).
• LLM Engine: Feeds that text into a local LLM (like Llama, Qwen, or any OpenAI-compatible model).
• Text-to-Speech: Generates a voice response (local TTS).
• Digital Human Rendering: Animates a 3D or 2D character (via Live2D, VRM, or custom models) that lip-syncs and gestures.

It’s essentially a pipeline: voice → text → thought → speech → animation, all running on your machine.

Why It’s Cool

Most digital human frameworks are either:

• Cloud-dependent (needs internet, costs money, privacy concerns)
• Closed source (can’t customize, limited to their models)
• Tied to one specific LLM (can’t swap to your favorite or fine-tuned model)

Fay fixes all three.

Key features:

• Fully offline: Works without internet. Privacy-friendly and no recurring costs.
• Plug-and-play LLM: Supports any OpenAI-compatible API (including local ones like Ollama, vLLM, or llama.cpp). Swap models in a config file.
• Model-agnostic digital human: Works with Live2D, VRM, and even simple Unity or custom renders. You bring the character, Fay brings the logic.
• Extensible: Add your own TTS engine, ASR backend, or animation logic. It’s designed to be modular.
• No vendor lock-in: You control every component. Want to use a fine-tuned medical LLM with a custom avatar? Go ahead.

Use cases:

• Personal AI assistant with a face
• Interactive kiosks (no internet needed)
• Livestreaming avatars (local, low latency)
• Research and prototyping (swap models without rebuilding everything)

How to Try It

Getting started is pretty straightforward if you have Python and some GPU muscle (recommended for real-time TTS/ASR).

1. Clone the repo:

   git clone https://github.com/xszyou/Fay.git
   cd Fay
   

2. Install dependencies:

   pip install -r requirements.txt
   

3. Configure your LLM (edit config.json or use environment variables):

   • Set your LLM API endpoint (e.g., http://localhost:11434/v1 for Ollama)
   • Choose your TTS engine (e.g., Edge-TTS, VITS, or local Coqui)

4. Run:

   python main.py
   

   It’ll start a web interface or console UI. Speak into your mic, and Fay will respond via your configured digital human.

5. Add your own character:

   • For Live2D: drop your .model3.json files in the models folder.
   • For VRM: drop .vrm files.

Full docs are in the README — it’s well written and includes examples.

Final Thoughts

Fay isn’t trying to be the most polished consumer product. It’s a developer tool first. If you’ve ever been frustrated by being forced into one ecosystem for AI avatars, this is your escape hatch.

It’s not perfect — you’ll need decent hardware for low latency, and the animation pipeline could use more polish. But the modularity and offline-first design make it a solid foundation for anything from a desktop assistant to a museum kiosk.

If you want to experiment with digital humans without selling your privacy or signing up for a monthly subscription, give Fay a shot. It’s refreshingly open.

Found this interesting? We share open source projects like this daily. Follow us @githubprojects for more.