ENOVA: Open Source LLM Serving with Auto Scaling Smarts

Deploying LLMs in production is still a pain. You either overprovision GPUs and burn cash, or underprovision and watch your app crash under load. Most solutions make you guess the right config.

ENOVA is an open source LLM serving engine that takes the guesswork out. It automatically scales your inference infrastructure and even recommends optimal configurations for your models. Here's why it might save your next deployment.

What It Does

ENOVA is a lightweight serving layer for large language models that sits between your app and your inference backend (like vLLM, TGI, or raw Hugging Face). It handles:

• Auto scaling – Spins up and down GPU instances based on real time traffic, not static rules.
• Configuration recommendations – Scans your model and workload to suggest batch sizes, GPU memory allocation, and concurrency settings.
• Load balancing – Distributes requests across multiple model replicas with smart queuing.
• Fault tolerance – Automatically retries failed requests and swaps out unhealthy instances.

Think of it as an intelligent proxy for your LLM endpoints that optimizes resource usage without manual tuning.

Why It's Cool

Most auto scaling solutions use simple thresholds like "scale up when CPU > 80%." That doesn't work well for LLMs because inference latency is non linear. A single large request can spike memory usage differently than many small ones.

ENOVA takes a different approach. It profiles your actual model's behavior under load and builds a cost model that predicts resource needs. It then scales based on that model, not generic metrics.

The configuration recommendation feature is particularly useful. Instead of reading conflicting blog posts about optimal batch sizes for Llama 3, you just point ENOVA at your model and let it run benchmarks. It'll tell you, "with 24 GB VRAM, use batch size 8 and tensor parallelism 2 for this workload."

Key features worth noting:

• Provider agnostic – Works with AWS, GCP, Azure, or on prem GPUs
• Simple YAML config – No complex Kubernetes operators required
• Real time dashboard – See token throughput, latency, and costs per request
• Cost optimization – Automatically picks cheaper spot instances when possible

How to Try It

Installation is straightforward with Docker or pip:

# Using pip
pip install enova-serve

# Or with Docker
docker pull ghcr.io/emerging-ai/enova:latest

Then create a config file:

model: "meta-llama/Llama-3-8B"
provider: "aws"
instance_type: "g5.2xlarge"
min_instances: 1
max_instances: 5
auto_config: true

Start the server:

enova serve --config config.yaml

It'll benchmark your model, generate a configuration, and start serving with auto scaling enabled. You can find more examples in the GitHub repo.

Final Thoughts

ENOVA isn't trying to replace vLLM or TGI. It's a layer above them that handles the operational pain. If you've ever had to manually tune batch sizes or explain to your manager why GPU costs doubled last month, give it a try.

The auto config feature alone saves hours of benchmarking. And the auto scaling means you can sleep without worrying about traffic spikes. It's one of those tools that quietly solves the boring but critical problems.

If you're running LLMs in production and aren't at the "just throw more GPUs at it" stage, ENOVA is worth a look.

Follow us at @githubprojects for more open source dev tools.