Your One Stop Shop for 250+ LLMs: Meet the Portkey AI Gateway

If you've ever tried to switch an app from OpenAI to Anthropic or hit the rate limits on a popular model, you know the pain of managing multiple LLM APIs. Each one has its own SDK, authentication quirks, and error handling. It's not terrible with two providers, but it gets messy fast when you want to experiment with a dozen different models for your use case.

That's where the Portkey AI Gateway comes in. It's an open-source project that gives you one fast, unified API to talk to over 250 language models from different providers — and it handles all the ugly stuff behind the scenes.

What It Does

The gateway acts as a middleware layer between your app and any LLM provider. You send one standard request to the gateway, and it translates it into the format the target model expects. This means you can swap models or providers with a single parameter change — no code changes, no SDK rewrites, no new authentication logic.

Under the hood, it supports everything from OpenAI and Anthropic to open-source models via Hugging Face or local deployments. The API is designed to be familiar (OpenAI-compatible, actually), so your existing code with openai library calls can just point to the gateway URL and keep working.

Why It's Cool

Speed first. This isn't a bloated proxy. The gateway is built in Go and prioritizes low latency — on average, it adds less than 2ms overhead per request. That's basically nothing for most apps.

Provider fallbacks built in. You can define a primary model (like GPT-4) and a fallback (like Claude 3) if the first one fails or times out. Your user never sees the error, they just get a response.

Fine-grained control. You can set rate limits per user or per API key, cache responses for identical prompts, and even log every request for debugging. No more black box failures.

One SDK to rule them all. Instead of needing separate SDKs for each provider, you just use the Portkey SDK (or raw HTTP calls). This dramatically reduces your dependency count and the surface area for bugs.

How to Try It

The easiest way is to run it locally using Docker:

docker run -p 8787:8787 portkey/gateway

Then point your existing LLM requests to http://localhost:8787 with your target provider in the header. For example, using curl:

curl -X POST http://localhost:8787/ \
  -H "Content-Type: application/json" \
  -H "x-portkey-provider: openai" \
  -d '{"model": "gpt-3.5-turbo", "messages": [{"role": "user", "content": "Hello!"}]}'

You can also use the hosted version at Portkey if you want to skip the setup.

Final Thoughts

This project solves a real, boring problem in a clean way. It's not trying to be the next big thing — it's just a well-made tool that saves you time and headaches when dealing with multiple LLM providers. If you're building anything that consumes more than one language model, or you want to add fallback logic without rewriting your code, give it a try. It's one of those things you'll wonder why you didn't start using sooner.

Found this on @githubprojects