On-Device Speech AI in Swift Without the Cloud

You've probably built an app that needs to transcribe audio, figure out who's speaking, or generate speech from text. And you've probably run into the same wall: cloud APIs mean latency, per-request costs, and audio leaving your user's device. The Argmax Open-Source SDK is a Swift package that puts those capabilities directly on-device, so you get speech-to-text, speaker diarization, and text-to-speech without a server round-trip.

What It Does

The SDK is a collection of turn-key on-device inference frameworks, each wrapping a specific model family:

• WhisperKit handles speech-to-text using OpenAI Whisper
• SpeakerKit handles speaker diarization using Pyannote
• TTSKit handles text-to-speech using Qwen-TTS

It's distributed as a Swift package, so it slots into your existing Xcode workflow through Swift Package Manager (there's a Homebrew option too). Everything runs locally on the device, and the README points to a Hugging Face model repository for the WhisperKit CoreML models. The package is tested on each release and lists its supported Swift versions and platforms on Swift Package Index.

WhisperKit also ships with a Swift CLI and a local server option, so you're not limited to embedding the library in an app—you can also expose transcription over HTTP endpoints if that fits your architecture better.

Why It's Cool

• Three speech problems, one package. Transcribing audio, separating speakers, and generating speech are usually three separate integrations with three separate SDKs. Here they're bundled under one roof, which means one dependency line and a consistent Swift API surface.

• The privacy angle is real. On-device inference means audio never leaves the user's hardware. For apps handling voice memos, meetings, or anything sensitive, that's not a nice-to-have—it's often the deciding factor.

• It respects large inputs. The README specifically calls out memory-efficient loading for large files. Anyone who's tried to transcribe a long recording on a phone knows that's the part that usually falls over.

• Model selection is built in. You're not stuck with whatever model ships by default. WhisperKit includes guidance on model selection and even generating models, so you can trade off size against accuracy for your use case.

• The local server option is a smart escape hatch. If your app isn't native—or you just want to test transcription from a script—the server mode with documented API endpoints, supported parameters, and client examples gets you there without writing Swift UI code first.

• Honest scoping. The README is upfront about what the open-source SDK doesn't do. Real-time transcription with speakers, custom vocabulary for frontier accuracy, and Android support live in the Pro SDK. That kind of clarity saves you from discovering the gap after you've committed.

How to Try It

1. Add the package to your project via Swift Package Manager. In Package.swift:

dependencies: [
    .package(url: "https://github.com/argmaxinc/argmax-oss-swift.git", from: "0.0.0")
]

(Check the repository for the current version tag.)

1. In Xcode, add the package dependency through File > Add Package Dependencies and point it at the repo. The README's Prerequisites and Xcode Steps sections cover anything platform-specific you'll need first.

2. For WhisperKit, start with the Quick Example in the README to get a basic transcription running, then look at Model Selection to pick a model that fits your device and accuracy needs.

3. If you want to test transcription from outside an app, build and start the local server, then hit its API endpoints from any HTTP client. The README includes supported parameters and client examples.

4. Explore TTSKit's quick example and generation options if you need speech output—including custom voices and real-time streaming playback.

The full instructions, including the Homebrew path and CLI usage, are in the repo: github.com/argmaxinc/argmax-oss-swift. There's also a Discord community linked from the README if you get stuck.

Final Thoughts

This is a practical pick if you're building an Apple-platform app that needs speech capabilities and you care about privacy, latency, or per-request costs. The three-framework structure means you can adopt just the piece you need—WhisperKit alone is useful even if diarization and TTS aren't on your roadmap. Just go in knowing the boundaries: if you need real-time multi-speaker transcription or Android, the README will point you to the Pro SDK. For everyone else, this is a solid, no-nonsense starting point for on-device speech in Swift.

Follow @githubprojects for more developer tools and open source projects.