Real-Time Music Generation on Your Mac: Meet Magenta RealTime 2

You've probably played with AI music generators before, but there's always a catch: you type a prompt, wait twenty seconds, and get back a clip you can't tweak. What if you could generate audio as fast as it plays, right on your laptop, and even hook it into your DAW? That's exactly what Magenta RealTime 2 (MRT2) sets out to do.

MRT2 is an open-weights model for real-time music generation from the Magenta team at Google, and the repository gives you everything you need to run it, build on it, or embed it into your own applications. It's a full stack: a Python inference library, a C++ engine, and pre-built apps that show what's possible.

What It Does

MRT2 is a state-of-the-art model for real-time music generation, and "real-time" here means something specific: it generates audio faster than playback speed. That's a hard constraint that most music generation models don't even attempt to meet.

The architecture breaks down into a few key pieces. There's an open-weights model hosted on Hugging Face, a Python library called magenta-rt that handles inference with both JAX and MLX backends, and a C++ inference engine (magentart::core) built for efficient streaming on Apple Silicon. On top of that, the repo includes example applications: an AUv3 plugin for DAWs, a standalone macOS app, and a couple of experimental apps for exploring note control and prompt space.

The hardware story is worth paying attention to. Real-time streaming requires Apple Silicon, and you get two model sizes to choose from. The smaller mrt2_small (230M parameters) runs real-time on any M-series Mac, including Air models. The larger mrt2_base (2.4B parameters) produces higher quality output but needs a Pro Max chip for real-time streaming. Both models can also run offline (non-real-time) inference on any Apple Silicon Mac or NVIDIA GPU via the Python library.

Why It's Cool

What makes MRT2 interesting isn't just that it generates music—it's the way the project is packaged for real use.

• It's genuinely real-time. The whole point is generating audio faster than playback. That's a fundamentally different experience from waiting on a batch job. You can iterate on prompts and hear results immediately.

• You get a choice of backends. The Python library supports both JAX and MLX. If you're on Apple Silicon, MLX gives you native Metal performance. If you're on NVIDIA hardware, JAX has you covered.

• There's a C++ engine for real applications. The magentart::core library isn't just a wrapper around Python—it's a proper C++ inference engine. That's what makes DAW plugins and standalone apps possible. You're not limited to running things in a Jupyter notebook.

• The example apps are actually useful. The AUv3 plugin and standalone macOS app are "all-in-one" — they're not throwaway demos but functional starting points. The Jam and Collider apps explore different interaction models (note control vs. prompt space), which gives you a sense of what's possible beyond the obvious use cases.

• It's open-weights, not just open-source. The model weights are on Hugging Face, so you're not renting inference from a hosted API. You own the whole stack.

One thing I appreciate: the README is honest about hardware limitations. They don't claim the 2.4B model runs on an M1 Air. The compatibility table is right there in the repo, so you know exactly what to expect before you download anything.

How to Try It

The quickest path is the Python CLI on Apple Silicon. It takes a few commands:

# Install uv if you haven't and create a venv
curl -LsSf https://astral.sh/uv/install.sh | sh
uv venv --python 3.12
source .venv/bin/activate

# Install dependencies (Python dev)
uv pip install "magenta-rt[mlx]"

# Download resources: style model and codec model
mrt models init
# Download the streaming model you want to use
mrt models download
# Generate 4 seconds of music
mrt mlx generate --prompt "disco funk" --duration 4.0 --model=mrt2_base

If you want to do local development, you can clone the repo with submodules and install it in editable mode:

git clone --recurse-submodules https://github.com/magenta/magenta-realtime.git
cd magenta-realtime
uv pip install -e ".[mlx]"

For the C++ route, you'll build a command-line tool called hello_mrt2 that does the same thing from the terminal:

uv pip install "cmake<3.28"
cmake . -B build
cmake --build build --target hello_mrt2 -j10

./build/examples/hello_mrt2/hello_mrt2 \
    ~/Documents/Magenta/magenta-rt-v2/models/mrt2_base/mrt2_base.mlxfn \
    ~/Documents/Magenta/magenta-rt-v2/resources \
    100 \
    --prompt "ambient pads with sub bass"

The full documentation lives at magenta.github.io/magenta-realtime, and if you'd rather skip the build entirely, there are pre-built apps and plugins at magenta.withgoogle.com/mrt2.

Final Thoughts

MRT2 is for developers who want to build music generation into something real—a plugin, an app, an instrument. The real-time constraint shapes everything about the design, and the project gives you multiple levels to work at: a simple CLI, a Python library, or a C++ engine. It's not going to run on every machine, but if you're on Apple Silicon, it's a remarkably accessible way to get into streaming music generation. Future updates will add supervised fine-tuning, which should open up even more possibilities. If you've been waiting for a reason to tinker with AI music on your Mac, this is a solid place to start.

Follow @githubprojects for more developer tools and open source projects.