Building 3D Worlds by Describing Them: LocalGPT Runs Locally and Speaks Natural Language
You've probably had an idea for a 3D scene in your head — a desert with pyramids, a cozy cabin in the snow — and then remembered that turning that idea into geometry, materials, and lighting usually means hours in a modeling tool or writing a lot of boilerplate. LocalGPT takes a different approach: you describe the world you want, and it builds it. It's open source, and it runs on your own machine.
What It Does
LocalGPT is a tool for building explorable 3D worlds with natural language. You give it a prompt, and it generates geometry, materials, lighting, audio, and behaviors. The world-building side ships as a standalone binary called localgpt-gen, built on the Bevy game engine and written in Rust (it's published on crates.io, so the install is a single cargo command).
The system covers a solid range of primitives out of the box: parametric shapes like boxes, spheres, cylinders, capsules, planes, toruses, pyramids, tetrahedrons, icosahedrons, and wedges. On top of that you get PBR materials (color, metalness, roughness, emissive, alpha, double-sided), several light types (point, spot, directional) with configurable color and intensity, and behaviors such as orbit, spin, bob, look_at, pulse, path_follow, and bounce. There's also audio — ambient sounds like wind, rain, forest, ocean, and cave, plus spatial emitters.
When you're done, you can export to glTF/GLB, to HTML for viewing in a browser, or as screenshots. Complete worlds can be saved and loaded as reusable "world skills."
There's a second piece beyond world building: localgpt itself is an AI assistant with chat, memory, and a daemon mode.
Why It's Cool
It runs locally, and that's the whole point. The tagline isn't a footnote — "open source, runs locally" is the architecture. For anyone who's been watching AI tooling drift toward hosted services with per-request pricing, a local-first 3D generator is a genuinely different proposition. Your prompts, your scenes, and whatever style the system learns about you stay on your machine.
The memory system learns your taste. This is the detail that stood out most. LocalGPT's memory tracks your creative style across sessions — palettes, lighting preferences, entity templates — and applies them automatically in future generations. That's a small feature with outsized impact: instead of re-specifying "warm desert tones, low sun angle" every single time, the system starts to anticipate you. It turns a series of one-off prompts into something closer to a working relationship with a tool.
Headless mode makes batch generation practical. You can queue world experiments and generate without a window at all:
localgpt-gen headless --prompt "Build a cozy cabin in a snowy forest" --style "Studio Ghibli"
That opens up overnight batch runs, CI pipelines, and scripted variations — the kind of thing that's awkward when a tool assumes a GUI. Being able to generate a hundred style variations while you sleep is a real workflow, not a demo trick.
MCP support means it plugs into the editors you already use. Run localgpt-gen mcp-server and you can drive world generation from Claude Desktop, Codex Desktop, or any MCP-compatible editor like VS Code, Zed, or Cursor. For CLI tools (Claude CLI, Gemini CLI, Codex CLI), a --connect flag routes tool calls to your existing window. You don't have to adopt a new interface — you bring the one you're already in.
The export options are pragmatic. glTF/GLB for pipelines, HTML for sharing something viewable in a browser, screenshots for documentation. It's the unglamorous plumbing that determines whether generated content actually goes anywhere.
How to Try It
If you just want to run it, install from crates.io:
# World Building
cargo install localgpt-gen
# AI Assistant (chat, memory, daemon)
cargo install localgpt
If you want to hack on the code, clone the repo and use cargo run to iterate without installing:
git clone https://github.com/localgpt-app/localgpt.git
cd localgpt
# World Building
cargo run -p localgpt-gen -- "Create a desert scene with pyramids"
# AI Assistant
cargo run -- chat
cargo run -- daemon start
From there, you can start interactive mode with just localgpt-gen, or pass an initial prompt directly:
localgpt-gen "Create a desert scene with pyramids and a UFO hovering above"
You can also load an existing scene with --scene ./world.glb, or add --verbose if you want to see what's happening under the hood.
The repository is at github.com/localgpt-app/localgpt. Full options — feature flags, headless builds, Docker — are documented in the project's installation docs.
Final Thoughts
LocalGPT is best suited to developers who want to experiment with AI-driven 3D generation without handing their prompts to a cloud service, and who are comfortable in a Rust and Bevy environment. The headless mode and MCP server support suggest it's aimed at people who want this in a pipeline, not just in a demo window. If you've been curious about text-to-3D but put off by the setup cost or the hosting model, this is worth a look — the memory system alone is the kind of feature that changes how you use a tool day to day.
Follow @githubprojects for more developer tools and open source projects.