Getting Neural Networks Onto Devices Without Dragging Along a Runtime

You've trained a model in PyTorch, it works great on your laptop, and now you need it to run on a phone or an embedded board. That's usually where the dependency hell begins. ncnn is a framework built specifically to avoid that problem: it's a high-performance neural network inference framework with no third-party runtime dependencies, aimed at mobile, embedded, and desktop deployment.

What It Does

ncnn runs neural network inference across CPU and Vulkan GPU backends, and it's designed to be dropped onto phones, PCs, browsers, and edge devices. The "no third-party runtime dependencies" part is the headline — you're not shipping a pile of libraries alongside your model just to get it to execute.

The project comes with pnnx, a tool for converting PyTorch and ONNX models into ncnn's own format. The recommended path for newcomers is straightforward: PyTorch to pnnx to ncnn. You export your model, and you get two files — a .param file describing the network and a .bin file holding the weights.

The API is available in both C++ and Python. You load the param and model files into an ncnn::Net, create an extractor, feed in an input tensor, and pull out the result. That's the whole loop.

It's worth noting this isn't a side experiment — ncnn is used in Tencent applications including QQ, Qzone, WeChat, and Pitu, so it's been through real production traffic.

Why It's Cool

• The dependency story is the real selling point. Mobile and embedded deployment is often less about raw speed and more about what you can actually fit and link on the target device. A framework with no third-party runtime dependencies removes a whole category of build and packaging headaches.

• Vulkan gives you GPU acceleration without CUDA. If you're targeting Android or other platforms where CUDA isn't an option, having a Vulkan backend means you can still reach the GPU. That's a practical choice for the hardware people actually ship on.

• The Python path lowers the barrier. You can prototype the export in a PyTorch environment with a pip3 install pnnx, then test inference in Python with numpy arrays before committing to the C++ integration. That's a nice progression from experiment to production.

• The build targets are unusually broad. The build documentation covers Linux, Windows, macOS, Raspberry Pi 3 and 4, POWER, Android, NVIDIA Jetson, iOS, WebAssembly, AllWinner D1, and Loongson 2K1000. That list tells you something about the project's ambitions — it's not just "phones," it's the long tail of edge hardware.

• Prebuilt releases exist. There's a releases page with downloadable artifacts, including a full source zip, so you don't have to start from a bare clone if you'd rather grab a package.

How to Try It

The quickest route starts in a PyTorch environment. Install pnnx:

pip3 install pnnx

Then export a model. Here's the example from the README — a small conv/relu/linear network:

import torch
import torch.nn as nn
import pnnx

class Model(nn.Module):
    def __init__(self):
        super().__init__()
        self.conv = nn.Conv2d(3, 8, 1)
        self.relu = nn.ReLU()
        self.fc = nn.Linear(8, 4)

    def forward(self, x):
        x = self.conv(x)
        x = self.relu(x)
        x = x.mean((2, 3))
        return self.fc(x)

model = Model().eval()

x = torch.rand(1, 3, 224, 224)
pnnx.export(model, "model.pt", (x,))

That produces model.ncnn.param and model.ncnn.bin. From there you can run inference in C++:

#include "net.h"

ncnn::Net net;
net.load_param("model.ncnn.param");
net.load_model("model.ncnn.bin");

ncnn::Mat in(224, 224, 3);

auto ex = net.create_extractor();
ex.input("in0", in);

ncnn::Mat out;
ex.extract("out0", out);

Or stay in Python if you want to sanity-check the output first:

import numpy as np
import ncnn

net = ncnn.Net()
net.load_param("model.ncnn.param")
net.load_model("model.ncnn.bin")

x = np.zeros((3, 224, 224), np.float32)
mat = ncnn.Mat(x)

ex = net.create_extractor()
ex.input("in0", mat)

ret, out = ex.extract("out0")
print(np.array(out).shape)

For the full workflows, the README points to the pnnx tools directory, the wiki page on using ncnn with PyTorch or ONNX, the Python API docs, and the examples folder. Build instructions live in the wiki and cover all those platforms listed above.

You can find the project and releases at github.com/Tencent/ncnn.

Final Thoughts

ncnn is aimed at people who have a trained model and a constrained device, and who'd rather not spend a week fighting link errors. The lack of runtime dependencies and the breadth of supported build targets make it a reasonable default when you're deploying to hardware you don't fully control. If you're working in pure desktop Python with no deployment constraints, you probably don't need it — but the moment "runs on a phone" or "runs on a Pi" enters the picture, it's worth a look. The PyTorch-to-pnnx path means you can test the whole thing in an afternoon before committing to anything.