Kolmogorov-Arnold Networks: When Neural Nets Learn to Rewire Their Own Activations

Intro

If you've been following ML research, you've probably seen the recent buzz around Kolmogorov-Arnold Networks (KANs). The central idea is deceptively simple but genuinely surprising: instead of using fixed activation functions on the nodes of a neural network, KANs learn the activation functions themselves — on the edges between nodes. That small shift unlocks some interesting behavior.

The community has already started collecting implementations, papers, and tools. One place to start is the awesome-kan repository on GitHub, which aggregates everything KAN-related in one place.

What It Does

KANs are a new type of neural network architecture inspired by the Kolmogorov-Arnold representation theorem. In a standard neural net, each node has a fixed activation (like ReLU or sigmoid), and the edges between nodes are weighted linear sums. KANs flip this: the edges contain learnable, univariate functions (often splines), and the nodes simply sum the incoming edge outputs.

That means the network can adapt its "activation" behavior per connection, rather than applying a single function to all inputs. The result is a model that can learn more complex internal representations without needing as many parameters or layers in some cases.

Why It’s Cool

A few things stand out:

• Interpretability. Because activations are learned on edges, you can actually visualize what each connection is doing. The splines are smooth and low-dimensional, making them easier to inspect than the high-dimensional weight spaces of traditional nets.
• Parameter efficiency. Early experiments suggest KANs can match or beat MLPs with far fewer parameters on tasks like symbolic formula learning, function approximation, and some physics problems. That's a big deal if you're working on resource-constrained devices or need fast inference.
• Flexible function approximation. Splines can represent a much richer family of functions than fixed activations. So KANs naturally handle discontinuities, piecewise behaviors, and sharp transitions that trip up standard ReLU-based networks.

The awesome-kan repo catalogs the original PyTorch implementation, a JAX version, a minimal KAN implementation, and several experimental branches. There are also papers, math-friendly tutorials, and benchmarks.

How to Try It

The easiest way to get started is to clone the PyTorch KAN implementation (linked in the repo). Here's the minimal setup:

git clone https://github.com/1ssb/torchkan.git   # one of the featured implementations
cd torchkan
pip install -r requirements.txt
python example.py                                 # runs a small comparison vs MLP

If you prefer a more hands-on demo, the repo includes a Jupyter notebook that trains a KAN on the classic XOR problem and visualizes the learned spline functions. You can also find implementations in JAX and TensorFlow.

For researchers, the awesome-kan repo has a curated list of papers, including the original 2024 preprint by Liu et al. and follow-up work on scaling, initialization, and hybrid architectures.

Final Thoughts

KANs are still early — the community hasn't fully figured out when they beat standard architectures, and they can be slower to train due to the spline updates. But the core idea is refreshing. It challenges the assumption that fixed activations are optimal, and it opens the door to more interpretable, adaptable networks.

If you work on small-scale models, symbolic regression, or any domain where you'd like to see why a network makes certain decisions, give KANs a spin. Even if they don't replace Transformers tomorrow, they're a fun experiment that might give you a new perspective on network design.

Curated by @githubprojects