Crawl Free Proxy Nodes, Filter the Junk, and Speed-Test Them Yourself

If you've ever grabbed a public subscription link full of free nodes, you know the drill: half of them are dead, a quarter of them are painfully slow, and a few just refuse to connect at all. You end up manually testing them one by one, which is about as fun as watching paint dry. SubCrawler is a project that automates that whole mess—it crawls free nodes, filters out the problematic ones, and gives you tools to speed-test what's left on your own machine.

What It Does

SubCrawler is a GitHub-based pipeline for collecting and cleaning up free proxy nodes. It crawls nodes, applies a filter to remove ones that are likely to cause speed-test problems, and then generates subscription files you can import into clients like v2rayN, SS, or SSR. The output is base64-encoded by default, so it drops straight into most clients without extra work.

The project runs primarily through GitHub Actions, which means the crawling and filtering happen automatically on a schedule. But here's the catch the README is upfront about: those GitHub Actions run on overseas servers, so what works there might not work for you locally. That's why SubCrawler also ships with a local build script. You can run localFilter.bat on Windows or bash localFilter on Linux and macOS to re-filter the nodes on your own machine and get a result that actually reflects your network. The README explicitly recommends this two-step approach: use the action-generated list, then run a local second-pass filter before you rely on it.

Why It's Cool

• It's honest about its own limitations. The README doesn't pretend the GitHub Action output is perfect. It flat-out says the action nodes are tested on foreign servers and "国内不保证能用" (not guaranteed to work domestically), then tells you to re-test locally. That kind of transparency is rare and genuinely useful.

• The filter is specific, not vague. Rather than saying "we remove bad nodes," the README lists exactly what gets cut: SSR with none, rc4, or rc4-md5 encryption; SS with aes-128-cfb, aes-256-cfb, or rc4-md5; and VMess with none, grpc, h2, or auto. These are the configurations that tend to break speed tests, so the filter is targeting real pain points rather than guessing.

• It handles protocols that others skip. The generated node lists include separate outputs for VLESS and Hysteria2, with a note that these aren't speed-tested because LiteSpeedTest doesn't support them. Instead of silently dropping them, SubCrawler gives you the raw nodes and lets you decide.

• The local testing workflow is practical. The README walks through using LiteSpeedTest on a Linux server, including a sample config.json tuned for ping-only testing with high concurrency (1024) and a 5-second timeout. It even shows the exact commands to copy the output to an nginx path and base64-encode it into a V2Ray subscription. That's a complete, copy-pasteable workflow.

• It's more than just nodes. SubCrawler also generates hosts files—one for ad blocking, one for re-filtered Google hosts, and one for GitHub and common domains. Plus there's a curated list of IP and DNS leak-testing sites for when you want to verify your proxy is actually anonymous.

How to Try It

1. Head to the repository and grab the pre-generated node lists from the sub/share/ directory. The VLESS and Hysteria2 files are separate, and there's a combined list as well. These are base64-encoded, so you can paste them directly into v2rayN or similar clients.

2. If you want nodes that actually work on your network, clone the repo and run the local filter:

   # Windows
   localFilter.bat
   
   # Linux / macOS
   bash localFilter
   

   Or if you prefer:

   chmod +x localFilter && ./localFilter
   

3. For serious local testing, set up LiteSpeedTest on a Linux box. The README provides a config.json you can use as a starting point—it's set to ping-only mode with sorting by speed. Run it, copy the output to your nginx static path, and base64-encode it into a subscription file.

4. If you need to convert the subscription format, the README links to several subconverter options, including a local build that supports VLESS, Hysteria2, and AnyTLS.

5. For hosts files, you can grab the ad-blocking, Google, or GitHub lists directly from the sub/share/ folder.

Final Thoughts

SubCrawler isn't trying to be a polished, one-click solution—and that's fine. It's a set of scripts and workflows for people who are comfortable running a command line and want more control over their free node pipeline. The biggest strength is the local re-filtering step, which acknowledges that a node's usefulness depends entirely on where you're connecting from. If you're the type who already tinkers with proxies and wants a cleaner starting point, this will save you time. If you're looking for a fully managed service, you'll need to look elsewhere. For everyone in between, it's a solid, no-nonsense tool worth bookmarking.

Follow @githubprojects for more developer tools and open source projects.