Tired of Manually Filtering Your Proxy Nodes? Let a Crawler Do the Grunt Work

If you've ever maintained a proxy subscription, you know the drill: you scrape a bunch of subscription URLs, paste them into your client, and end up with a mess of dead nodes, slow connections, and outdated protocols. The filtering, testing, and converting cycle is tedious. That's where Subcrawler comes in—a project that automates the heavy lifting of crawling, filtering, and testing proxy nodes so you can focus on actually using them rather than babysitting your config files.

What It Does

Subcrawler is a tool that aggregates proxy nodes from various sources, filters out problematic ones, and runs them through speed tests—all while generating ready-to-use subscription files. The project is built around GitHub Actions for automated cloud runs, but it also ships with local scripts for users who want more control over the testing environment.

At its core, the project does three main things:

• Filters nodes by removing entries with known weak or broken encryption methods (like SS's aes-128-cfb or VMESS's none and grpc types)
• Tests nodes for speed and availability, either through GitHub Actions (using foreign servers) or locally on your own machine
• Generates output in base64-encoded subscription format that works directly with clients like v2rayN, SS, and SSR

The architecture is straightforward: you either run the included scripts (localFilter.bat on Windows, localFilter on Linux/Mac) or let the GitHub Action handle it automatically. The project also produces supplementary files like ad-blocking hosts lists and curated Google hosts.

Why It's Cool

What makes Subcrawler stand out is its honest approach to the proxy-testing problem. It doesn't pretend that a single test from a foreign server will work perfectly for everyone.

• It acknowledges regional differences. The README is upfront that GitHub Action tests run on foreign servers, meaning results might not hold up domestically. It explicitly recommends doing a local secondary speed test to filter nodes for your actual use case. That's refreshingly practical.

• It handles protocols others skip. The project generates nodes for vless and hysteria2, but notably marks them as "untested" because the popular LiteSpeedTest tool doesn't support them. That transparency—saying "here are nodes, but we couldn't verify them with our standard tooling"—is more useful than pretending everything is equally reliable.

• It's a complete ecosystem, not just a crawler. Subcrawler isn't just about pulling nodes. It points you to subscription converters (including a locally-hosted subconverter that supports vless/hy2/anytls), recommends speed-testing tools for different scenarios, and even provides a curated list of IP anonymity checkers to verify your proxy actually hides you properly.

• The local testing workflow is well thought out. The README shows a concrete workflow for self-hosting: use LiteSpeedTest with a ping-only config, copy the output to your nginx directory, then base64-encode it to generate a v2ray subscription link. It's a complete pipeline you can replicate on your own server.

• It filters intelligently. The node filtering isn't just about dead connections—it's about removing nodes with outdated or insecure encryption methods that are more likely to cause issues or get blocked.

How to Try It

Getting started with Subcrawler is straightforward, and you have two paths depending on your needs.

Option 1: Use the generated subscriptions

If you just want ready-made nodes, the project publishes them via GitHub Actions. You can grab:

• vless nodes (untested, since LiteSpeedTest doesn't support the protocol)
• hysteria2 nodes (also untested)
• All other merged nodes

These come as base64-encoded subscriptions, so they'll work directly with v2rayN, SS, and SSR clients. For other formats, you'll need to run them through a subscription converter—the README links to several options.

Option 2: Build locally for better results

For the most reliable nodes, run the filtering and testing on your own machine:

# On Windows
localFilter.bat

# On Linux or Mac OS
bash localFilter

# Or make it executable first
chmod +x localFilter && ./localFilter

If you want to set up your own speed-testing server, the README provides a LiteSpeedTest config example with ping-only mode:

{
  "group": "Default",
  "speedtestMode": "pingonly",
  "pingMethod": "googleping",
  "sortMethod": "rspeed",
  "concurrency": 1024,
  "testMode": 2,
  "timeout": 5,
  "fontSize": 24,
  "outputMode": 4,
  "unique": true,
  "language": "en",
  "theme": "rainbow"
}

You can check out the full repository at github.com/leon406/subcrawler for more details, including the complete node filtering rules and additional generated hosts files.

Final Thoughts

Subcrawler is best suited for anyone who's comfortable with a bit of command-line work and wants a more self-sufficient proxy setup. It's not a polished consumer app—it's a toolkit that gives you the pieces to build your own reliable node pipeline. The project's willingness to mark certain nodes as untested and its recommendation to re-test locally shows a level of honesty that's rare in this space. If you're tired of stale subscriptions and want more control over your proxy infrastructure, this is a solid foundation to build on. Just be ready to do a little configuration yourself—that's where the real reliability comes from.

Follow @githubprojects for more developer tools and open source projects.