waymore: When the Wayback Machine Isn't Enough, Download the Responses Too

If you've done any bug bounty or web reconnaissance work, you've probably leaned on tools like waybackurls or gau to pull historical URLs from the Wayback Machine and other archives. They're great at what they do. But here's the thing—they only give you the URLs. They don't give you what was actually on those pages. That's where waymore comes in.

What It Does

waymore is a Python tool that pulls URLs from multiple archival sources, but with a key difference: it can also download the archived responses for URLs found on the Wayback Machine and URLScan. That means you're not just getting a list of endpoints—you're getting the actual HTML, JavaScript, and other content that was served at those URLs. You can then search through all of that for additional links, developer comments, hidden parameters, or anything else that might be useful.

The tool aggregates URLs from a solid list of sources: Wayback Machine, Common Crawl, Alien Vault OTX, URLScan, Virus Total, GhostArchive, and Intelligence X (though that last one requires an academia or paid tier). It supports Python 3.7 and above, and you can install it via pip or pipx if you prefer an isolated environment.

One thing the author is upfront about: this tool is slow. It's built for coverage, not speed. The README explicitly states this multiple times, so don't expect it to be your quick-and-dirty URL fetcher.

Why It's Cool

The core differentiator here is response downloading. Most tools stop at the URL level. waymore goes a step further and pulls the actual archived content, which opens up possibilities that other tools just don't offer. You can grep through historical JavaScript files for API endpoints that were never linked anywhere. You can find developer comments that reveal internal paths or debug parameters. You can discover form fields and hidden inputs that might still be active on the live site.

• It handles rate limiting properly. The README points out that other tools often just stop when they hit rate limits, leaving you with incomplete results and no indication that anything went wrong. waymore deals with the rate limiting now in place across these sources, so you get more complete data and actually know when something's missing.

• It covers more sources than most alternatives. While waybackurls pulls from the Wayback Machine and one Common Crawl index, and gau adds Alien Vault, URLScan, Virus Total, and Intelligence X, waymore pulls from all of those plus GhostArchive. If you're trying to be thorough, that extra coverage matters.

• The author gives you practical advice. There's a warning in the README about a common mistake: passing a file of subdomains instead of just the domain. The tool is designed to handle the domain and find all subdomains itself. Doing it the wrong way is slower and can cause you to miss things. That kind of guidance is helpful when you're trying to get the most out of a tool.

• It's honest about its limitations. The README doesn't pretend this is a fast tool. It tells you upfront that it's meant for coverage. That's refreshing—you know what you're getting into before you start.

How to Try It

Getting started is straightforward. You'll need Python 3.7 or higher.

Install it in your default Python environment:

pip install waymore

Or install directly from the repository:

pip install git+https://github.com/xnl-h4ck3r/waymore.git -v

If you prefer an isolated environment, you can use pipx:

pipx install git+https://github.com/xnl-h4ck3r/waymore.git

One note from the README: if you already have a config.yml file, it won't be overwritten during installation. A new file called config.yml.NEW will be created instead. If you want the updated config, you'll need to remove the old one and rename the new file.

Once installed, you can run waymore against a domain. Just remember—pass the domain itself, not a list of subdomains. The tool will find the subdomains for you, and it'll be faster and more complete that way.

For full usage details and all the available arguments, check out the repository: https://github.com/xnl-h4ck3r/waymore

Final Thoughts

waymore isn't trying to be the fastest tool in your arsenal, and it's not pretending to be. It's for those times when you need to be thorough—when you're doing deep reconnaissance and you want to make sure you're not missing anything. If you're the kind of person who's already using waybackurls or gau, waymore is a natural next step when you need more than just a list of URLs. It's a tool built by someone who clearly understands the workflow and the gaps in existing solutions. Give it a shot on your next recon job, and see what you've been missing.

Follow @githubprojects for more developer tools and open source projects.