WaterCrawl: A Self-Hosted Web Crawler and Search Engine You Can Actually Run Yourself
You've probably needed to pull data off a website at some point—maybe for a side project, a research task, or feeding content into an AI pipeline. And you've probably discovered that the tools for the job are either paid APIs with usage limits or a pile of scripts you have to babysit. WaterCrawl is an open-source project that bundles crawling, scraping, and search into a single self-hosted application, so you can run the whole thing on your own infrastructure.
What It Does
WaterCrawl is a web application built on Python, Django, Scrapy, and Celery. Those pieces tell you most of the story: Django handles the web application and REST API, Scrapy does the crawling and scraping, and Celery manages asynchronous task processing so long-running crawls don't block anything else.
The core functionality splits into two related jobs. First, it crawls and scrapes websites with configurable options for depth, speed, and targeting specific content. Second, it provides a search engine that finds relevant content across the web, with multiple search depths (basic, advanced, and ultimate) and multi-language support with country-specific targeting. Results can be monitored in real time via Server-Sent Events, and there's a REST API with OpenAPI documentation along with client libraries. It also ships with integrations for platforms like Dify and N8N.
Why It's Cool
It's self-hosted, which means your data stays yours. If you're crawling content for an AI pipeline or a research project, you might not love the idea of routing everything through a third-party service. Running it locally gives you full control over what gets crawled and where the results go.
The architecture is boring in the best way. Django, Scrapy, and Celery are all mature, well-documented tools with large communities. That means when something breaks (and it will), you're debugging against a stack that thousands of people have already written about. No exotic dependencies, no proprietary black boxes.
Asynchronous processing with real-time feedback. Crawls and searches report progress through Server-Sent Events, so you can actually watch what's happening instead of polling an endpoint and hoping. For longer crawls, this matters more than you'd think.
Search depths as a knob. The basic, advanced, and ultimate search depth options give you a way to trade thoroughness for speed and cost. It's a small design decision, but it maps well to how people actually use search—sometimes you need a quick answer, sometimes you need to be exhaustive.
An API-first design with ecosystem integrations. The OpenAPI-documented REST API plus integrations for Dify and N8N suggests this was built to slot into automation and AI workflows rather than stand alone. If you're building something that needs web data as an input, that's a practical starting point.
How to Try It
The fastest path is Docker. Clone the repo, copy the environment file, and bring it up:
git clone https://github.com/watercrawl/watercrawl.git
cd watercrawl
cd docker
cp .env.example .env
docker compose up -d
Then open http://localhost and you should see the application.
One important caveat from the README: if you're deploying on a domain or IP other than localhost, you must update the MinIO configuration in your .env file. Specifically, change MINIO_EXTERNAL_ENDPOINT from localhost to your actual domain, and update MINIO_BROWSER_REDIRECT_URL and MINIO_SERVER_URL to match. Skip this and file uploads and downloads will break. The README points to a DEPLOYMENT.md for the full deployment guide, which is worth reading before you put this anywhere production-adjacent.
If you want to contribute or run it for development, there's a separate CONTRIBUTING.md with instructions.
The repository is at github.com/watercrawl/watercrawl.
Final Thoughts
WaterCrawl is best suited for developers who need web crawling and search as a component of something larger—an AI pipeline, an automation workflow, a data collection project—and who'd rather own the infrastructure than rent it. The stack is approachable, the deployment is containerized, and the API-first design means it's meant to be integrated rather than just clicked around in. If that describes your situation, it's worth spinning up locally and seeing how it fits. The fact that it's open source and self-hosted means the worst case is you spend an afternoon with Docker and learn something.
Follow @githubprojects for more developer tools and open source projects.