Crawly: Build Elixir Spiders for Web Scraping and Data Mining

If you've ever tried web scraping in Elixir, you know it can feel a bit like wrestling with a distributed system when all you wanted was a simple spider. Most scraping tools are written in Python or Node, but if you're building an Elixir app and need to extract structured data from websites, pulling in a foreign language stack can feel heavy.

Enter Crawly. It's an Elixir-native framework for writing web spiders that feel right at home in the BEAM ecosystem. No mix of Python dependencies or awkward interop — just pure Elixir, backed by OTP.

What It Does

Crawly gives you a declarative way to define spiders that crawl web pages, extract data, and pipe it into whatever pipeline you want. You define a module with a parse/1 callback, tell it which URLs to start with, and decide how to handle the response. Under the hood, it uses HTTPoison for requests, Floki for HTML parsing, and OTP for concurrency and resilience.

It's essentially Scrapy for Elixir, but built with processes, supervisors, and all the fault-tolerance goodies you'd expect from the BEAM.

Why It's Cool

The concurrency story is the standout. Crawly spawns worker processes for each request, so you're not waiting on slow sites to block your entire spider. If a request times out or returns a 500, the process dies cleanly, and the supervisor restarts it. That's not something you get for free in most scraping tools.

You can also swap out the HTTP client, the storage backend, or the middleware pipeline with little ceremony. Need to persist scraped data into a database instead of a file? Plug in a custom Storage module. Want to use a proxy pool? There's middleware for that.

Another win: the pipeline pattern. You define transformations on scraped items (like cleaning up whitespace or converting strings to integers) using simple Elixir pipes. It stays readable and testable.

How to Try It

Add Crawly to your mix.exs:

defp deps do
  [
    {:crawly, "~> 0.13"}
  ]
end

Then define a minimal spider:

defmodule MyApp.MyFirstSpider do
  use Crawly.Spider

  @impl Crawly.Spider
  def base_url(), do: "https://example.com"

  @impl Crawly.Spider
  def init() do
    [
      start_urls: ["https://example.com/items"]
    ]
  end

  @impl Crawly.Spider
  def parse(response) do
    {:ok, document} = Floki.parse_document(response.body)
    titles = Floki.find(document, "h2.title") |> Enum.map(&Floki.text/1)

    items =
      Enum.map(titles, fn title ->
        %Crawly.ParsedItem{
          item: %{title: title},
          id: title
        }
      end)

    %Crawly.ParsedItemCollection{items: items}
  end
end

Run it with:

Crawly.Engine.start_spider(MyApp.MyFirstSpider)

For a full demo, check the GitHub repo – there are examples for handling pagination, AJAX-loaded content, and custom pipelines.

Final Thoughts

Crawly isn't trying to replace your entire data mining stack. It's a focused, idiomatic tool for when you need to scrape a few hundred pages and want to stay inside Elixir's ecosystem. The concurrency model makes it efficient, and the OTP underpinnings mean it won't fall over the moment a site is slow.

If you're already building things in Elixir and need to bring in external data, Crawly is worth a look. It's not flashy, but it gets the job done without pulling in a different language or runtime. Sometimes that's exactly what you need.

— @githubprojects