Quickwit: The Search Engine That Skips the Heavy Lifting

Let’s be honest: standing up Elasticsearch to search through your logs feels like bringing a forklift to move a couch. It works, sure, but the memory footprint, the cluster management, the constant tuning—it’s a lot. For developers who just want to grep through terabytes of traces without babysitting a JVM heap, there’s a new tool in the shed.

Quickwit is an open-source search engine built specifically for logs and traces, but it doesn’t try to be a general-purpose database. Instead, it takes a radically different approach: it stores data on cheap object storage (like S3) and searches it directly there, without needing a local disk index. That changes the economics of log search completely.

What It Does

Quickwit ingests your log and trace data, indexes it, and writes that index to cloud storage. When you query, it reads the relevant index files straight from S3 (or compatible storage) and returns results in milliseconds. No local copies, no cluster of beefy nodes, no data replication for failover. Just a lightweight server that coordinates queries and lets the storage layer do the heavy lifting.

Under the hood, it uses a columnar layout (similar to Parquet) and a custom merge policy to keep index files small and query-friendly. It also supports distributed search out of the box, so you can throw multiple instances at a query and split the work across shards stored in your bucket.

Why It’s Cool

Most search engines treat storage like a precious resource to be carefully managed. Quickwit flips that: storage is cheap and infinite. That mindset unlocks some genuinely clever outcomes.

• Zero-disk architecture – You can run a Quickwit server on a tiny VM or even a container that gets killed and recreated. If it dies, no data loss, because everything lives in the bucket. It’s stateless from an indexing perspective.
• Pay for what you query – Since files are stored object-by-object, you only download the parts you need. A query for a specific trace ID touches fewer bytes than a full text search. That means your S3 egress bill actually scales with usage, not with data volume.
• Schema on read – You define a strict schema (or dynamic mode) and Quickwit uses it to prune files aggressively. If your query filters on a timestamp and a service name, it skips entire files that can’t match. This is old-school database partitioning, but applied to logs.
• First-class tracing – It understands trace IDs and spans natively. You can search across hundreds of millions of spans to find a single slow operation without pre-aggregating into a separate store.

It’s not a drop-in replacement for every search use case. For full-text search on user content, Elasticsearch or Typesense is probably better. But for high-volume, low-retention, structured log search, Quickwit feels like the tool that was actually designed for the job.

How to Try It

The quickest way to get a feel is to run the demo locally. You need Docker (or just download the binary).

# Pull the image
docker pull quickwit/quickwit

# Start a local server with an S3-compatible emulator built in
docker run -p 127.0.0.1:7280:7280 quickwit/quickwit --service metastore --service indexer --service searcher --data-dir /tmp/qwdata

Then index a sample log file:

# Create a simple index config
cat > my_index.yaml <<EOF
version: 0.6
index_id: my_logs
doc_mapping:
  field_mappings:
    - name: body
      type: text
    - name: timestamp
      type: i64
      fast: true
    - name: service
      type: text
      fast: true
EOF

# Ingest a file
quickwit index create --index-config my_index.yaml
quickwit index ingest --index my_logs --input /path/to/logs.json

Then hit the search API:

curl "http://localhost:7280/api/v1/my_logs/search?query=service:auth+AND+timestamp:[1685000000000+TO+1685001000000]"

The docs on GitHub have a full tutorial with realistic sample data, plus deployment recipes for Kubernetes if you want to stress test it.

Final Thoughts

I’ll be straight: Quickwit isn’t going to replace your favorite log viewer overnight. It has a learning curve, and the query syntax takes some getting used to if you’re coming from Lucene-based tools. But the core idea—search directly on object storage—is genuinely exciting. It makes logs feel like a data problem instead of an infrastructure problem.

For small teams with large log volumes, or for anyone tired of paying for 8GB of RAM just to index data that will live for 30 days, this is worth a serious look. It’s still young, but the architecture is solid and the project is moving fast. If you’re already on S3 or GCS, the integration story is basically free. Give it a spin on your next side project; you might retire your log cluster for good.

Found this useful? Follow @githubprojects for more open-source discoveries.