Stop Guessing Whether Your LLM App Actually Works

You've built a RAG pipeline, hooked it up to LangChain, and watched it return answers that seem right. But "seems right" isn't a metric. How do you know if your LLM app is actually performing well? How do you catch regressions when you change your prompt or swap your embedding model? If you've found yourself manually eyeballing responses and hoping for the best, you're not alone—and that's exactly the problem Ragas is built to solve.

Ragas is an open-source toolkit for evaluating LLM applications with objective metrics, auto-generated test data, and production-aware feedback loops. Instead of subjective gut checks, you get measurable, repeatable evaluation workflows.

What It Does

Ragas sits on top of your LLM app and gives you two core capabilities: evaluation metrics and test data generation.

For evaluation, it provides a collection of pre-built metrics that work with both LLM-based and traditional scoring methods. You can use these out of the box, or you can define custom metrics. The README shows a DiscreteMetric example where you specify allowed values (like "accurate" or "inaccurate") and a prompt that tells the LLM how to evaluate a response. It's a clean abstraction—you're essentially teaching an LLM to grade your app's output on your terms.

For test data, Ragas can automatically generate comprehensive test datasets when you don't have one ready. This is huge, because building a good eval set by hand is tedious and often biased toward what you already expect your app to do well on. Ragas aims to cover a wider range of scenarios than you'd probably think to test yourself.

The project is Python-based, installable via pip, and integrates with popular frameworks like LangChain and observability tools. It also includes a CLI (ragas quickstart) that scaffolds a complete RAG evaluation project for you, so you don't have to wire everything together from scratch.

Why It's Cool

The thing that stands out about Ragas is that it's not just a metrics library—it's a full evaluation workflow tool. Here's what makes it interesting:

• It gives you a number instead of a vibe. LLM evaluation is notoriously squishy. Ragas pushes toward objective, repeatable metrics that you can track over time. That means you can actually compare versions of your app and know whether a change helped or hurt.

• Custom metrics are first-class citizens. The DiscreteMetric API is elegant. You define the scale, write a prompt, and Ragas handles the evaluation. You're not locked into whatever metrics the library ships with—you can grade your app on the criteria that matter for your specific use case.

• Test data generation solves the cold-start problem. If you're just starting to evaluate your app, you probably don't have a solid test set. Ragas generates production-aligned test data for you, which is a much better starting point than writing a handful of examples that cover only the happy path.

• The quickstart CLI removes friction. The ragas quickstart command creates a project structure for you. There are templates for RAG evaluation now, with agent evaluation, LLM benchmarking, prompt evaluation, and workflow evaluation on the roadmap. It's clear the project is thinking about evaluation holistically, not just for one use case.

• It's built for iterative improvement. The mention of building feedback loops from production data is the most practical feature here. Your eval suite shouldn't be static—it should evolve as you learn what your users actually ask and where your app actually fails.

How to Try It

Getting started is straightforward. First, install the package:

pip install ragas

Or install directly from source:

pip install git+https://github.com/vibrantlabsai/ragas

The fastest path to a working setup is the quickstart CLI:

# List available templates
ragas quickstart

# Create a RAG evaluation project
ragas quickstart rag_eval

# Specify where you want to create it
ragas quickstart rag_eval -o ./my-project

Once you have a project, you can define custom metrics. Here's the pattern from the README:

import asyncio
from openai import AsyncOpenAI
from ragas.metrics import DiscreteMetric
from ragas.llms import llm_factory

# Setup your LLM
client = AsyncOpenAI()
llm = llm_factory("gpt-4o", client=client)

# Create a custom aspect evaluator
metric = DiscreteMetric(
    name="summary_accuracy",
    allowed_values=["accurate", "inaccurate"],
    prompt="""Evaluate if the summary is accurate and captures key information.

Response: {response}

Answer with only 'accurate' or 'inaccurate'."""
)

From there, you can evaluate responses against your metric and start building a feedback loop with production data. For more details, check out the repository and the documentation.

Final Thoughts

Ragas is for anyone who's building LLM apps beyond a quick prototype—if you're shipping something to users, you need to know whether it's working, and Ragas gives you a structured way to find out. It's especially useful if you're using RAG or planning to build more complex agentic workflows, since those are exactly the systems that get hard to evaluate as they grow.

The project is still evolving—agent evaluation and workflow evaluation are listed as "coming soon"—but the core is already practical. You can start using it today to get real metrics on your app. And honestly, having a number to track is better than crossing your fingers and hoping your prompt tweak didn't break something.

Follow @githubprojects for more developer tools and open source projects.