Giskard v3: Modular, Async-First Testing for Agentic Systems

Testing agentic systems is hard. You build a pipeline of LLM calls, tool integrations, and decision logic, and then you need to verify the whole thing actually works end to end. Unit tests don’t cut it. Traditional integration tests feel wrong. And debugging an agent that decides to hallucinate a function call? Good luck.

Giskard v3 just dropped, and it’s built for exactly this problem. It’s a testing framework designed from the ground up for modern, async-first Python agents. Think of it as Pytest, but for your complex reasoning loops.

What It Does

Giskard v3 lets you write test suites for AI agents and LLM pipelines. You define scenarios — like “user asks for a refund” or “agent gets a malformed JSON input” — and Giskard runs your agent through them, evaluating responses against rules you set. It supports custom metrics, LLM-as-a-judge evaluators, and automated regression detection.

The big shift in v3 is modularity. Instead of one monolithic test runner, you now compose test workflows from reusable blocks. Agents, evaluators, and data generators are all separate, pluggable components. And it’s fully async, so you can test concurrent agent behaviors without setup headaches.

Why It’s Cool

Three things stand out:

1. Async-first architecture. Most agent frameworks (LangGraph, CrewAI, custom asyncio pipelines) are async. Giskard v3 leans into that. You write test fixtures like async def my_agent_runner() and it works naturally. No hacks, no blocking calls.

2. Composable evaluators. You can chain LLM-based judges with deterministic checks. For example, “run the agent, then check its output has a valid JSON format and contains no harmful content” — all in one test.

3. Built for regressions. Once you define test scenarios, Giskard helps you detect when your agent’s behavior degrades after a prompt change or model swap. That’s the killer feature for real products.

Developers are using this to catch subtle issues like an agent suddenly ignoring instructions or a tool call returning the wrong type. It’s like having a QA team that reads every conversation.

How to Try It

pip install giskard

Or if you’re using Poetry/uv (it’s recommended):

uv add giskard

Write a quick test:

import giskard as gs
from giskard.llm import LLMProvider

# Define a simple agent interaction
async def test_refund_scenario():
    agent = my_custom_agent  # Your async agent
    scenario = gs.Scenario(
        input="I want a refund for order 12345",
        expected_behavior="handles refund politely"
    )
    
    # Run and evaluate
    result = await gs.run(agent, scenario)
    assert result.passed

Full docs and examples are in the repo: github.com/Giskard-AI/giskard

Final Thoughts

If you’re building agents that do real work — not just chatbots, but tools that call APIs, make decisions, or handle user data — you need structured testing. Giskard v3 feels like the right abstraction. It doesn’t try to do everything (it’s not a monitoring tool, not a framework) but it nails the core problem: making it easy to verify your agent behaves itself in a wide range of scenarios.

Start with one test. Then add a scenario for every bug you fix. Your future self will thank you.

Found on @githubprojects