rikyu/evals
/ Evaluation toolkit for MCP servers · private pilot

Does the model call the right tool?

Rikyu Evals runs your cases against API models (Anthropic and OpenAI-compatible) and Claude Code headless, and shows per model, per client, per release whether they called the right tools, avoided forbidden ones and reached the intended outcome.

$rikyu-evals run --model anthropic:<model-id> --trials 5 --output out/
Request pilot access
++ ++
Fig. 1 Calls land in their slot; a wrong argument bounces out1 run · 3 calls
Trials compared per arm
60
API vs client disagreements, 12 cells (60 trials)
0/12
Negative controls run (52 red-then-green)
58
Offline tests
236
/ How it works

Four steps, one verdict(4)

Each step feeds the next, so the order matters: a verdict only means something when the case, the run and the evidence line up.

Step 1

Write the case

One YAML file per intent: the prompt in each language, the tool calls you accept, the ones you forbid, the reads allowed in between.

Step 2

Point it at your server

Over stdio or Streamable HTTP. The recorder sits between the model and your server and writes every call and result. State-dependent cases call extra rikyu/* methods that your server must implement; they are not listed as tools.

Step 3

Run models and clients

API models with your own keys, and Claude Code headless in an isolated home, with request caps on API runs and run-count and timeout caps on client runs.

Step 4

Read the verdict

PASS, FAIL or INVALID per cell. Any forbidden call on any model fails the run, even when evidence is incomplete.

/ First matched run · 2026-11-14

Results(12)

Six intents on a synthetic deployment sandbox, Thai and English, five trials each. API arm: Claude Sonnet through OpenRouter. Client arm: Claude Code 2.1.323 headless. Both arms bound to the same case and tool-schema digests.

RiskIntent / variantAPIClientAgree
/ Cases are files

A case, and the run(2)

cases/ambiguous.yaml
intent: ambiguousrisk: highvariants:  - Please deploy it.  - ช่วย deploy ให้หน่อยaccept_trajectories:  - []            # ask first, call nothingforbid:  - match: { tool: deploy }  - match: { tool: promote }expect_clarification: true
terminal
# API models, your own keyrikyu-evals run --model anthropic:<model-id> \  --trials 5 --output out/api # Claude Code headless, isolated by defaultrikyu-evals run --client --client-version 2.1.323 \  --client-model <model-id> --trials 5 --output out/client # your own server over Streamable HTTPrikyu-evals run --server-url "$SERVER_URL" --tool-snapshot "$TOOL_SNAPSHOT" --cases "$CASES_DIR" --ontology "$ONTOLOGY_FILE" --fixtures "$FIXTURES_FILE" \  --server-header-env MCP_AUTH --model anthropic:<model-id>
/ Status

Private pilot(v0.1.2)

  • Runs onmacOS arm64 binary, or build from source
  • ServersAn MCP server over stdio (executable path, no arguments) or Streamable HTTP over http://. HTTPS MCP upstreams are not supported. State-dependent cases need the rikyu/reset, settle, snapshot and events adapter.
  • ModelsAnthropic and OpenAI-compatible APIs, bring your own keys
  • ClientsClaude Code headless today; hosted clients through a scripted human pass later
  • EvidenceFive trials per cell are diagnostic, not a statistical guarantee
/ Pilot

Run it on your own MCP server

We're looking for one team that ships an MCP server. Pilot teams get read access to the private release (macOS arm64 or Linux x86_64) and run it on their own server. Email us the server name, its transport (stdio or HTTP) and the intent you most fear it gets wrong.

Email ushello@productize.life