Training & evaluation

Difficulty labels are not ground truth.

A prompt “looks hard” is not a useful target. Run the task through every candidate model, score the outputs, and label it with the cheapest tier that actually passes.

Benchmark every tier

Use the provider-backed runner when your models are reachable through an OpenAI-compatible chat completions API.

cp router.example.json router.json
cp examples/benchmark.config.example.json benchmark.config.json
cp examples/benchmark_prompts.example.jsonl benchmark_prompts.jsonl

OPENAI_API_KEY=... PYTHONPATH=src python3 examples/benchmark_pipeline.py \
  --config benchmark.config.json

Set the three model IDs in router.json, then point benchmark.config.json at your prompt file and output path.

{
  "router_config": "router.json",
  "prompts": "benchmark_prompts.jsonl",
  "output": "data/benchmark.jsonl",
  "provider": {
    "base_url": "https://api.openai.com/v1",
    "api_key_env": "OPENAI_API_KEY"
  },
  "judge": { "model": "gpt-4.1-mini" }
}

Each prompt row supplies the task, the judge reference or rubric, and an optional group for leakage-safe splitting.

{"prompt":"Implement and test an API rate limiter.","reference":"Must include token bucket logic, concurrency safety, and tests.","group":"rate-limit-1"}

What gets recorded

• A normalized 0–1 score for low, medium, and high.
• Per-tier latency and provider failure details.
• Task group and weight for leakage-safe splitting.
• The original prompt, ready for the training loader.

Train and calibrate

smol-router train data/benchmark.jsonl \
    --acceptable-score 0.8 \
    --output model.json

smol-router evaluate data/locked-test.jsonl --model model.json

Training uses weighted softmax regression over deterministic hashed features, then fits scalar temperature on held-out examples. Related task groups never cross the automatic split.

Read the right metrics

• Under-route rate: how often the policy selected an incapable tier.
• Mean shortfall: how many capability levels were missed.
• Regret: realized cost compared with the cheapest sufficient model.
• Calibration: log loss and Brier score for probability quality.
• Savings: selected cost compared with always choosing high.

Production loop

1. Sample representative prompts from real traffic.
2. Score all candidate models with task-specific checks or validated review.
3. Lock an untouched test set and define a maximum under-route rate.
4. Tune costs and penalties from actual price, latency, and failure impact.
5. Re-benchmark whenever an underlying model changes.