smol.router

Introduction.

smol.router is a small, calibrated classifier that sends each prompt to the cheapest model likely to answer it correctly, then escalates when reality disagrees.

Rather than using large language models to route traffic—which defeats the purpose of cost-saving—smol.router uses a lightweight linear classifier trained directly on your benchmarked data. It runs locally without heavy dependencies, adding less than a millisecond of overhead.

Core Principles

• Predict capability: A hashed-feature linear model estimates the minimum tier (low, medium, high) required by the prompt.
• Price the risk: A deterministic policy bounds the decision, balancing provider cost against the real damage of under-routing.
• Check the answer: Validate provider output at the application edge and automatically step up a tier if it fails.

Why this approach?

Intelligence is expensive. You should use exactly enough for each request. Hand-written prompt lengths or regex matching are fragile, while LLM-based routing is slow and costly.

By benchmark-testing your actual models to see which tier passes your evaluators, smol.router learns the underlying distribution of your prompt capability requirements.

Overview

Next Steps

Get started by setting up the router locally, explore the Python API, or dive into how to construct a robust benchmark dataset.

• Quickstart: Install, train, and run the router
• Python SDK: Integrate routing into your application code
• Training: Build a benchmark dataset and evaluate model decisions