How does Nadir's classifier decide which model to use?

It works in three stages. A sentence embedding model encodes the prompt, the embedding is compared by cosine similarity against learned centroids for simple, medium, and complex tiers, and the resulting tier maps to a model selection strategy. The post puts total classification overhead at about 50 to 100ms.

Does the classifier need a GPU?

No. The post says the embedding model runs locally on CPU, is 22MB, and loads in about 2 seconds on startup. The embedding is used only for classification, not generation, so a small model is enough to separate prompts like "what is 2+2" from "design a distributed system".

What does the classifier output?

It outputs tier probabilities, such as simple=0.99, medium=0.003, complex=0.003, a score from 0.0 (simple) to 1.0 (complex), and a tier name. Scores below 0.2 are simple, 0.2 to 0.7 medium, and above 0.7 complex.

How are models chosen within a tier?

The ranker filters available models by the user's allowed list, then sorts by cost within the tier. Simple prompts go to the cheapest available model, medium to a mid-tier model, and complex prompts to a premium model.

How our binary classifier routes prompts in 50ms

Under the hood: DistilBERT embeddings, centroid matching, and a 3-tier system that decides if your prompt needs a premium model or not.

Architecture overview

Nadir's binary classifier has three stages:

Embedding - DistilBERT encodes the prompt into a 384-dim vector (~10ms)
Centroid matching - Compare against learned centroids for simple/medium/complex (~1ms)
Tier assignment - Map complexity tier to model selection (~0ms)

Total: ~50-100ms classification overhead.

Stage 1: Embedding

We use sentence-transformers/all-MiniLM-L6-v2 to encode prompts. It runs locally on CPU, no GPU needed. The model is 22MB and loads in ~2 seconds on startup.

Why not a larger model? Because the embedding is just for classification, not generation. MiniLM-L6 captures enough semantic signal to distinguish "what is 2+2" from "design a distributed system" reliably.

Stage 2: Centroid matching

During training, we compute centroid embeddings for each complexity tier from labeled examples. At inference time, we compute cosine similarity between the prompt embedding and each centroid.

The classifier outputs:

Tier probabilities: e.g., simple=0.99, medium=0.003, complex=0.003
Confidence score: 0.0 (simple) to 1.0 (complex)
Tier name: simple / medium / complex

Stage 3: Model selection

The tier maps to a model selection strategy:

Tier	Strategy	Example models
Simple (score < 0.2)	Cheapest available	Gemini Flash, GPT-3.5
Medium (0.2 - 0.7)	Mid-tier	GPT-4o-mini, Haiku
Complex (score > 0.7)	Premium (benchmark model)	Sonnet, GPT-4o, Opus

The ranker filters available models by the user's allowed list, then sorts by cost within the tier.

Accuracy

The binary classifier alone hit routing accuracy in the high 80s on small held-out evals. On RouterBench held-out (n=11,420), the prompt-only classifier preserved 96.6% of the always-Opus reference. A separate reference-assisted cascade reached 98.3%, but its verifier had the expensive answer unavailable to production. That result is a research ceiling, not deployed performance.

What's next

We're training a confidence-aware analyzer that escalates uncertain classifications to a secondary check. If the primary classifier scores between 0.3-0.7, it runs a lightweight keyword analysis before committing to a tier. This should push accuracy above 98%.