The result, first Context selection removes about a third of the input tokens on retrieval-style requests without costing you answers. Measured on a public benchmark, every answer graded blind by an independent judge against the known-correct answer: 31.8% to 34.1% of input tokens removed, depending on context size. 100% of answer paragraphs retained across 350+ judged pairs. Not one question lost the passage it needed. Net accuracy unchanged, and better on the largest contexts: 95.9% with selection against 94.8% without. That last number is the one worth pausing on, because it is not a rounding artifact. Long noisy context degrades models by itself. Cutting the passages a request does not need removes distractors along with the tokens, so on exactly the bloated-retrieval traffic where the savings are largest, the answers get slightly better at the same time. The rest of this post is how we got there, starting with two obvious ideas that failed. Open source. The selection algorithm this post benchmarks now ships as a standalone Python package: pip install barber-llm (import name barber). Zero required dependencies, eval harness and judge prompt included, MIT licensed. Code at github.com/NadirRouter/barber. The question we started with You can shrink an image and the model sees the same thing. You can play audio at 2x and transcription accuracy barely moves. Both are billed by signal size, so degrading the signal cuts the bill. So: can you do that to text? Remove 20% of the letters in each word? Run it through a compression algorithm? We tested the obvious ideas before building anything, because the cheapest experiment is the one that stops you building the wrong thing. What does not work, with numbers Removing letters makes text more expensive The intuition: models read mangled text fine (they do), so chop letters and pay less. The problem is the tokenizer. BPE gives many common whole words a single token. customer, carrying its leading space, is 1 token. Chop it to custome and it is no longer a word the tokenizer knows, so it splits into 2. You deleted a letter and doubled the cost of the word. It is a tax, not a law: sweeping the o200k vocabulary, about half of single-token words split this way and the rest survive intact. Either way the savings never arrive. | Scheme | o200k | cl100k | | --- | --- | --- | | Drop last 20% of each word | 1.34x | 1.40x | | Drop random 20% of letters | 1.57x | 1.62x | | Remove interior vowels | 1.68x | 1.69x | Aggregated over prose, instructions, code, and a RAG passage, on both current byte-level BPE tokenizers. Every letter-removal scheme costs more, not less. The model can still read it. You just pay extra for the privilege. Classic compression runs backwards On a 756-byte sample of varied prose, gzip shrinks the bytes to 0.61x. But a model cannot read gzip output, so you base64 it, and that round trip costs 3.20x the original tokens. bzip2 lands at 3.30x, lzma at 3.71x. Bytes are not tokens. The usual explanation for this, that compressed data is maximum entropy with nothing left for BPE to merge, is wrong on inspection. Base64 of gzip still contains 560 distinct 2-grams, and 43% of its tokens still span more than one character. BPE merges fine. The problem is simpler: base64 expands by 4/3, and the merges it finds are short, so you end up getting 1.46 characters per token on a stream that used to get 5.68. Compression ratios are also length dependent, which is why the corpus size is quoted above. Compression only wins on pathologically repetitive text, and the honest fix for repetition is to delete the repeats, which is exactly what the rest of this post is about. The underlying reason both fail is not that text is incompressible. It plainly is not: run bzip2 or lzma over the same bytes and they still beat what BPE achieves. The reason is that the tokenizer is already a compression algorithm sitting between you and the meter, it is the one whose output you are billed for, and it offers no quality dial. An image has a resolution knob and audio has a playback rate, both of which trade fidelity for size continuously. Text has no equivalent. Every character-level edit either changes the meaning or, as measured above, lands you on a worse path through the same merge table. Images and audio are billed by the signal. Text is billed by the code. What works: drop whole chunks, not letters The only operation that reliably reduces the bill is removing whole pieces of context the request does not need. Selection removes tokens, the billing unit. Compression rearranges characters below it. Context selection in Nadir works like this, per request: Take the latest user message as the query. Never prune it. Find large retrieved-context and tool-output blocks. The system prompt is never touched. Split each block into chunks at natural boundaries. Embed chunks and query, score each chunk by relevance to the query. Keep the top fraction (the keep knob), plus three guards: lead and tail chunks always survive, safety and policy language is never dropped, and any chunk containing a rare term from the question is pinned. That last guard is why the answer paragraph survives even aggressive budgets. Dropped runs collapse into a one-line marker. Nothing is rewritten or summarized. Chunks survive verbatim or vanish. None of this is a new idea. Query-conditioned selection is well-trodden research territory. What we had not seen published is the part that matters for a bill you are accountable for: does it harm the answers, measured properly? How we measured harm Token counts cannot tell you whether the model lost the answer. Input similarity cannot either. The only honest test is comparing the responses. So we ran a paired A/B: every question answered twice, once with full context and once with selected context, then graded blind by an independent judge model against the known-correct answer. Setup. Dataset: HotpotQA (distractor config, Yang et al. 2018, CC BY-SA 4.0), public and reproducible. Each item has a question, a short gold answer, and typically 10 Wikipedia paragraphs of which 2 contain the answer. Native items are small, about 1.3K tokens, so we scale the haystack by padding with real paragraphs drawn from other items in the set, targeting roughly 4K and 12K tokens per request. Generator and judge: MiniMax M3 over the OpenAI compatible API, temperature 0. The judge grades both answers against the reference, blind to which used selection, with answer order randomized. The metric that matters is the regression rate: how often selection turned a right answer wrong. The baseline run surfaced something worth knowing on its own: with no selection at all, generator accuracy fell from 100% on native-size contexts to 88% on the padded 12K ones. Same questions, just more distractors. Long noisy context hurts models by itself, which becomes relevant below. The encoder is everything Our first run used a cheap lexical scorer (word overlap). It saved 71% of tokens and failed the quality gate badly: 94% answer-paragraph retention, 12.1% regression rate. Word overlap cannot tell that "how do I get my money back" matches a paragraph about refunds. Swapping in a semantic embedding model fixed it: | Scorer | Answer retention | Regression | Tokens saved | | --- | --- | --- | --- | | Lexical (word overlap) | 94% | 12.1% | 71% | | Semantic embeddings | 100% | 1.4% | 35% | The scorer is the whole ballgame, so we tested two encoders head to head on the same question slice: BAAI/bge-small-en-v1.5 and mmbert-embed-32k-2d-matryoshka, a fine-tune of jhu-clsp/mmBERT-base from the JHU Center for Language and Speech Processing. They tied on quality. We ship bge as the default and support the mmbert model through a served endpoint for the case bge cannot cover: a single chunk past 512 tokens, which bge silently truncates and mmbert's 32K window reads whole. The gate: 400 questions, two context sizes The ship decision ran on roughly 400 fresh questions at keep=0.6, medium and large contexts, about 350 judged pairs after filtering: | | Medium (~4K) | Large (~12K) | | --- | --- | --- | | Tokens saved | 31.8% | 34.1% | | Answer-paragraph retention | 100% | 100% | | Full-context accuracy | 97.2% | 94.8% | | Selected-context accuracy | 96.0% | 95.9% | | Regression rate | 4.0% raw, ~2.3% after judge artifacts | 2.3% | Three things stand out. Selection never dropped an answer. Across 350+ judged pairs, every paragraph needed to answer the question survived selection. The rare-term pinning guard did its job at every budget we tested. On large contexts, selection made answers better. 95.9% vs 94.8%, while removing a third of the tokens. This is the baseline finding paying off: the model was tripping over distractors, and selection removed them. Fewer tokens, better answers, on exactly the bloated-retrieval traffic where the savings are largest. The regressions are churn, not decay. When we count both directions, selection flipped about as many wrong answers right as right answers wrong (on the large run: 6 improvements, 4 regressions). The failures are behavioral: with a thinner haystack the model occasionally hedges or refuses even though the answer paragraph is present. Net accuracy across both runs is within noise of zero change. What ships Context selection ships as an opt-in option in Nadir's context optimization pipeline. Defaults from the benchmark: keep=0.6, semantic scorer, all guards on. Every block is scored against the question that follows it, not the latest one, so the decision is a pure function of the block and its own query and reproduces byte-identically as a conversation grows. That is what keeps a provider prompt cache hitting turn after turn. And it composes with the existing safe and aggressive transforms, which attack formatting waste while selection attacks relevance waste. Because your workload is not HotpotQA, the same harness that produced this post runs as a shadow evaluation against real traffic: sample requests, answer them both ways, judge the pair. The regression rate you act on is measured on your requests, not ours. It rolls out log-only first, then shadow, then as an opt-in flag, and a customer whose measured rate crosses the threshold gets selection turned off rather than a footnote. How to read these numbers Every benchmark is a claim about the world made from a sample of it. Here is where ours is weaker than the table above makes it look, collected in one place so you do not have to go hunting. We missed our own bar. We pre-registered a target of under 1 to 2% regression before running anything, and the medium run came in at 4.0% raw. The churn analysis is why we shipped regardless: a regression matched by an improvement elsewhere is movement, not decay. But the bar was set first and we missed it, and the honest response is the shadow eval and the auto-disable trigger, not a quietly softened bar. The keep budget is fixed, not tier-aware. It is tempting to prune harder for a request routed to a strong model, since the model can fill more gaps from its weights. We built that and then took it out, because a per-request keep budget is incompatible with a per-conversation stable prefix: the same history block would be re-decided the moment a conversation routed to a different tier, and re-billed at cache-write rates. Both gate runs used a fixed keep=0.6, so that is the only setting with measured regression behind it, and it is what ships. Tier-aware pruning waits for a per-conversation place to pin the decision. Our haystacks are easier than HotpotQA's reputation suggests. The questions are genuinely multi-hop, but the padding that scales a 1.3K-token item up to 12K is drawn from unrelated articles. So most of what selection discriminates against at the large size is off-topic Wikipedia, not the near-topic distractors HotpotQA ships natively. Telling a refund passage from an article about photosynthesis is easier than telling it from another refund passage. Read the results as evidence that selection does not break answers, not as a worst case. The scorer comparison is not perfectly controlled. Both runs used the same keep budget, but a relevance floor also drops any chunk scoring far below the best one, and lexical scores collapse to near zero off-topic while cosine similarities stay bunched together. The lexical run therefore pruned about twice as hard as its budget asked for. Part of that 12.1% is the scorer being wrong and part is it simply cutting more. The direction is not in doubt; the size of the gap is not a clean measurement of scorer quality alone. The encoder's context window is headroom, not a proven edge. The mmbert model's 32K is YaRN-extended from a native 8K, and its card publishes no long-context retrieval benchmark. We rely on it to avoid truncating chunks, which is a capacity argument. We have not independently stress-tested retrieval quality out at the top of that window. One judge, one generator. Every answer was written and graded by MiniMax M3. Blind grading with randomized answer order controls position bias, and reference-based grading against the gold answer controls for fluent-but-wrong responses. It does not control for a model's blind spots being correlated with its own. Thank you, MiniMax The entire evaluation, roughly 2,000 model calls as generator and judge across baseline, encoder comparison, and both gates, ran on MiniMax M3 through their OpenAI compatible API, for well under $20 of credit. M3 held up as a blind grader across the whole run: consistent, cheap, and fast enough that we could afford to judge every answer twice rather than spot-check. Genuine thanks to the MiniMax team for making evaluation at this rigor this affordable. The takeaway You cannot speed up text the way you speed up audio. The tokenizer already compressed it, and every character-level trick we measured costs more than it saves. What you can do is stop sending passages the request does not need, which is the one operation that removes the thing you are actually billed for. The reason this belongs in a router rather than in your application code is that the router is where it composes and where it is watched. It runs after model selection and before dispatch, so it sees the same request the router priced and the response the model returned, which is what lets the savings and the regression rate land on one dashboard against real traffic. Selection on its own is a library. Selection with the routing decision beside it, measured on your own requests and reversible the moment quality moves, is the part you would rather not build yourself. Point your base URL at Nadir, flip the flag, and watch the savings and the regression rate on the same dashboard. If the quality number moves, turn it off. That is the whole pitch. Run it yourself. The library and the harness behind these numbers are open source: pip install barber-llm, then barber-eval --n 200 --keep 0.6 --size large. Source at github.com/NadirRouter/barber.