Cutting Agent Token Burn by 54%: Field Trials of Local ONNX Semantic Caching and Karpathy Surgical-Edit Rules
When a team of autonomous coding and research workers runs around the clock, token burn does not come from one giant prompt. It comes from quiet leaks: repeated variations of the same lookup question hitting paid cloud APIs, coding agents rewriting whole files when a small diff was needed, and retry loops caused by guessed library signatures. This week we took open-source patterns spotted in developer reels, wired two of them into our production router and agent rules, and measured what changed. Every number below is copied from our own logs, named in the Sources list at the end.
The interventions we tested
| Intervention | Upstream source | What we wired into production |
|---|---|---|
| 1. Local ONNX semantic cache | @datasciencebrain reel · Xenova/all-MiniLM-L6-v2 ONNX + SQLite WAL | our semantic cache in front of our frontier router (cache on): singleton ONNX Runtime 1.31.0 CPU embedder, variable-length WordPiece (no_padding()), a 3-stage lookup gate (policy_version filter, lazy TTL eviction, similarity >= 0.82) and invalidate-similar |
| 2. Karpathy surgical-edit rules | multica-ai/andrej-karpathy-skills (220,951 GitHub stars on 19 Oct 2026) | Distilled Think Before Coding, Simplicity First, Surgical Changes and Goal-Driven Execution into one rules file, one combined rules file |
| 3. Structural output rules | drona23/claude-token-efficient (6,096 stars) | Tested the base 7-line file alone (Test A), then folded its coding, analysis and agents profiles into the combined rules file (Test B) |
Live versioned documentation injection (upstash/context7) and cross-session memory (claude-mem) were named in the same batch of reels but were not wired or measured in this trial. The record lists three built artifacts, so those two wait for a later note.
Part 1: The local ONNX semantic cache (1.84 ms median embed, 6.435 ms end-to-end hit, 0 false hits at 0.82)
Most semantic caches fail in production for one of two reasons: the local embedding model pads every short question out to 256 tokens, or the similarity threshold is chosen by guesswork and near-miss questions get false hits. We implemented a singleton ONNX Runtime (1.31.0, CPUExecutionProvider) session over Xenova/all-MiniLM-L6-v2 with variable-length WordPiece tokenization (no_padding()) backed by SQLite in WAL mode, and ran a 60-query calibration sweep across 15 intents, 4 phrasings each.
| Operation | Prompt | Cosine similarity | Cosine distance | Lookup latency | Tokens |
|---|---|---|---|---|---|
| Cold call through our frontier router (cache on) (MISS, then Bedrock Claude Sonnet 4.6) | Karpathy principles question, policy live_bench_v1 | 0.0 | 1.0 | 7.138 ms cache lookup, then the cloud call | 34 in + 132 out billed |
| Exact repeat through our frontier router (cache on) (HIT) | same question | 1.0 | 0.0 | 6.435 ms | 0 (saved 34 in + 132 out) |
| Rephrased through our frontier router (cache on) (HIT) | List the four main principles of Andrej Karpathy's coding guidelines in 4 short bullet points. | 0.9495 | 0.0505 | 6.437 ms | 0 (saved 34 in + 132 out) |
| Demo store, then exact repeat (HIT) | Can I cancel my order? | 1.0 | 0.0 | 3.122 ms | 0 (saved 339 in + 23 out) |
Demo rephrased lookup (HIT at >= 0.82) | How do I cancel my order? | 0.8947 | 0.1053 | 2.978 ms | 0 (saved 339 in + 23 out) |
| Demo adversarial 1: subscription vs order (MISS) | Can I cancel my subscription? | 0.6933 | 0.3067 | 2.573 ms | routes to the LLM |
| Demo adversarial 2: COD fee vs availability (MISS) | Is there any extra charge for cash on delivery? vs Is cash on delivery available? | 0.8192 | 0.1808 | 3.682 ms | routes to the LLM |
| Targeted clause invalidation | invalidate-similar "cancel order policy", min similarity 0.50 | 0.7627 | 0.2373 | n/a | 1 row evicted |
Single-query ONNX CPU embedding (no_padding) | one short query | n/a | n/a | 1.84 ms median | 0 |
How do I cancel my order?, 0.6933 on Can I cancel my subscription?, and 0.8192 on the hardest wrong-intent pair in the 60-query set.Is cash on delivery available? (cod_limit) versus Is there any extra charge for cash on delivery? (cod_fee) at similarity 0.819. At threshold 0.80 the cache takes 11 hits with 1 false hit. At 0.82 (max distance 0.18) it takes 8 hits (13.3%) with 0 false hits. At 0.85 it takes 5 hits with 0 false hits. So 0.82 is the loosest threshold that never crosses intents on this set, and it is the shipped default. A note on the record: the narrative report of this trial printed a 31 of 60 (51.7%) hit rate at 0.82; the raw sweep log says 8 of 60, and this page prints the raw log.| Threshold | Max distance | Hits of 60 | Hit rate | LLM calls | False hits |
|---|---|---|---|---|---|
| 0.50 | 0.50 | 40 | 66.7% | 20 | 13 |
| 0.60 | 0.40 | 32 | 53.3% | 28 | 4 |
| 0.65 | 0.35 | 27 | 45.0% | 33 | 4 |
| 0.70 | 0.30 | 21 | 35.0% | 39 | 3 |
| 0.75 | 0.25 | 13 | 21.7% | 47 | 2 |
| 0.80 | 0.20 | 11 | 18.3% | 49 | 1 |
| 0.82 (default) | 0.18 | 8 | 13.3% | 52 | 0 |
| 0.85 | 0.15 | 5 | 8.3% | 55 | 0 |
| 0.90 | 0.10 | 3 | 5.0% | 57 | 0 |
| 0.95 | 0.05 | 0 | 0.0% | 60 | 0 |
Part 2: Why a 7-line "be concise" file did nothing on its own, and why the combined rules cut output tokens by 54.0%
Next we tested two prompt-efficiency repositories on AWS Bedrock (global.anthropic.claude-sonnet-4-6): drona23/claude-token-efficient and multica-ai/andrej-karpathy-skills. First, a note on the viral reel for andrej-karpathy-skills: the creator says the rules file stops Claude from hallucinating fake APIs. We read all 65 lines of the upstream file; the word API does not appear in it. Preventing API hallucination needs live documentation retrieval, not static behavioural rules.
Test A passed only the 7-line root rules file ("Thorough in reasoning, concise in output.") on one task (task-414): 1,136 output tokens at baseline, 1,153 with the file. The repository's own README says as much: "baselines on current models already exhibit 0% preamble, sycophancy". Test B passed our combined rules file (the 4 Karpathy rules plus the drona23 profile rules) as --system-file on three standard prompts, and then on a fourth TypeScript task:
| Task | Prompt | Baseline output tokens (words) | Combined rules output tokens (words) | Delta |
|---|---|---|---|---|
T1_async_await | Explain async/await in JavaScript. | 1,024 (436) | 526 (282) | -48.6% tokens |
T2_code_review | Review this code: for (let i = 0; i <= arr.length; i++) { console.log(arr[i]); } | 350 (177) | 83 (30) | -76.3% tokens |
T3_rest_api | What is a REST API? | 332 (213) | 176 (108) | -47.0% tokens |
| Total, 3 prompts | 1,706 (826) | 785 (420) | -54.0% tokens, -49.2% words | |
karpathy_ab_test (TypeScript buildUrl with optional query parameters) | same prompt, two modes | 402 tokens, 98 lines, 7.19 s | 155 tokens, 21 lines, 3.99 s | -61.4% tokens, -44.5% time |
task-414)? Because those lines ban conversational pleasantries that Claude Sonnet 4.6 already omits by default. The bloat in frontier coding output is architectural: speculative helper classes, unrequested error wrappers, and multi-paragraph walkthroughs after the code. When we combined Karpathy's Simplicity First and Surgical Changes with drona23's structural rules ("Return code first. Explanation after, only if non-obvious.", "State the bug. Show the fix. Stop."), output dropped from 1,706 to 785 tokens on the 3 prompts and from 402 to 155 tokens on the TypeScript task. On the code review the answer went from 177 words to 30 words, the same 30-word shape the drona23 README reports for its own benchmark.What we shipped
Both mechanisms are active in our tools. Every call through our frontier router (cache on) checks our semantic cache (threshold 0.82, TTL 86,400 s, namespaced by policy_version, with a hash of any system prompt folded into the namespace) before touching a cloud endpoint, and coding dispatches can pass the combined rules file as --system-file. The one Karpathy rule we leave off unattended overnight workers is the stop-and-ask rule, so they do not stall waiting for a human.
Sources
All figures above are copied from our own lab records of 19 October 2026: the token-limits integration report (repository stars, Test A, demo latencies and token savings, invalidation); the A/B token benchmark (T1 to T3 tokens and words); the live cache test (cold, repeat and rephrased calls); the semantic cache sweep (the 60 queries and the closest wrong pair); the live frontier benchmark (the TypeScript task); and the rules file used in the test (65 lines, no occurrence of the word API). Corrections welcome.