Precision on Demand: Escalating Weight Bits at Uncertain Decoding Steps in a Small Local Model

Mohammed Nayeem Khan, Logesh L, Aliyas Ahmad Misri

Precision on Demand: Escalating Weight Bits at Uncertain Decoding Steps in a Small Local Model: 4 upvotes on Hugging Face Daily Papers, #10 of 21 papers on 2026-11-12. Day-by-day upvote history.

Upvotes over time

It had 4 upvotes by its first day, 4 upvotes by the next day, 4 upvotes by day 3, 4 upvotes by day 7, and has 4 now. It ranked #83 of 133 papers in its week and #244 of 340 in its month.

Local decoding usually fixes one weight precision for the whole generation. We study a different allocation: store 8-bit weights, decode with their top 4 bits, and redo a step at 8 bits only when the low-precision output distribution looks uncertain (entropy above a threshold or a small top-two margin). Quantization is simulated, the model is Qwen3-0.6B, and the benchmark is GSM8K. Cost is counted in weight bytes read per token. On 36724 replayed steps the entropy of the low pass ranks argmax flips with AUC 0.93. At a 15% escalation budget it catches 78% of flips and the margin gate 94%, against the 15% expected for a random gate of equal rate (a computed expectation, not a run). In 150 live problems all-high scores 64.0% and all-low 55.3% (McNemar p=0.053). An entropy gate with a 16% escalation rate scores 62.7% at 0.87 of the all-high weight bytes. Its paired difference is -1.3 (-8.0 to +5.3) points to all-high and +7.3 (+0.0 to +14.7) points to all-low (p=0.080), so the live data do not separate the gate from either end. Verified speculation reaches zero KL at similar bytes in the same cost model. A live random control, a thinking-mode phase rule, a second model and live KL were not run.

Paper page on Hugging Face · arXiv

Data: hysts-bot-data/daily-papers-stats and the Daily Papers API. Open data: tardellirs/paper-pulse-data. Sister project: Model Pulse, the download history of every model on the Hub.