Discover how speculative decoding uses a small draft model to propose tokens that your main model then verifies in a single pass, on supported devices, with zero quality loss.
TokForge is free. Android is live on Google Play; iPhone and iPad are in public beta on TestFlight. No account, and it works with the network off once a model is downloaded.
Reading on a computer? Check which models fit your phone, or see real measured speeds on the phone-speed leaderboard.
Large language models are incredibly powerful. A 14-billion-parameter model can handle complex reasoning, creative writing, and nuanced conversation. But on a mobile device, there's a tradeoff: power versus speed.
Large models on a phone generate slowly, and the larger the model the slower it gets. Every token requires a full forward pass through the entire model, which is expensive on hardware that has to worry about heat and battery. What that costs on a specific handset is on the phone-speed leaderboard, which records decode tokens per second per device rather than offering an average that fits nobody.
The bottleneck: One large model. One forward pass per token. Sequential generation. If you need 100 tokens, you're doing 100 full model passes.
A hosted API feels faster for two structural reasons that have nothing to do with the model being better: it runs on datacentre accelerators, and it batches many people's requests together to keep those accelerators busy. On a phone you have one request and whatever compute is left after the operating system has taken its share.
Speculative decoding is a technique that trades compute for speed by introducing a small, fast "draft" model alongside your large "target" model. Here's the insight:
The key idea: The draft model proposes multiple tokens ahead. The target model verifies them all in a single batch. If the draft guesses correctly (which happens most of the time on predictable text), you get multiple tokens for the computational cost of roughly one large-model forward pass.
Accepted proposals come out without a full forward pass each, so throughput rises while the output text stays mathematically identical. You're not sacrificing accuracy for speed. You're just being smarter about how you generate tokens.
Think of it like writing versus proofreading. Writing something from scratch is slow. You need to think about each sentence, each word. The large model generating tokens one at a time is like writing from scratch.
But checking someone else's draft is much faster. You scan through it quickly, verify the ideas, and catch any mistakes. The large model checking multiple pre-drafted tokens is like proofreading.
In speculative decoding, the draft model is your quick writer, and the target model is your fast proofreader.
On predictable text (dialogue, prose, documentation), the draft model guesses correctly much of the time. Even with occasional misses, you're still generating multiple tokens per target-model forward pass. The overhead of the draft model is negligible because it's so small and fast.
Result: accepted proposals arrive without a full forward pass each, with zero quality loss. Every output token is verified by the full target model, so you get identical results.
Here's how speculative decoding performs on actual TokForge models:
Draft-verified decoding active, with the live speed indicator in the chat toolbar
On our test devices (Dimensity 9400 and Snapdragon 8s Gen 3 class), dense Qwen3 models with the standard draft pack are where the gain is clearest. Speedup varies by device, model, and text domain, and TokForge disables spec decode automatically on combinations where it does not help.
Speculative decoding shines on predictable text:
Speedup is lower on unpredictable content:
Where drafting does not pay off on your device and model, TokForge's self-calibration disables spec decode rather than slow you down.
Chat showing speculative decoding speed badge
Compatible models are marked with a Spec Decode badge in the Model Manager. To enable:
To enable or disable speculative decoding programmatically:
POST /settings/spec_decode_enabled
Content-Type: application/json
{
"value": true
}
// Response
{
"spec_decode_enabled": true,
"draft_model_size": "0.36GB",
"compatible_models": ["qwen3-14b", "qwen3-8b", ...]
}
You can also check the list of spec-decode-compatible models:
GET /models?filter=spec-decode-compatible
// Response includes models tagged as spec-decode compatible
To get the best bang for your buck, use speculative decoding when:
You can leave it on all the time. Where a device and model pair does not benefit, TokForge's self-calibration turns spec decode off automatically. And the quality is always identical.
This is the critical guarantee of speculative decoding: the output quality is mathematically identical to normal generation.
Here's why: every token produced by speculative decoding is verified by the target model. The draft model is just a speedup trick. It never creates the final output. It only proposes candidates, and the target model checks them.
Think of it this way: Speculative decoding is like using a calculator to speed up arithmetic. The calculator proposes answers, you verify them. If the calculator is right, great. You save time. If it's wrong, you catch it immediately. Either way, your final answer is correct.
In formal terms, speculative decoding uses a technique called acceptance-rejection sampling. The probability distribution of output tokens is preserved exactly. You get the same statistical distribution of responses as you would without spec decode.
So feel free to turn it on and forget about it. Your outputs will be the same quality, just faster.
Speculative decoding works on most modern devices, but there are a few requirements:
TokForge checks these requirements automatically and shows a warning if spec decode is unavailable for your setup.
Speculative decoding is one of several techniques TokForge uses to maximize mobile performance. Here are related guides:
Learn how model quantization reduces memory footprint and enables larger models on mobile devices.
Read Guide →Discover how AutoForge automatically selects the best model, quantization, and inference settings for your device.
Read Guide →Compare MNN, NCNN, and other inference backends to understand which is best for your use case.
Read Guide →Try speculative decoding on your mobile device today. Get the app from Google Play.
Google Play →Speculative decoding is a proven technique that delivers real speedups on mobile devices. By pairing a small draft model with your large target model, TokForge produces accepted tokens faster on supported devices with zero quality loss.
If you're looking to get the most out of on-device LLMs, speculative decoding is a technique worth understanding, and even better, a feature worth using.