Discover how speculative decoding uses a small draft model to propose tokens that your main model then verifies in a single pass, with no loss in answer quality. In TokForge it is off by default and turns on only where it measured faster on that phone.
TokForge is free. Android is live on Google Play; iPhone and iPad are in public beta on TestFlight. No account, and it works with the network off once a model is downloaded.
Reading on a computer? Check which models fit your phone, or see real measured speeds on the phone-speed leaderboard.
Large language models are incredibly powerful. A 14-billion-parameter model can handle complex reasoning, creative writing, and nuanced conversation. But on a mobile device, there's a tradeoff: power versus speed.
Large models on a phone generate slowly, and the larger the model the slower it gets. Every token requires a full forward pass through the entire model, which is expensive on hardware that has to worry about heat and battery. What that costs on a specific handset is on the phone-speed leaderboard, which records decode tokens per second per device rather than offering an average that fits nobody.
The bottleneck: One large model. One forward pass per token. Sequential generation. If you need 100 tokens, you're doing 100 full model passes.
A hosted API feels faster for two structural reasons that have nothing to do with the model being better: it runs on datacentre accelerators, and it batches many people's requests together to keep those accelerators busy. On a phone you have one request and whatever compute is left after the operating system has taken its share.
Speculative decoding is a technique that trades compute for speed by introducing a small, fast "draft" model alongside your large "target" model. Here's the insight:
The key idea: The draft model proposes multiple tokens ahead. The target model verifies them all in a single batch. If the draft guesses correctly (which happens most of the time on predictable text), you get multiple tokens for the computational cost of roughly one large-model forward pass.
Accepted proposals come out without a full forward pass each, so throughput rises while the output text stays mathematically identical. You're not sacrificing accuracy for speed. You're just being smarter about how you generate tokens.
Think of it like writing versus proofreading. Writing something from scratch is slow. You need to think about each sentence, each word. The large model generating tokens one at a time is like writing from scratch.
But checking someone else's draft is much faster. You scan through it quickly, verify the ideas, and catch any mistakes. The large model checking multiple pre-drafted tokens is like proofreading.
In speculative decoding, the draft model is your quick writer, and the target model is your fast proofreader.
On predictable text (dialogue, prose, documentation), the draft model guesses correctly much of the time. Even with occasional misses, you're still generating multiple tokens per target-model forward pass. The draft model is small, but it is not free: on some phones and some kinds of text the extra work costs more than it saves. That is why TokForge measures before it turns speculative decoding on.
Result: accepted proposals arrive without a full forward pass each, with zero quality loss. Every output token is verified by the full target model, so you get identical results.
Here's what testing on real phones showed:
In TokForge 1.3.6.1 (Google Play), speculative decoding is on by default only where a side-by-side test on that chip measured it faster than normal generation. Today that means dense Qwen3 8B and 14B models on the CPU of some flagship phones, and Qwen3 4B on some mid-range Snapdragon phones. Everywhere else it stays off by default. In the beta (1.3.8 and newer), the flagship defaults for Qwen3 8B and 14B are retired; the mid-range Snapdragon default for Qwen3 4B stays, and you can switch it Off, Auto or On yourself in Settings (Advanced mode) > Speed & engine > Performance > Measured on your phone.Beta: 1.3.8 and newer Results vary by device, model, and kind of text. For speeds on specific phones, see the phone-speed leaderboard.
Speculative decoding shines on predictable text:
Speedup is lower on unpredictable content:
Where drafting does not pay off on your device and model, TokForge's self-calibration disables spec decode rather than slow you down.
Qwen3 models on the MNN engine carry a Spec Decode badge in the Model Manager. In TokForge 1.3.6.1 there is no switch to flip in Settings. In the beta (1.3.8 and newer), the Draft model speculation row in Settings (Advanced mode) > Speed & engine > Performance > Measured on your phone has Off, Auto and On.Beta: 1.3.8 and newer Here is how it works:
With the Developer API server (Advanced) turned on in Settings, you can install
the Acceleration Pack and switch speculative decoding on or off from a script. Every call needs
the access token from that screen. To install the Qwen3 pack (use "llama" for the
Llama 3 pack):
POST /control/install-acceleration-pack
Authorization: Bearer <token>
Content-Type: application/json
{
"pack_id": "qwen",
"enable": true
}
To turn it off, or back on, and to read the current value:
POST /settings/spec_decode_enabled
Authorization: Bearer <token>
Content-Type: application/json
{
"value": false
}
GET /settings/spec_decode_enabled
// Response
{
"key": "spec_decode_enabled",
"value": false
}
To get the best bang for your buck, use speculative decoding when:
Free-form prose and roleplay gain less, and on some phones they run slower with a draft model. You don't have to manage this yourself: where a device and model pair does not benefit, TokForge's self-calibration keeps spec decode off. And the quality is always identical.
This is the critical guarantee of speculative decoding: the output quality is mathematically identical to normal generation.
Here's why: every token produced by speculative decoding is verified by the target model. The draft model is just a speedup trick. It never creates the final output. It only proposes candidates, and the target model checks them.
Think of it this way: Speculative decoding is like using a calculator to speed up arithmetic. The calculator proposes answers, you verify them. If the calculator is right, great. You save time. If it's wrong, you catch it immediately. Either way, your final answer is correct.
In formal terms, the standard form of speculative decoding uses acceptance-rejection sampling. The probability distribution of output tokens is preserved exactly. You get the same statistical distribution of responses as you would without spec decode.
So where it is on, you can forget about it. Your outputs are the same quality, and where it helps, they arrive sooner.
Speculative decoding ships in the Android app. There are a few requirements:
When AutoForge cannot test speculative decoding, for example because the Acceleration Pack is not installed or the model has no matching draft, it says so and skips that step.
Speculative decoding is one of several techniques TokForge uses to maximize mobile performance. Here are related guides:
An optional Android setting that lets longer conversations fit in less memory.
Read Guide →See how AutoForge tests speed settings on your phone and lets you apply the best ones.
Read Guide →Compare MNN and llama.cpp (GGUF), and the CPU and GPU paths each one can use.
Read Guide →Try speculative decoding on your Android phone. Get the app from Google Play.
Google Play →Speculative decoding is a proven technique that can deliver real speedups on some phones. By pairing a small draft model with your large target model, TokForge produces accepted tokens faster where it measured a gain, with zero quality loss.
If you're looking to get the most out of on-device LLMs, speculative decoding is a technique worth understanding, and even better, a feature worth using.