Speculative decoding attacked the most expensive loop in LLM serving — one sequential target-model pass per token — by verifying drafts in parallel. DFlash attacks the loop everyone left serial: the drafting itself. Instead of proposing $K$ tokens in $K$ autoregressive steps, a block-diffusion drafter emits the entire block in a single forward pass.
That parallelism is the whole trick, and it costs something: positions that no longer coordinate with their neighbors, and predictions that decay toward the end of the block. DFlash 2 keeps the parallel backbone and repairs exactly those two weaknesses — a candidate selector that picks a coherent path through each position’s top tokens, and a lightweight local convolution that carries information between adjacent positions.
This post covers the two generations. DSpark starts from the same DFlash backbone and takes a different fork: a sequential correction head plus confidence-scheduled verification. It is a sibling method, not a third DFlash release.
The problem: one expensive token at a time
An autoregressive language model factorizes a sequence as
\[p(x_{1:T}) = \prod_{t=1}^{T} p(x_t \mid x_{<t}).\]That factorization is a strict dependency chain. Before generating token $t+1$, the model must know token $t$. A request for 100 new tokens therefore needs roughly 100 sequential target-model passes.
This is especially wasteful at low batch sizes. A decode step often spends more time moving model weights through memory than doing arithmetic, so the GPU stays busy without being well utilized.
Speculative decoding changes the schedule:
normal decoding
target -> token 1 -> target -> token 2 -> target -> token 3
speculative decoding
drafter -> [token 1, token 2, token 3]
|
v
target verifies the entire candidate block in parallel
A cheap drafter proposes future tokens. The expensive target model scores the whole proposal in one verification pass, accepts the longest valid prefix, corrects the first rejection, and repeats. For greedy decoding, verification checks whether each proposed token matches the target model’s choice.
For sampling, the lossless claim is stronger. If the drafter proposes token $y$ from distribution $q$ and the target assigns distribution $p$, verification accepts $y$ with probability
\[\alpha(y) = \min\left(1, \frac{p(y)}{q(y)}\right).\]If it rejects, it samples a correction from the residual distribution proportional to $[p-q]_+$. If every drafted token survives, the target can also emit one bonus token from its next-token distribution. This accept-or-correct construction makes the committed output distributed exactly as the target model; the drafter changes latency, not the model being sampled.
That gives us the two quantities that matter:
- Draft cost: how long it takes to propose a block.
- Acceptance length: how many tokens the target accepts per verification cycle.
If $\tau$ counts all committed tokens in a cycle, including that possible target bonus token, the average latency per token is
\[L = \frac{T_{\text{draft}} + T_{\text{verify}}}{\tau}.\]A drafter with excellent predictions but several sequential steps can lose to a slightly less accurate drafter that emits the whole block at once. Likewise, a longer acceptance length is useful only when its extra verification work costs less than the target passes it saves.
Why ordinary speculative drafting still has a serial bottleneck
Classic speculative decoding uses a smaller autoregressive model as the drafter. If it proposes $K$ tokens, it still performs $K$ sequential draft steps:
small drafter: d1 -> d2 -> d3 -> d4
large target: verify [d1, d2, d3, d4]
The target is invoked less often, which is valuable, but the proposal phase remains a chain. Methods such as EAGLE-3 make that chain fast and accurate by predicting target-model features, yet they do not remove the chain itself.
DFlash asks a more aggressive question:
Can every position in the draft block be predicted in the same forward pass?
DFlash: generate the block all at once
DFlash replaces the autoregressive drafter with a lightweight block-diffusion model — typically a small (five-layer) stack of Qwen-style Transformer layers. Given the latest verified token — called the anchor — it fills the remaining positions with mask tokens and predicts all future positions simultaneously.
The name can be misleading if you are picturing image diffusion. DFlash does not run a long denoising trajectory over the block. At inference time it performs one masked, bidirectional prediction pass. “Block diffusion” describes the non-causal masked objective and in-block attention pattern, not an iterative sampling loop.
Why not just use an off-the-shelf diffusion LLM as the drafter? Training a small block-diffusion model directly tends to give low acceptance length, while a large diffusion LLM carries a heavy memory footprint and high drafting cost. DFlash is engineered for the narrow job: propose one block, cheaply, in one pass.
verified context: [ ... x_{t-2}, x_{t-1}, x_t ]
^
anchor
draft input: [ x_t, MASK, MASK, MASK, MASK, ... ]
|
v
one parallel draft pass
|
v
draft proposal: [ x_{t+1}, x_{t+2}, x_{t+3}, x_{t+4}, ... ]
Its embedding table and language-model head can be shared with the frozen target model, while the draft-specific layers learn to turn target features and masked positions into token predictions.
Conditioning on the target model
Predicting a block from masks alone would be hopeless. DFlash therefore reuses information the target model has already computed.
Hidden states from several target layers are concatenated, projected, and normalized into a context representation:
\[H_t = \operatorname{RMSNorm}\!\left(W_c[H^{(\ell_1)};\ldots;H^{(\ell_5)}]\right).\]Inside each draft attention layer:
- queries come from the draft block;
- keys and values come from both the target context and the draft block;
- positions inside the draft block can attend to one another non-causally.
More precisely, target features are injected only through attention keys and values:
\[Q_i = W_i^Q H_d, \qquad K_i = [W_i^K H_t; W_i^K H_d], \qquad V_i = [W_i^V H_t; W_i^V H_d].\]They therefore provide a persistent read-only memory at every draft layer without being reprocessed through the drafter’s query path, output projection, or MLP. This is cheaper than rerunning the full target prefix and richer than conditioning on only its final hidden state.
The target features tell the drafter what the prefix means. Bidirectional in-block attention lets every masked position reason about the entire proposed block.
Conceptually:
target hidden states ----> projection ----> context K/V ----+
|
[anchor, MASK, ...] -----> draft embeddings ----> Q, K, V --+--> draft layers
|
v
target LM head
This is the key distinction from autoregressive drafting. DFlash is not generating position 2 from a sampled position 1, then position 3 from the sampled position 2. All positions emerge from a shared parallel computation.
Training with anchors
During training, DFlash samples many anchor positions from each target sequence. Every anchor defines a short future block. The drafter sees the prefix features and a block containing the anchor plus masks, then learns the ground-truth tokens at the masked positions. Multiple blocks are packed into one training example with a sparse attention mask: tokens can interact bidirectionally inside their own block and read the matching target context, but cannot leak information across blocks.
Earlier positions get extra weight, and for good reason: acceptance is prefix-committed, so if position 1 is rejected, positions 2 through $K$ cannot be committed even if they happen to be correct. DFlash uses the exponentially decaying weight
\[w_k = \exp\left(-\frac{k-1}{\gamma}\right)\]for position $k$ in a block of size $\gamma$. The objective encodes the serving metric directly: improving an early token can unlock the entire suffix, while improving an isolated late token may change nothing.
The payoff is substantial, but the headline needs context. The paper reports more than 6× lossless acceleration in its best tested setting and up to 2.5× higher speedup than EAGLE-3. Across its core Qwen3 Transformer experiments, the reported average speedup is 4.9× for greedy decoding and 4.1× at temperature 1. In an optimized SGLang setup, Qwen3-8B reaches up to 5.1× on math at concurrency 1; the gain narrows as concurrency rises and the target becomes better utilized.
These are research results on specific H200/B200 systems, models, kernels, block sizes, and workloads—not universal constants. DFlash was trained primarily on roughly 800,000 target-generated responses, uses different draft depths for some model families, and is sensitive to runtime integration. Comparing only a paper’s best speedup hides most of the engineering problem.
Where the abstraction leaks
DFlash’s parallelism does not make every design choice free.
- The checkpoint is tied to the target. It consumes selected internal target layers and normally shares the frozen target embedding and LM head. A checkpoint is not a generic draft model that can be moved between arbitrary targets.
- Training and serving block sizes should agree. The paper finds that changing the inference block size away from the trained size reduces acceptance.
- Deeper is not automatically faster. An eight-layer drafter can accept a longer prefix while a five-layer drafter wins on end-to-end speed because its proposal pass is cheaper.
- Long context needs matching training. Acceptance degrades beyond the context lengths seen during training; long-context adaptation recovers much of it.
This is why acceptance length and throughput must be reported together.
Where DFlash loses tokens
One parallel pass removes draft latency, but it creates a coordination problem.
Suppose each position independently exposes its highest-scoring token:
position: 1 2 3 4
top token: "the" "the" "model" "is"
Each token can look plausible in isolation while the path is locally incoherent. The second "the" may have been a strong marginal prediction, but it is a poor continuation of the first "the".
DFlash’s positions are not literally independent — their representations interact through attention. But the final token selection is unary: each position chooses from its own vocabulary logits without explicitly conditioning that choice on the token selected immediately before it.
Two failure modes follow.
1. Candidate selection leaves accuracy on the table
The correct token is often present among a position’s top candidates even when it is not ranked first. Inco AI’s DFlash 2 analysis reports a large gap between top-1 recall and top-16 recall. On one Qwen3-4B GSM8K setup, the measured mean acceptance length rose from 4.27 for the top-1 path to an oracle 6.79 if the correct option could always be selected from the top 16.
The draft model knows more than argmax reveals. It needs a cheap way to choose a coherent path through those candidates.
2. Accuracy decays toward the suffix
Later positions have less direct certainty from the anchor and depend more heavily on relationships inside the block. This produces suffix decay: accuracy falls as we move farther from the verified prefix.
Simply adding Transformer layers helps, but it taxes every position and makes the drafter slower. The DFlash 2 experiments found that expanding a five-layer drafter to fifteen layers improved later positions, but increased draft–verify cycle latency by 15.2%.
The better question is not “How do we make the entire drafter larger?” It is “Which missing dependency causes the suffix to fail?”
DFlash 2: keep the backbone parallel, repair local dependencies
DFlash 2 introduces two targeted modules:
- a candidate selector that chooses a coherent path through each position’s top tokens;
- a lightweight local convolution that carries information between neighboring positions.
The expensive draft backbone still runs once for the whole block.
The candidate selector
For each position, DFlash 2 keeps the top $k$ candidates from the original draft logits. The public models use $k=16$. It then scores transitions between candidates at adjacent positions:
\[S_t(a,b) = U_t(b) + \langle A(a) \odot H(h_t), B(b) \rangle.\]Here:
- $U_t(b)$ is the original DFlash score for candidate $b$ — “does $b$ fit this position?”
- $a$ is the token selected at the preceding position;
- $A(a)$ and $B(b)$ are compact token embeddings;
- $H(h_t)$ uses the current hidden state to gate which dimensions matter.
The second term asks, “does $b$ make sense after $a$, in this context?” All pairwise transition scores are computed in parallel. Only the final walk through the precomputed scores is sequential, and that walk is tiny compared with another Transformer or full-vocabulary LM-head pass.
DFlash logits
position 1: {a1, a2, ... a16}
position 2: {b1, b2, ... b16}
position 3: {c1, c2, ... c16}
|
v
score adjacent candidate pairs in parallel
|
v
walk one coherent path: anchor -> a? -> b? -> c?
The selector converts strong candidate recall into a stronger accepted path without making the draft Transformer autoregressive.
The two-tap dynamic convolution
The selector improves the final choice, but it cannot recover a token missing from the top-$k$ set. Fixing suffix decay requires improving the backbone itself.
DFlash 2 adds a short dynamic depthwise convolution before and after each attention and MLP sublayer:
\[\operatorname{Conv}_k(x)_t = k_{t,0} \odot x_t + k_{t,1} \odot x_{t-1}.\]Each position mixes its representation with its immediate predecessor. The kernel is content-dependent, groups channels to stay small, and never crosses a block boundary. Position 1 reads from the verified anchor; later positions read from their left neighbor.
This looks sequential on paper, but it is not a token-by-token recurrence. Every position’s two-tap convolution is evaluated together. The operation creates a dedicated local information path while attention focuses on reading the longer target context.
In the authors’ ablation, these convolutions added roughly 3% parameters and 0.7% cycle latency. They recovered much of the suffix benefit of adding ten full Transformer layers.
What the measured improvement looks like
In a matched Qwen3.5-4B comparison across GSM8K, MATH-500, HumanEval, MBPP, and MT-Bench, the DFlash 2 authors report:
| Drafter | Mean acceptance length |
|---|---|
| MTP | 4.54 |
| DFlash | 4.92 |
| DSpark | 5.49 |
| DFlash 2 | 5.97 |
That is 1.05 additional accepted tokens per cycle over DFlash, or about 21%, while the convolution and selector together add a reported 1.3% to draft–verify cycle latency.
The comparison is promising, but it should be read correctly. These are results from the DFlash 2 authors under their matched setup, not a law that DFlash 2 will be 21% better on every model or serving stack. Acceptance length is also not throughput: a system can accept more tokens yet run slower if the drafter, verifier, memory layout, or synchronization overhead is inefficient.
DSpark’s 5.49 sits between DFlash and DFlash 2 in this table because both methods attack the same suffix-decay problem. They do it differently. DFlash 2 stays fully parallel. DSpark adds a sequential head, then spends its unique budget on deciding how much of the block is worth verifying.
DFlash vs DFlash 2
| Property | DFlash | DFlash 2 |
|---|---|---|
| Draft backbone | Block-diffusion Transformer | Same parallel backbone |
| Backbone passes per block | One | One |
| In-block interaction | Bidirectional attention | Attention plus local dynamic convolution |
| Token selection | Per-position unary logits | Top-$k$ candidates plus predecessor-conditioned selector |
| What it fixes | Autoregressive draft latency | Path coherence and suffix decay |
| Complexity | Simpler and more established | Slightly larger and newer |
| Current ecosystem status | Broader runtime support | Experimental or recently added support |
DFlash is the cleaner baseline. DFlash 2 is the more ambitious drafter when compatible checkpoints and runtime support exist.
There are also current implementation constraints. The vLLM Speculators documentation notes that DFlash 2 presently requires the verifier’s full vocabulary, and that its public training objective is not fully specified. Speculators therefore documents its own experimental split objective for training the unary logits and selector. Checkpoint names may look interchangeable, but architecture, vocabulary mapping, training recipe, and serving support still matter.
How to evaluate a DFlash checkpoint
Do not start with tokens per second alone. Measure the chain that produces it:
- Per-position acceptance: reveals suffix decay hidden by a single average.
- Mean acceptance length: shows how many tokens each target verification commits.
- Draft and verify latency separately: identifies whether the proposer or target is the bottleneck.
- End-to-end inter-token latency: captures synchronization, kernels, and scheduler overhead.
- Workload sensitivity: compare code, math, chat, long context, sampling, batch size, and concurrency.
The relevant optimization target is not “most accepted tokens.” It is lowest end-to-end cost per committed target token.
Final perspective
DFlash’s central contribution is structural: it removes the sequential loop from drafting. DFlash 2 shows that parallel generation does not require abandoning token dependencies; the right dependencies can be restored with cheap local operators and a small path selector.
The progression is:
autoregressive drafter
|
v
DFlash: parallelize the entire draft block
|
v
DFlash 2: repair local coherence without serializing the backbone
DSpark is the other fork from the same backbone. It spends a little sequential compute to repair the suffix, then asks a serving question DFlash 2 does not: once you can draft a long block, how much of it should the target actually verify?
The bigger lesson is broader than one algorithm family: inference gets faster when we separate the work needed to guess from the work needed to be correct.
References
- Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast Inference from Transformers via Speculative Decoding, 2022.
- Charlie Chen et al. Accelerating Large Language Model Decoding with Speculative Sampling, 2023.
- Jian Chen, Yesheng Liang, and Zhijian Liu. DFlash: Block Diffusion for Flash Speculative Decoding, 2026.
- Inco AI. DFlash 2: Keep Drafting Parallel, August 2026.
- Xin Cheng et al. DSpark: Confidence-Scheduled Speculative Decoding with Semi-Autoregressive Generation, 2026.
- vLLM Project. DFlash 2 — Speculators Documentation, accessed September 2026.