Orthrus: Parallel Token Generation That Doesn't Change Your Model's Output

Speculative decoding cut LLM inference latency by predicting multiple tokens ahead and validating them with the base model. It works — but you pay for it with a separate draft model, a second KV cache, and acceptance rates that fall off when the drafter misreads the distribution. Orthrus is a research direction that aims for the same speedup without those overheads. It bolts a trainable diffusion attention module onto each layer of a frozen autoregressive Transformer and uses it to emit blocks o...