Makes me think how many submissions are just better than this one by quietly removing that line...

Hey r/MachineLearning!

We're excited to share ORL (On-Policy Residual Learning), our new research on making diffusion language models (dLLMs) significantly faster without sacrificing generation quality.

With ORL and our custom inference engine dlmrt, we've achieved:

• 2,788 tokens/s on a single RTX 5090 — LLaDA-2.1-mini + ORL, NVFP4
• 7,708 tokens/s on 8× B200 — DiffusionGemma 26B MoE + ORL, NVFP4
• Up to 6.9× higher end-to-end throughput on our QwenDLM-4B model, at comparable or better quality
• 1.74× faster than DFlash speculative decoding at matched accuracy in our single-request H100 comparison

All numbers are for single-request decoding (batch size 1), not aggregate throughput across concurrent requests. The GPU figures above are peak results on specific benchmarks, not averages across arbitrary prompts.

We've open-sourced the models, training code, evaluation pipeline, and inference engine.

What's the idea behind ORL?

Unlike autoregressive LLMs, which generate one token at a time, diffusion LLMs can refine and finalize multiple tokens in parallel. But getting them to converge in fewer forward passes is challenging.

Our approach has two key ingredients:

1. On-policy residual learning

Instead of predicting everything from scratch at each denoising step, the model carries forward its previous predictions as residual context and progressively refines them.

The catch: these residuals are generated by the model itself, and their distribution changes during training. Training on stale, offline residuals creates a mismatch with inference.

ORL addresses this by training directly on fresh trajectories generated by the current model, replaying the exact residual states encountered during inference.

2. Learning to reduce uncertainty faster

Simply supervising the final output doesn't necessarily teach the model to converge faster.

We introduce Adjacent Entropy Contraction, which encourages each denoising step to reduce uncertainty relative to the previous step. We also gate the loss to avoid unnecessarily pushing already-confident predictions.

The result: the model learns to finalize more tokens per forward pass while maintaining or improving accuracy.

Some benchmark results

We tested ORL on three model families:

• QwenDLM-4B: Our diffusion model converted from Qwen3-4B. ORL improves tokens per forward by up to 6.2× and end-to-end throughput by approximately 6.9× with dlmrt. HumanEval pass@1 improves from 82.9% to 89.6%.
• LLaDA-2.1-mini: ORL requires only 4.2M self-generated training tokens and 36 H100 GPU-hours of post-training, delivering up to 3× higher throughput at comparable quality.
• DiffusionGemma 26B MoE: ORL further improves an already highly parallel diffusion model, with up to 1.2× more tokens per forward pass.

For a more direct comparison with autoregressive decoding, on a single H100, our QwenDLM-4B + ORL reaches 1,298 tokens/s, versus 746 tokens/s for DFlash and 405 tokens/s for EAGLE3 under relaxed verification at comparable measured accuracy. These are end-to-end system comparisons: ORL uses dlmrt, while the speculative decoders use vLLM.

We also built a custom inference engine: dlmrt

We developed dlmrt, optimized specifically for low-latency, single-request diffusion decoding.

Instead of launching and synchronizing every denoising step from the CPU, dlmrt captures the entire denoising loop in a CUDA graph and executes it on the GPU.

It also supports NVFP4 inference on Blackwell GPUs, including the consumer RTX 5090.

With identical models and decoding settings, dlmrt runs 1.2–1.6× faster than SGLang/vLLM in our single-H100 comparisons.

On the training side, we also co-designed the system around efficient residual-state capture, specialized attention kernels, and compute-balanced packing, accelerating training steps by up to 5.2×.

Everything is open source

• Paper: https://arxiv.org/abs/2610.20687
• Project page + interactive demos: https://introspective-diffusion.github.io/orl-dlm/
• Training code and evaluation: https://github.com/Introspective-Diffusion/ORL
• Inference engine (dlmrt): https://github.com/Introspective-Diffusion/DLMRT
• Model checkpoints: https://huggingface.co/yifanyu

If you have an RTX 5090, you can try the LLaDA-2.1-mini ORL NVFP4 checkpoint with dlmrt.

We'd love feedback from the community, especially anyone interested in reproducing the results, testing different workloads, or contributing to the inference engine.

Curious to hear what you think about diffusion LLMs for local inference. Is single-request latency the main thing you'd want optimized, or are there other bottlenecks you'd prioritize?

Basically the title. Due to the nonsensical situation of NeurIPS being fully sold out before the results were even released, I couldn't get workshop registration (my paper is in the main track, which apparently doesn't have any sort of gaurantee for registrations in other parts of the conference like workshops).

My experience with conferences has been with full registrations, so I don't really know how the experience goes if I only have registration for the main conference and not the workshops. Is there any specific networking opportunities/usefulness, or should I cut my stay short and leave after the main conference ends? Is there any use staying after that if I don't have registration for workshops?

Hi,

I just finished my masters and got a neurIPS workshop acception. Unfortunately the lab I did it with do not want to fund neither the registration nor flights and accommodation.

I am thinking about getting the workshop only ticket and pay for it myself.

Would you say it is worth it, if I just have a workshop poster?

Thanks

For the past 4 weeks I’ve been training models of the Drosophila brain to control a car in a drag racing simulator. To make the training watchable, I turned it into a racing league and stream it live on Twitch.

How it works

• Each “pilot” is a fly brain model of ~2,400 neurons [based on: FlyWire connectome / own architecture / spiking network?]
• Training: [RL algorithm / evolutionary strategy / plasticity rule], 10000000+ episodes so far
• The best pilots produce “offspring”: the next generation inherits their weights, so the league has dynasties (gen 2, child of the season champion, etc.)
• Every Grand Prix is a set of races under different conditions (engine, grip, dry/wet). A fly is “verified” only if it finishes all scenarios cleanly
• Skill is measured with an Elo rating, like in chess

Current state (Season 11, GP 86): the dry-strip record is 10.06 s; the champion has 56.31 s total across 5 scenarios. Last GP was won by a second-generation fly, 0.02 s ahead of its rival.

What surprised me: [1–2 real observations, e.g. “flies discovered launch control on their own”, “descendants overtook the parent in 300 episodes”, “one fly learned to exploit…”]

🔴 Live: twitch.tv/nekrampos
[💻 Code: https://github.com/boeing240/flypilot\]

Happy to answer questions about the model, the training setup and the visualization. Ideas for new race scenarios are welcome too.

hey, CS student here. for the past few days I've been building DenseCore, a small deep learning framework in rust with a pytorch-like python API.

the thing I'm kind of proud of: if you point `import torch` at it and run karpathy's nanoGPT unchanged, you get the exact same numbers as pytorch. same initial weights from manual_seed, same loss, same gradients, same weights, bit for bit, for all 50 steps I tested, even with dropout on. not "close enough", literally identical.

getting there was a rabbit hole. pytorch's CPU attention isn't the textbook formula, it's a flash attention kernel, and it switches to a different path when dropout is on. adam rounds differently on the last few elements of each thread's chunk because of how the compiler fuses the tail loop. clip_grad_norm_ secretly does reciprocal() * max_norm instead of a division. stuff like that everywhere.

also found out pytorch doesn't match itself across machines. same version, mac vs linux x86, only 1 of 26 ops I tried gave the same bits. so mine only matches pytorch on apple silicon macs.

it's tiny though, about 2 MB vs ~580 MB for pytorch.

anyway the honest question: is this useful to anyone, or is it just a cool learning project? if I keep going I'm thinking either keep copying pytorch until most of it works, or make it a lightweight drop-in for small cpu stuff like serverless/CI, or turn it into a tool that tells you why two pytorch runs give different results. would any of those be useful to you? or should I move on? brutal answers are fine.

repo: https://github.com/lncch/DenseCore

(`python scripts/demo_bit_exact.py` runs the nanoGPT comparison if you want to check)

Estimating the Theoretical Parameter Capacity of a Hypothetical GPT-6-Class Model

I've been exploring a hypothetical question: How large could a model's parameter count be if a major AI provider allocated a substantial GPU cluster to inference while targeting 66 output tokens per second per active user?

This is a rough hardware-based thought experiment, not a claim about OpenAI's actual infrastructure or the architecture of GPT-6 Astra.

Assumptions

• GPU resources: 800,000 H100-equivalent GPUs available across the broader infrastructure.
• Allocation scenario: 10–30% allocated to the hypothetical model's inference workload.
• Per-user generation speed: 66 tokens/second.
• Concurrent users: 10,000–100,000.
• GPU memory: 80 GB per H100, giving 64 PB across 800,000 GPUs before allocation.
• Weight format: FP8, approximately one byte per parameter.
• H100 peak performance: Approximately 1.98 PFLOP/s dense FP8 per GPU, under the relevant hardware configuration.

A rough memory-based estimate

If 30% of the assumed GPU pool were allocated to inference:

800,000 × 0.30 = 240,000 H100-equivalent GPUs.

At 80 GB of memory per GPU, this allocation would provide approximately 19.2 PB of aggregate GPU memory. At roughly one byte per FP8 parameter, that corresponds to a theoretical weight-storage capacity of approximately 19.2 trillion parameters.

However, this is only a memory-based upper bound under these assumptions. Actual usable capacity would be lower because of KV cache, runtime overhead, communication buffers, redundancy, and other memory requirements.

What about inference throughput?

The required output throughput would be:

• 10,000 concurrent users × 66 tokens/s = 660,000 tokens/s.
• 100,000 concurrent users × 66 tokens/s = 6.6 million tokens/s.

Whether a model of a given size could sustain those rates depends on its architecture, batching strategy, memory bandwidth, parallelism, workload, and actual hardware utilization. Peak FLOPS alone cannot establish that it can.

Important limitations

1. The 800,000-GPU figure is a hypothetical resource pool, not a verified count of GPUs owned or operated by OpenAI.
2. The 10–30% allocation is an illustrative assumption, not a known allocation for GPT-6 Astra.
3. No verified parameter count or architecture for GPT-6 Astra is established by this calculation.
4. Parameter count, model architecture, and inference throughput cannot be inferred reliably from GPU count alone.

Conclusion: Under this hypothetical scenario, 19.2T parameters is a rough FP8 weight-memory ceiling before overhead—not a prediction of GPT-6 Astra's actual size. A credible throughput estimate would require additional assumptions about architecture and measured serving efficiency.

I'd be interested in feedback from people experienced with large-scale inference: what assumptions would you change first, and how would you estimate sustainable throughput at this scale?

Half of one sentence in my paper trails onto page 10...

This must have happened after a small edit and recompile of the pdf on resubmission?

but this got me desk rejected??? Is this normal?? This wasn't even flagged by the official LLM feedback?

Like whats the point of providing LLM feedback if it can't catch this. I don't need an AI slop breakdown of my papers strengths and weaknesses, I need to know that i didn't miss any ridiculous formatting mistakes that would lead to a desk rejection.

This whole thing feels criminal and humiliating :(

Especially after already being de-anonymized...