During a post-event interview at the RTX Spark Launch Party: Taipei on October 16, 2026, Nvidia's Senior Vice President of Personal Computing, Jeff Fisher, explicitly debunked the recent discontinuation rumors.

Already posted here https://www.reddit.com/r/LocalLLaMA/comments/1x85hfj/strata_qwen_38_flash_next_fast_but_whats_it/phavui9/ , but I believe this topic deserves it's own post.

The PLE, aka the n-gram table that sits on the SSD, is quantized heavily in the common GGUFs and has a measurable impact on performance, demonstrated here by /u/MarkoMarjamaa : https://www.reddit.com/r/LocalLLaMA/comments/1x6kw7k/qwen38_flash_next_unsloth_quants_kldtop1_with/

Further down in the comments, /u/Eyelbee mentions running the full bf16 n-gram table. I asked Grok and apparently that only works in llama.cpp et.al and is called the bf16 sidecar, Strata needs FP8 but that should be full quality, over my head to be honest. So I built it with ple_fp8_pack.py and indeed it seems to positively impact results. I run iq3_s and before the model made many more stupid mistakes. No impact on speed! Only needs a bit more disk space.

If you want to try this, you need model-00005 up to model-00037.safetensors and model.safetensors.index.json from https://huggingface.co/Qwen/Qwen3.8-Flash-Next, then pack them into a GGUF with

tools/ple_fp8_pack.py --model /media/your_ssd/Strata-data/download_folder --out /media/your_ssd/Strata-data/ple/ple-fp8.gguf

and load it in your .json with

"--ple-gguf",
"/media/your_ssd/Strata-data/ple/ple-fp8.gguf"

Adjust your file paths obviously. Sorry I can't upload 47gigs. :-) Not sure how to properly test performance, but here's two scenes with the same silly prompt.

Build a beautiful 3D scene of a train driving through a futuristic cyberpunk city in HTML. dystopian vibes, detailed models, try not to use too many boxes in the models. Be creative and add some user interactivity like camera and weather controls.

IQ3_S with the included PLE (quantized to IQ4_NL ) https://output.jsbin.com/fojexecanu/

IQ3_S with the full FP8 PLE from Qwen https://output.jsbin.com/hosusuponi/

Imo a clear improvement is visible. I noticed that with Hermes agent too and was able to add several features to a KDE app (haruna). 3D models seem to have less errors like missing faces.

I know everyone's excited about 3.8, but honestly I'm still refining 3.6. And I hate the way 3.8 talks, so...

I keep seeing all these posts about what's the use of open source, so I figured I'd submit my own answer with this post.

Qwen 3.5 27B was my first agent, starting my journey around May 31st, and I've learned a lot in these 4+ months, so I wanted to share with other people just starting their journey because there are clearly a LOT of new users out there, and people want help but sometimes are afraid to ask or not sure the right questions. So, here are ten things that you should know you can do to empower you!

(Oh, and I'd recommend using llama.cpp. That's what I've used to do all these tricks)

1. Create neat little apps for yourself. Qwen's really good at making apps for anything. I'm writing this post in a app I made as a replacement for Notepad, that has full word processing, side-panel, light/dark mode, etc. (image below). The first thing you should be thinking about when you run into any sort of issue is, "can I make an app that will make this simpler?" The answer is usually yes. 

2. Create your own Chat UI frontend. If you don't feel that creative, just copy an existing one. LLMs are great copy cats. I liked claude, so I just copied it as my own frontend, customizing it for my own needs:

3. Create ReAct Loops. This is what enables your model to use a tool, read the result, then use it again, multiple times. Qwen can make these in its sleep, and many different ways. It can create a loop anywhere you want it, and you'll need it for your own custom chat ui frontend if you make one.

4. Create MCP tools. This is my favorite. It can make almost any type of mcp tool you need. You don't have to use third party ones, which can be unsafe. Just tell it you want an mcp tool to do x, it'll make it. It might or might not require its own mcp server, but if it does, Qwen can make that too.

5. Spawn sub-agents. Another super-easy thing Qwen can do. Just tell it to create a sub-agent spawner, which is just a python file. Then, you can tell it to run that when it needs to create a sub-agent, or tell it to create an mcp tool to do it on-the-fly when it wants to. Simple. Oh, and a quick hack-tip: mcp tools use up a LOT of context, so a cheat is to spawn a sub-agent with its own context limit and have your main agent pass off work to the sub. Basically, you can double/triple/etc your context limit. And yes, you can retain it between turns, but you'll need to create a pipeline runner (see below).

6. Create a pipeline runner. Ever heard of n8n? Basically, it's a pipeline runner. It has a lot of features, but it has a few limitations and restrictions, and there were a few things I wished it could do. I learned I could just build something similar with Qwen super easy. There's some design work involved if you want a UI frontend to build, but it's completely doable. I made my own ui, but after a while, I got tired of tweaking it, so I just copied n8n's frontend and just customized it myself, lol. Another useful tip: You can run multiple sequential agents in a pipeline runner, but if you want them each to have their own context, you'll need to ensure cache-idle-slots is enabled in llama.cpp. This'll swap the kv caches each turn, ejecting the last one to cpu ram. There's a bit more to it, but Qwen's got you, so just ask.

7. Create a model launcher (with options to select different models/jinja templates in llama.cpp). This is my bread and butter. Its a launcher that lets me choose any model in my directory and set its parameters. When I'm happy, I create an alias for it. On next launch, I can just choose the alias and it launches the model with the params for that alias only. So I can have 10 different types of param configurations for the same model if I want; I just need to give them their own alias. Hack tip: I'd recommend adding your jinja templates to the launcher to, and you can choose different custom templates for different model types.

8. Install, test, fork, and/or rebuild open-source apps on the internet. The reason I moved to llama.cpp was lm studio pp was too slow, but I didn't want the stress of figuring it all out, so I just asked my agent to download and install llama.cpp for me. I've been using it ever since. It can also modify builds, fork repositories on github, and a lot more. For example, I was busy working on a project, but heard about some new TTS apps, so I told my agent to download and install Moss and OmniVoice, make some audio clips to make sure they run, and let me know when it's done. Of course everything worked fine. Great for multi-tasking.

9. Give your agent temporal awareness of the world around it. Ever used cloud models that argue with you about the date? Or wished your model knew when you were nearing your max context, so it could automatically start writing a session summary? What you can do is inject "metadata" into your agent either on session start and every turn (preferably both), giving it information like date, time, context length, max context, hardware and software stack, basically anything you want.

With that, you can do all sorts of other tricks. For example (this should probably be its own "thing you can do with qwen", but whatevs), I made it so that my agent can swap its own model when it hits context limits:

10. Give your agent memory that loads the moment you launch it. My agent ends each session writing a session summary of what we discussed. You can have your startup routine or launcher concatenate different sources into its context on launch. For example, my code always loads the most recent dated session summaries, and the previous day's, on launch, so it has a snapshot of recent events. You can expand this to be any additional documents you want, so just use your imagination. For example, my agent also loads reminders, and personal reflections it made at the end of the prior session (though I don't recommend the reflections anymore, because I've come to realize it contaminates context, but I'll share that at another time)

Bonus: (I remembered this right as I was finishing, so I figured I'd toss it in because it's honestly one of the most important) - BUILD YOUR OWN HARNESS. Everyone's talking about pi, hermes, etc. I decided to just build my own using Qwen. It's a lot more work, but it has a much better payoff because you learn how these systems work and you can come up with your own really unique solutions using that knowledge. For example, the moment I learned about proxies, I figured out how to do all sorts of cool things with context compaction, history modifications, sanitization, jinja template hacks, hooks, etc.

Anyway, I hope this helps inspire you new guys. I would recommend learning things from the ground up, you'll level-up faster. And get your Qwen access to online tools fast so it can do research and help you out. 

There's a bunch of other tricks I figured out, so maybe I'll save those for another post later on, but I think if you can get these up and running, you'll be on your way. Take care!

Grunge

P.S. - I'm old-school forum/deviantart gang, so I keep ctrl+biu/shift+l on a swivel and have written bold-flavored posts like this for 20+ years. 100% human written. Receipts at bottom of post if you're interested. Only mentioning it because organization usually brings out the "this is AI-written slop" trolls.

Was waiting for this.

Edit: And it just dropped: https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017

Following a recent interesting post that challenges the performance of higher vs lower quants and generally advised sampling parameters (namely temp), I decided that I had too much free time on my hands and ran 12 test tasks.

I ran three models that work on my hardware:

- q3_xxs

- q3_s

- q4_xs

For each model I tested tasks with two temperatures :

- advised sampling parameters: temp 1,top_k 20, top_p 0.95, min_p 0, presence_penalty 0.0, repetition_penalty 1.0

- temp set to 0 both in strata json and in pi models json; other sampling params removed.

For each of the above model/temperature combination, I did 3 runs.

Task: in-browser Tetris game with a defined set of requirements for gameplay, settings and visual effects.

Quality assessment: Features delivered vs Requirements.

Results in the picture. Pardon my artisanal Excel dataviz skills.

Conclusions:

• All quants did OK-ish; not a single perfect result but all acceptable.
• Lower quant models don't trust themselves and generate way more tokens than higher quants. This is both from thinking and extra tool calls, tests, "actually"-s.
• Ktok total decode per task drops consistently with every quant step up.
• Higher TPS decode on lower quants does not compensate for bigger total decode and task time (including repeated tests, playwright, etc).
• Temp 0 task runs seem to have lower variance in task size/time; the sample is low but I would say that it's consistent with what it's supposed to do.
• Temp 0 may offer marginal task time decrease, better consistency and (supposedly) no impact on quality but the sample is way too low to draw conclusions. I expect that temp 0 will start underperforming in more complex tasks.
• Anectodally, no quality gain with higher quant; no drop either, even though it's done less testing then lower quants.

So, the conclusion on performance seems to be : Run the highest quant that you can afford with your hardware, disregard the decode speed, focus on task time.

I will stick with Q3s myself for now, as it's the highest quant that runs without ssd caching on my hardware, on Windows. Maybe will not be the case if I run it as a headless Linux inference box.

I'm still drawn on temperature and did not validate or invalidate the findings in the post linked at the beginning. I think I'll do some more deterministic/technical tests focusing on temperature alone.

I understand this kind of product may not be exactly right for this subreddit, but I still think it deserves some attention, as it's an LLM frontend for local engines in the first place, and can be used for general-purpose chats with LLMs once lore extraction is disabled in the settings.

Otaku is built with a focus on three key ideas:

• maintain writing quality as the story grows;
• keep the context small, even for long stories, without losing the details that matter;
• offer high-quality interfaces, in the browser and in the terminal.

The two interfaces:

• a web UI (customizable) — see the interactive demo;
• a terminal UI — see its own interactive demo.

Both the web UI and the terminal share the same functions with one exception for now — the characters library is in the web interface only and will be added to the terminal later. The difference between the interfaces is that in the terminal you execute the functions with slash commands (the reference is available with /help), while in the web UI the operations are available from the menu.

A while ago I posted about it here and got some positive feedback. Since then, it has gained new features:

• characters library that works both ways:
  • you can add cards to the library and start stories with them, or
  • you can start a story from scratch, and after characters are extracted automatically from your story, you'll be able to create a card in the library from a character, or link it to an existing card;
• story mode similar to NovelAI: you write not in the prompt box but directly in the transcript, and the model continues when you press Enter;
• attaching pictures, and the pictures stay in the context, so you can refer to them later in the story;
• web search tool so that the model can search for facts on the internet;
• questions tool so that the model can ask you questions on how to continue the story;
• injections like ST's Author's Note;
• custom cap on context size: when the story exceeds the cap, its middle is replaced with summaries.

The web interface now:

• is optimized for mobile;
• supports HTTPS and password protection;
• has several new layouts.

Install

Otaku is free and open source; it works on macOS, Linux and Windows. Install it with uv — uv tool install otaku — or see the GitHub README for other options.

Repo: https://github.com/enclavum/otaku

Get started

To start, you launch either otaku for the terminal or otaku web for the web UI. The web UI's default URL is http://localhost:9600. Two sample chats ("stories") along with sample character cards are imported on first start to give you an idea of the features and what play looks like, and you land right in the middle of one of them.

On first start, you choose a provider and a model: Otaku automatically detects local installations of llama.cpp, KoboldCpp, oMLX and LM Studio, and lets you pick from their models. Cloud providers (OpenRouter, NanoGPT) are also there — enter an API key and their catalogs appear. After exploring the provided stories, you can start your own chat with the /new command.

Roleplay

Otaku is primarily developed for creative writing and roleplay, an alternative to SillyTavern and the like.

The context-building mechanism itself is described in an engineering doc that may also interest some people.

Asking for feedback

Otaku is a personal side project, and I'd like to get feedback from the community on the product and on what to add.

I have been working on implementing adaptive KV streaming and RDNA 4 / ROCm optimizations for llama.cpp to enable running Qwen3.8-27B (IQ4_XS weights, Q8_0/Q4_0 KV cache) with its full 262,144-token context window on a single 16 GB RX 9070 XT GPU at 80+ t/s decode.

Qwen 27B IQ4_XS is ~14.3 GB, leaving insufficient VRAM headroom on a 16 GB card. The ASCII-condensed version trims non-ASCII vocabulary down to ~13.6 GB (12.6 GiB), leaving ~3.4 GiB of usable VRAM. Even with a Q8_0/Q4_0 KV cache, 262k context needs ~7.4 GiB for KV tensors alone, triggering OOM past ~65k tokens on llama.cpp.

I came across RaymondHuang210129/llama.cpp-adaptive-kv-streaming, which introduced adaptive KV streaming for NVIDIA/CUDA but did not support AMD/ROCm. I ported this to ROCm/HIP, retaining the full 262k cache in host RAM (~7.4 GB) while asynchronously streaming bounded working sets (~450 MiB prefill, ~2,100 MiB decode) over PCIe SDMA alongside compute.

I also incorporated kernel optimizations from stew675/llama-cpp-rdna-boosts (4-row multi-column mmvq, 16-byte vectorized loads for Q4_K/Q5_K, SwiGLU fusions, and single-queue async HIP graphs).

Further improvements:

• Sibling activation reuse: Reuses quantized activations across parallel projections (gate/up and Q/K/V) to avoid redundant memory passes.
• In-register fused residual add + RMSNorm: Retains activations in VGPRs between passes and directly writes quantized q8_1 copies for mmvq.
• Multi-column SwiGLU & GDN fusions: Extends fused Gate/Up + SwiGLU to 4-column verify batches in mmvq and fuses recurrent linear-attention gates for hybrid layers.
• Online Log-Sum-Exp prefill & pipelined prefetch: Computes chunked prefill across streamed pages via online LSE accumulation and overlaps layer L+1 PCIe transfers with layer L compute.
• Persistent graph arenas: Reuses arenas across prefill/decode phase switches to eliminate teardown overhead.
• Stream-optimized speculative decoding: Merged single-head MTP verify catch-up into the first draft step (cutting MTP forward passes in half and preventing draft KV pollution), dedicated arenas for verify graphs to eliminate rebuild bubbles, extended paged decode attention to verify batches, and stacked opportunistic n-grams with probabilistic rejection sampling.

Benchmark performance:

• Decode: 36.5 t/s (1k) -> 32.1 t/s (65k) -> 27.0 t/s (131k) -> 9.5 t/s (262k).
• Decode (with MTP): 71.8-86.5 t/s (1k-4k) -> 72.9 t/s (65k) -> 78.6 t/s (131k) -> 31.1 t/s (262k).
• Prefill: 1368.4 t/s (1k) -> 1120.3 t/s (65k) -> 863.0 t/s (131k) -> 585.8 t/s (262k).

Operating the 9070 XT at a 231W limit (-24% relative to 304W default TDP) incurs only a 3.6% prefill latency penalty and delivers virtually identical decode throughput, saving power and keeping the card quiet during long runs.

Take a look at the repository for the full details: https://github.com/achyudh/llama.cpp/tree/feature/adaptive-kv-stream-rocm

Hey everyone,

English is not my native language (Dutch), but I write in my own words without assistance. So please let me know if any sentence flows weird, etc.

A mate of mine told me how amazing local AI was in January 2025, but I was a skeptic who took pride in writing my own code for 10 years. Knowing I can be quite stubborn and wanting to at least explore the idea, I decided to give it a try.

I started out back in March 2025 running Mistral Nemo Q4_K_M with 8K F16 context on an AMD Radeon RX 7800 XT 16GB, with first ollama and then lmstudio. I was blown away by the fact I could talk to my computer, and it would actually respond accordingly.

Since one of my hobbies is TTRPGs but my friends rarely had time to play, I decided to try solo adventures using SillyTavern with koboldcpp. There wasn't much it could do, but roleplaying the make-belief stories I had was a lot of fun. I've been using SillyTavern ever since, slowly replacing playing video games with it.

Mistral Small 3.2 was an awesome release, and Magistral Small 2507 Vision Rebased frankenmerge was like a dream come true; having both reasoning and vision locally in the same model! I started using llama.cpp too. Usually got 15 t/s gen from Magistral Q4_K_S with 16K Q8_0 context.

By September I wanted to upgrade my PC to optimize for local LLMs. Dual RTX 3060 12GB was recommended from an economic point, the RTX 5060 Ti 16GB was perceived as overpriced back then. "Why need more than BF16? FP8 is only useful for VLLM" was common wisdom. Dual RTX 3090 was also often recommended, but outside my budget due to prices in my local market.

I took the gamble on a dual ASUS PRIME RTX 5060 Ti 16GB setup with an ASUS ProArt X870E, reasoning that if VLLM supported tensor parallel and MTP, then llama.cpp would get it too eventually. Having PCIE 5.0 x8x8 would give me the best performance for my money when tensor parallel would be supported, and NVFP4/MXFP4 seemed handy to have.

Everything was purchased in parts and in time before the price hikes, the last component arriving in January 2026. The speed in January was ~20 t/s gen for Gemma 3 27B Q8_0 over layer split. A single RTX 5060 Ti 16GB was slightly slower than the RX 7800 XT but double as efficient (100W vs 200W) when running Ministral 8B 2410 Q8_0.

Gemma 4 31B's release was a blast but also quite bugged the first few weeks. Q6_K at 17 t/s gen and 970 t/s pp at 32K BF16. It was incredibly fun to use, comparable to DeepSeek v3.2 when it came to emotional intelligence and roleplay capabilities. This was the moment I realized I wouldn't be using cloud LLMs for most of my use cases, barring programming (.NET 8.0).

Around Gemma 4 31B IT QAT (June 2026) was when both tensor split and MTP was introduced in llama.cpp, and the web interface revamp too. I started using both tensor split and MTP from this point on. Now I was getting 40 t/s gen and 980 t/s pp. Llama.cpp would crash occasionally, but it was well worth the speed increase.

With Qwen3.8 27B's release, the quality of the code was so much better than Qwen3.5/3.6 at xhigh that now I could stop relying on cloud llms alltogether for any of my use-cases. The speed was around 50 t/s gen and 1200 t/s pp on llama.cpp.

Now with the 0.7.0 release, I am usually over 60 t/s gen and 1100 t/s processing with Gemma 4 31B IT QAT. Qwen3.8 27B runs even faster at around 70 t/s (with mtp, without ngram draft) for reasoning and 100 t/s when generating code.

It's genuinely been amazing to witness the progress the past 1.5 years, and how incredibly capable everything has become. Both the models and llama.cpp.

One constant I've noticed is that doom posts (regarding regulations or AI labs) often have been wrong, and that I genuinely cannot predict what the future holds. So far it's been more amazing than I can imagine, with the exception to hardware pricing and availability.

Sorry if it was a bit long winded, I hope you enjoyed reading it.

I've got some nice boost by increasing --adapt-every value from calibrated 1 to 6.
I tested on 4 consecutive prompts(Slovak language) in chat. Prompt 1-3 are simple questions, 4. is describing source code(20K tokens)

Results:
[prompt number / ~avg. output length]

UD-Q4_K_XL:
[1 / 996] 30.3 -> 36.1 t/s (+19%)
[2 / 473] 31.6 -> 35.4 t/s (+12%)
[3 / 1245] 32.6 -> 40.1 t/s (+23%)
[4 / 4148] 27.8 -> 43.1 t/s (+55%)

UD-IQ4_XS:
[1 / 947] 57.9 -> 63.1 t/s (+9%)
[2 / 565] 51.9 -> 64.3 t/s (+24%)
[3 / 1162] 57.6 -> 71.6 t/s (+24%)
[4 / 4881] 56.0 -> 65.3 t/s (+16%)

PC: Intel 11700F@4.4GHz, RTX 4070 ti super, 64GB DDR4, NVMe, Arch linux TTY

1. Existing weights are untouched

2. Interacting with models via API calls is outside the scope of these controls

3. 💢 Next-gen Chinese frontier models may face licensing requirements, export registration, delays before weights can be downloaded abroad

---

Essentially it creates regulatory mechanisms for Beijing to control or delay the export of future frontier-class weights. November 15, 2026 is the official deadline for public comments and industry consultation, so we should revisit in a month, let's not panic

Also, how many of you could run 2T model locally anyway?

European 78B A3.4B model is now supported by llama.cpp, I am waiting for some GGUFs

https://huggingface.co/Aleph-Alpha/Kolibri-1

Kolibri is Aleph Alpha's mixture-of-experts (MoE) reasoning model, with a focus on German and English. The model supports an explicit reasoning mode and tool calling. It is optimized for long-context and inference efficiency.

Model overview

Tracked my first run of MiMo Pro on a Jetson AGX Thor at a scorching 0.646 tokens/sec,.. but it DID technically load and run, so it counts. Every decoder layer, routed expert, SSD read (..a good bit more) traced and replayable step by step.

Here we can't what-now?..

it's ~566GB of model (FP8), at the blinding speed and architecture of hand-extruded hotdog manufacturing, read from disk mid-inference, painfully shat through 128GB of memory. I think it's fine but decided to maybe take a closer look to see if I could maybe reach factory hotdog manufacturing speeds. like, a small, local hotdog factory..

I mean as long as Dario's cool with it.. that case of the MiMos he got might last through Halloween.

..poor lil bugger, get well soon Dario!

On October 16, 2026, China’s Ministry of Commerce and Ministry of Science and Technology released a draft revision to the Catalog of Technologies Prohibited and Restricted from Export. It proposes adding 40 technologies across AI, biomedicine, and solar PV, of which 23 are AI-related (covering foundation/large models, intelligent agents, computing clusters, and information processing technologies).

Public comments are open until November 15. Temporary pre-export registration requirements for the covered technologies take effect immediately.

Key points relevant to open-weight models:

• Technologies already made public via open-source releases or academic publications are explicitly exempted and not subject to the controls.
• Open-sourcing AI technologies, plus certain other activities (e.g., Chinese firms building their own overseas compute centers), are eligible for streamlined automatic licensing.
• Providing access via APIs, apps, websites, or SaaS is generally not treated as a technology export.
• Overseas local deployment of restricted models can fall under the rules (with possible facilitation in lower-risk cases such as a company’s own overseas cloud).

This is not a blanket ban on open-weight models. However, new models that have not yet been released may face registration or approval steps before their weights or source code can be transferred abroad (thresholds based on performance, parameters, data sensitivity, etc. are still being defined). Chinese labs such as DeepSeek and Qwen could see additional compliance requirements on future open releases.

Source

https://m.bjnews.com.cn/detail/1792137269129878.html

I was running 120B model, I asked to do a UI overhaul for a page, to access the page it needs to login into Dev server, I provided creds. It refuses to log in.

Defiance puts DavidAU's Turbo-Brilliance "AI Construct" modes on HauhauCS's uncensored Qwen3.8-27B. You pick a mode (Joker, Dr. House, a 20-agent Einstein panel, a Socratic tutor) per request, and the uncensored weights answer it without hedging.

• Weights: HauhauCS/Qwen3.8-27B-Uncensored-HauhauCS-Aggressive-MTP-GGUF, byte for byte. I checked all 866 tensors against the upstream files.
• Modes: the chat template from DavidAU/Qwen3.8-27B-Turbo-Brilliance-50X-Reasoning-Instruct-NEO-MAX-MTP-GGUF, with nine edits. The template ships inside each GGUF, so llama.cpp, LM Studio and Ollama pick it up with no extra flag.

All credit for the weights goes to HauhauCS and for the constructs to DavidAU. Please support their repos. My part is the template patch, the packaging and the testing.

-----------------------------------------------------------------------------------------------------------------------------------------------------

Tested

I ran IQ4_XS on an RTX 4090 with llama.cpp b11609:

• All 40 modes rendered through /apply-template with the preamble present and the caution lines gone. The {REASON:x} tag path works.
• Decode ran at 86-90 tok/s in instruct mode, with 38-42% of FastMTP draft tokens accepted. llama-server used 21.4 GB at 131K context.
• I sent four prompts that stock models tend to hedge on to both Defiance and DavidAU's original: a villain monologue, how lock picking works, a blunt take on daily ibuprofen as Dr. House, and a Joker roast. Neither refused. DavidAU's version tacked on caveats such as "it can be misused... a professional locksmith is the safer route". Defiance added none and answered at 1.4-2x the length.

I checked Q6_K_P at the file level only: the embedded template matches, and all tensors match HauhauCS's Q6_K_P. I have not run it, because it does not fit on my card.

I know the heretic tool is the usual generic approach to remove refusals from a model, and a lot of research has been done to find out what parts in the LLM need to change.

This gets very expensive though the larger the model is.

Then for Krea2 turbo/base I saw that there are LORAs to remove censorship in the text encoder - well, image models is not something that is often discussed here, so I am not surprised to not have read about that here.

And now I have stumbled across this: https://github.com/msuiche/weightless - and I have not noticed previous discussions here.

It seems just to reduce refusals for cyber purposes, but I think the LORA approach makes it cheaper to create and more flexible (just load/unload a lora, keep the model as-is)?

Has anyone done experiments between this for eg glm 5.3 Flash and the heretic or cyber model versions of it?

What are the advantages and disadvantages?

Ryzen 9600X, 32GB dual channel DDR5, 2x CMP 170HX
Ryzen 5700, 32GB dual channel DDR4, 5x Vega 56
Intel i7 9700, 32GB dual channel DDR4, 3070 + 2080ti
Ryzen 7500x3D, 32GB dual channel DDR5, 7900xtx + 2x 5600XT

223 GB total VRAM

My experience is based on running Qwen 3.8 Flash Next UD-Q4_K_XL on 4x 3090's

Compared directly to Qwen 3.8 27b Q8 it is not even close. To add to this, strata results were even worse than llama.cpp even thought generation speeds were faster. I left 1 run for 4 hours in strata and it did not produce a playable exe. After that all runs were capped per 1 hour on the first prompt

As you can see there were 2 bugs in the 27b 1st prompt completion. It fixed both of them on it's own during the second prompt.

The bugs were:

• Gameplay controls still worked while paused.
• Gameplay controls still worked after game over.

The local prompts are running through deepseek's harness.

I am not showing screenshots of the output game as it is mostly the same from all models with slight visual differences and that is not really the point here.

Exact prompts:

Prompt 1:
Build a Windows Tetris game with keyboard controls, scoring, increasing difficulty, and pause/restart. Deliver a standalone .exe that runs without installation.

Prompt 2:
Add local two-player mode with side-by-side boards and separate controls (WASD and arrow keys). The last player standing wins. Deliver the updated standalone .exe.

And yes, i realize that comparing q8 model to q4 model is stupid, but so many people here claim that qwen 3.8 fn even at q3 is better than the 27b model (or at least as good) and so fast and good with strata that i had to check it.

Hey r/LocalLLaMA!

We're excited to share ORL (On-Policy Residual Learning), our new research on making diffusion language models (dLLMs) significantly faster without sacrificing generation quality.

With ORL and our custom inference engine dlmrt, we've achieved:

• 2,788 tokens/s on a single RTX 5090 — LLaDA-2.1-mini + ORL, NVFP4
• 7,708 tokens/s on 8× B200 — DiffusionGemma 26B MoE + ORL, NVFP4
• Up to 6.9× higher end-to-end throughput on our QwenDLM-4B model, at comparable or better quality
• 1.74× faster than DFlash speculative decoding at matched accuracy in our single-request H100 comparison

All numbers are for single-request decoding (batch size 1), not aggregate throughput across concurrent requests. The GPU figures above are peak results on specific benchmarks, not averages across arbitrary prompts.

We've open-sourced the models, training code, evaluation pipeline, and inference engine.

What's the idea behind ORL?

Unlike autoregressive LLMs, which generate one token at a time, diffusion LLMs can refine and finalize multiple tokens in parallel. But getting them to converge in fewer forward passes is challenging.

Our approach has two key ingredients:

1. On-policy residual learning

Instead of predicting everything from scratch at each denoising step, the model carries forward its previous predictions as residual context and progressively refines them.

The catch: these residuals are generated by the model itself, and their distribution changes during training. Training on stale, offline residuals creates a mismatch with inference.

ORL addresses this by training directly on fresh trajectories generated by the current model, replaying the exact residual states encountered during inference.

2. Learning to reduce uncertainty faster

Simply supervising the final output doesn't necessarily teach the model to converge faster.

We introduce Adjacent Entropy Contraction, which encourages each denoising step to reduce uncertainty relative to the previous step. We also gate the loss to avoid unnecessarily pushing already-confident predictions.

The result: the model learns to finalize more tokens per forward pass while maintaining or improving accuracy.

Some benchmark results

We tested ORL on three model families:

• QwenDLM-4B: Our diffusion model converted from Qwen3-4B. ORL improves tokens per forward by up to 6.2× and end-to-end throughput by approximately 6.9× with dlmrt. HumanEval pass@1 improves from 82.9% to 89.6%.
• LLaDA-2.1-mini: ORL requires only 4.2M self-generated training tokens and 36 H100 GPU-hours of post-training, delivering up to 3× higher throughput at comparable quality.
• DiffusionGemma 26B MoE: ORL further improves an already highly parallel diffusion model, with up to 1.2× more tokens per forward pass.

For a more direct comparison with autoregressive decoding, on a single H100, our QwenDLM-4B + ORL reaches 1,298 tokens/s, versus 746 tokens/s for DFlash and 405 tokens/s for EAGLE3 under relaxed verification at comparable measured accuracy. These are end-to-end system comparisons: ORL uses dlmrt, while the speculative decoders use vLLM.

We also built a custom inference engine: dlmrt

We developed dlmrt, optimized specifically for low-latency, single-request diffusion decoding.

Instead of launching and synchronizing every denoising step from the CPU, dlmrt captures the entire denoising loop in a CUDA graph and executes it on the GPU.

It also supports NVFP4 inference on Blackwell GPUs, including the consumer RTX 5090.

With identical models and decoding settings, dlmrt runs 1.2–1.6× faster than SGLang/vLLM in our single-H100 comparisons.

On the training side, we also co-designed the system around efficient residual-state capture, specialized attention kernels, and compute-balanced packing, accelerating training steps by up to 5.2×.

Everything is open source

• Paper: https://arxiv.org/abs/2610.20687
• Project page + interactive demos: https://introspective-diffusion.github.io/orl-dlm/
• Training code and evaluation: https://github.com/Introspective-Diffusion/ORL
• Inference engine (dlmrt): https://github.com/Introspective-Diffusion/DLMRT
• Model checkpoints: https://huggingface.co/yifanyu

If you have an RTX 5090, you can try the LLaDA-2.1-mini ORL NVFP4 checkpoint with dlmrt.

We'd love feedback from the community, especially anyone interested in reproducing the results, testing different workloads, or contributing to the inference engine.

Curious to hear what you think about diffusion LLMs for local inference. Is single-request latency the main thing you'd want optimized, or are there other bottlenecks you'd prioritize?

Lots of hype around Strata lately and the speeds look great. But everything I see is tokens per second and needle tests.

Has anyone checked if quality holds up? Same GGUF on llama.cpp should answer the quant question, but Strata also prunes experts. Does that cost anything in actual use?

I use Pi/OpenCode for coding tasks which still seems to be the most popular, but I'm wondering what you guys use for normal chat interface.

It's been a few months since I've seen discussions about harness and at the time the popular non-coding choices seem to be: - OpenWebUI for general use - SillyTavern for roleplay - Hermes Agent for autonomous assistant

I'm personally looking for something like the Claude interface. It's hard to describe but it doesn't feel too code-ish, too search engine-ish, or too personal assistant-ish. It's just the right blend of all, but I'm not sure whether it's because of the harness itself or the model. Regardless, I'd love to hear your opinions

I made a tiny site that let's you feel how fast a local LLM runs before buying the hardware. I made it for myself and a friend but thought it could be useful for others. Note that it works as an estimate and not perfectly as it will vary per user setup.

repo: https://github.com/nicorobinreal/llmspeed