I know everyone's excited about 3.8, but honestly I'm still refining 3.6. And I hate the way 3.8 talks, so...

I keep seeing all these posts about what's the use of open source, so I figured I'd submit my own answer with this post.

Qwen 3.5 27B was my first agent, starting my journey around May 31st, and I've learned a lot in these 4+ months, so I wanted to share with other people just starting their journey because there are clearly a LOT of new users out there, and people want help but sometimes are afraid to ask or not sure the right questions. So, here are ten things that you should know you can do to empower you!

(Oh, and I'd recommend using llama.cpp. That's what I've used to do all these tricks)

1. Create neat little apps for yourself. Qwen's really good at making apps for anything. I'm writing this post in a app I made as a replacement for Notepad, that has full word processing, side-panel, light/dark mode, etc. (image below). The first thing you should be thinking about when you run into any sort of issue is, "can I make an app that will make this simpler?" The answer is usually yes. 

2. Create your own Chat UI frontend. If you don't feel that creative, just copy an existing one. LLMs are great copy cats. I liked claude, so I just copied it as my own frontend, customizing it for my own needs:

3. Create ReAct Loops. This is what enables your model to use a tool, read the result, then use it again, multiple times. Qwen can make these in its sleep, and many different ways. It can create a loop anywhere you want it, and you'll need it for your own custom chat ui frontend if you make one.

4. Create MCP tools. This is my favorite. It can make almost any type of mcp tool you need. You don't have to use third party ones, which can be unsafe. Just tell it you want an mcp tool to do x, it'll make it. It might or might not require its own mcp server, but if it does, Qwen can make that too.

5. Spawn sub-agents. Another super-easy thing Qwen can do. Just tell it to create a sub-agent spawner, which is just a python file. Then, you can tell it to run that when it needs to create a sub-agent, or tell it to create an mcp tool to do it on-the-fly when it wants to. Simple. Oh, and a quick hack-tip: mcp tools use up a LOT of context, so a cheat is to spawn a sub-agent with its own context limit and have your main agent pass off work to the sub. Basically, you can double/triple/etc your context limit. And yes, you can retain it between turns, but you'll need to create a pipeline runner (see below).

6. Create a pipeline runner. Ever heard of n8n? Basically, it's a pipeline runner. It has a lot of features, but it has a few limitations and restrictions, and there were a few things I wished it could do. I learned I could just build something similar with Qwen super easy. There's some design work involved if you want a UI frontend to build, but it's completely doable. I made my own ui, but after a while, I got tired of tweaking it, so I just copied n8n's frontend and just customized it myself, lol. Another useful tip: You can run multiple sequential agents in a pipeline runner, but if you want them each to have their own context, you'll need to ensure cache-idle-slots is enabled in llama.cpp. This'll swap the kv caches each turn, ejecting the last one to cpu ram. There's a bit more to it, but Qwen's got you, so just ask.

7. Create a model launcher (with options to select different models/jinja templates in llama.cpp). This is my bread and butter. Its a launcher that lets me choose any model in my directory and set its parameters. When I'm happy, I create an alias for it. On next launch, I can just choose the alias and it launches the model with the params for that alias only. So I can have 10 different types of param configurations for the same model if I want; I just need to give them their own alias. Hack tip: I'd recommend adding your jinja templates to the launcher to, and you can choose different custom templates for different model types.

8. Install, test, fork, and/or rebuild open-source apps on the internet. The reason I moved to llama.cpp was lm studio pp was too slow, but I didn't want the stress of figuring it all out, so I just asked my agent to download and install llama.cpp for me. I've been using it ever since. It can also modify builds, fork repositories on github, and a lot more. For example, I was busy working on a project, but heard about some new TTS apps, so I told my agent to download and install Moss and OmniVoice, make some audio clips to make sure they run, and let me know when it's done. Of course everything worked fine. Great for multi-tasking.

9. Give your agent temporal awareness of the world around it. Ever used cloud models that argue with you about the date? Or wished your model knew when you were nearing your max context, so it could automatically start writing a session summary? What you can do is inject "metadata" into your agent either on session start and every turn (preferably both), giving it information like date, time, context length, max context, hardware and software stack, basically anything you want.

With that, you can do all sorts of other tricks. For example (this should probably be its own "thing you can do with qwen", but whatevs), I made it so that my agent can swap its own model when it hits context limits:

10. Give your agent memory that loads the moment you launch it. My agent ends each session writing a session summary of what we discussed. You can have your startup routine or launcher concatenate different sources into its context on launch. For example, my code always loads the most recent dated session summaries, and the previous day's, on launch, so it has a snapshot of recent events. You can expand this to be any additional documents you want, so just use your imagination. For example, my agent also loads reminders, and personal reflections it made at the end of the prior session (though I don't recommend the reflections anymore, because I've come to realize it contaminates context, but I'll share that at another time)

Bonus: (I remembered this right as I was finishing, so I figured I'd toss it in because it's honestly one of the most important) - BUILD YOUR OWN HARNESS. Everyone's talking about pi, hermes, etc. I decided to just build my own using Qwen. It's a lot more work, but it has a much better payoff because you learn how these systems work and you can come up with your own really unique solutions using that knowledge. For example, the moment I learned about proxies, I figured out how to do all sorts of cool things with context compaction, history modifications, sanitization, jinja template hacks, hooks, etc.

Anyway, I hope this helps inspire you new guys. I would recommend learning things from the ground up, you'll level-up faster. And get your Qwen access to online tools fast so it can do research and help you out. 

There's a bunch of other tricks I figured out, so maybe I'll save those for another post later on, but I think if you can get these up and running, you'll be on your way. Take care!

Grunge

P.S. - I'm old-school forum/deviantart gang, so I keep ctrl+biu/shift+l on a swivel and have written bold-flavored posts like this for 20+ years. 100% human written. Receipts at bottom of post if you're interested. Only mentioning it because organization usually brings out the "this is AI-written slop" trolls.

Ryzen 9600X, 32GB dual channel DDR5, 2x CMP 170HX
Ryzen 5700, 32GB dual channel DDR4, 5x Vega 56
Intel i7 9700, 32GB dual channel DDR4, 3070 + 2080ti
Ryzen 7500x3D, 32GB dual channel DDR5, 7900xtx + 2x 5600XT

223 GB total VRAM

During a post-event interview at the RTX Spark Launch Party: Taipei on October 16, 2026, Nvidia's Senior Vice President of Personal Computing, Jeff Fisher, explicitly debunked the recent discontinuation rumors.

Tracked my first run of MiMo Pro on a Jetson AGX Thor at a scorching 0.646 tokens/sec,.. but it DID technically load and run, so it counts. Every decoder layer, routed expert, SSD read (..a good bit more) traced and replayable step by step.

Here we can't what-now?..

it's ~566GB of model (FP8), at the blinding speed and architecture of hand-extruded hotdog manufacturing, read from disk mid-inference, painfully shat through 128GB of memory. I think it's fine but decided to maybe take a closer look to see if I could maybe reach factory hotdog manufacturing speeds. like, a small, local hotdog factory..

I mean as long as Dario's cool with it.. that case of the MiMos he got might last through Halloween.

..poor lil bugger, get well soon Dario!

Hey r/LocalLLaMA!

We're excited to share ORL (On-Policy Residual Learning), our new research on making diffusion language models (dLLMs) significantly faster without sacrificing generation quality.

With ORL and our custom inference engine dlmrt, we've achieved:

• 2,788 tokens/s on a single RTX 5090 — LLaDA-2.1-mini + ORL, NVFP4
• 7,708 tokens/s on 8× B200 — DiffusionGemma 26B MoE + ORL, NVFP4
• Up to 6.9× higher end-to-end throughput on our QwenDLM-4B model, at comparable or better quality
• 1.74× faster than DFlash speculative decoding at matched accuracy in our single-request H100 comparison

All numbers are for single-request decoding (batch size 1), not aggregate throughput across concurrent requests. The GPU figures above are peak results on specific benchmarks, not averages across arbitrary prompts.

We've open-sourced the models, training code, evaluation pipeline, and inference engine.

What's the idea behind ORL?

Unlike autoregressive LLMs, which generate one token at a time, diffusion LLMs can refine and finalize multiple tokens in parallel. But getting them to converge in fewer forward passes is challenging.

Our approach has two key ingredients:

1. On-policy residual learning

Instead of predicting everything from scratch at each denoising step, the model carries forward its previous predictions as residual context and progressively refines them.

The catch: these residuals are generated by the model itself, and their distribution changes during training. Training on stale, offline residuals creates a mismatch with inference.

ORL addresses this by training directly on fresh trajectories generated by the current model, replaying the exact residual states encountered during inference.

2. Learning to reduce uncertainty faster

Simply supervising the final output doesn't necessarily teach the model to converge faster.

We introduce Adjacent Entropy Contraction, which encourages each denoising step to reduce uncertainty relative to the previous step. We also gate the loss to avoid unnecessarily pushing already-confident predictions.

The result: the model learns to finalize more tokens per forward pass while maintaining or improving accuracy.

Some benchmark results

We tested ORL on three model families:

• QwenDLM-4B: Our diffusion model converted from Qwen3-4B. ORL improves tokens per forward by up to 6.2× and end-to-end throughput by approximately 6.9× with dlmrt. HumanEval pass@1 improves from 82.9% to 89.6%.
• LLaDA-2.1-mini: ORL requires only 4.2M self-generated training tokens and 36 H100 GPU-hours of post-training, delivering up to 3× higher throughput at comparable quality.
• DiffusionGemma 26B MoE: ORL further improves an already highly parallel diffusion model, with up to 1.2× more tokens per forward pass.

For a more direct comparison with autoregressive decoding, on a single H100, our QwenDLM-4B + ORL reaches 1,298 tokens/s, versus 746 tokens/s for DFlash and 405 tokens/s for EAGLE3 under relaxed verification at comparable measured accuracy. These are end-to-end system comparisons: ORL uses dlmrt, while the speculative decoders use vLLM.

We also built a custom inference engine: dlmrt

We developed dlmrt, optimized specifically for low-latency, single-request diffusion decoding.

Instead of launching and synchronizing every denoising step from the CPU, dlmrt captures the entire denoising loop in a CUDA graph and executes it on the GPU.

It also supports NVFP4 inference on Blackwell GPUs, including the consumer RTX 5090.

With identical models and decoding settings, dlmrt runs 1.2–1.6× faster than SGLang/vLLM in our single-H100 comparisons.

On the training side, we also co-designed the system around efficient residual-state capture, specialized attention kernels, and compute-balanced packing, accelerating training steps by up to 5.2×.

Everything is open source

• Paper: https://arxiv.org/abs/2610.20687
• Project page + interactive demos: https://introspective-diffusion.github.io/orl-dlm/
• Training code and evaluation: https://github.com/Introspective-Diffusion/ORL
• Inference engine (dlmrt): https://github.com/Introspective-Diffusion/DLMRT
• Model checkpoints: https://huggingface.co/yifanyu

If you have an RTX 5090, you can try the LLaDA-2.1-mini ORL NVFP4 checkpoint with dlmrt.

We'd love feedback from the community, especially anyone interested in reproducing the results, testing different workloads, or contributing to the inference engine.

Curious to hear what you think about diffusion LLMs for local inference. Is single-request latency the main thing you'd want optimized, or are there other bottlenecks you'd prioritize?

Hey everyone,

English is not my native language (Dutch), but I write in my own words without assistance. So please let me know if any sentence flows weird, etc.

A mate of mine told me how amazing local AI was in January 2025, but I was a skeptic who took pride in writing my own code for 10 years. Knowing I can be quite stubborn and wanting to at least explore the idea, I decided to give it a try.

I started out back in March 2025 running Mistral Nemo Q4_K_M with 8K F16 context on an AMD Radeon RX 7800 XT 16GB, with first ollama and then lmstudio. I was blown away by the fact I could talk to my computer, and it would actually respond accordingly.

Since one of my hobbies is TTRPGs but my friends rarely had time to play, I decided to try solo adventures using SillyTavern with koboldcpp. There wasn't much it could do, but roleplaying the make-belief stories I had was a lot of fun. I've been using SillyTavern ever since, slowly replacing playing video games with it.

Mistral Small 3.2 was an awesome release, and Magistral Small 2507 Vision Rebased frankenmerge was like a dream come true; having both reasoning and vision locally in the same model! I started using llama.cpp too. Usually got 15 t/s gen from Magistral Q4_K_S with 16K Q8_0 context.

By September I wanted to upgrade my PC to optimize for local LLMs. Dual RTX 3060 12GB was recommended from an economic point, the RTX 5060 Ti 16GB was perceived as overpriced back then. "Why need more than BF16? FP8 is only useful for VLLM" was common wisdom. Dual RTX 3090 was also often recommended, but outside my budget due to prices in my local market.

I took the gamble on a dual ASUS PRIME RTX 5060 Ti 16GB setup with an ASUS ProArt X870E, reasoning that if VLLM supported tensor parallel and MTP, then llama.cpp would get it too eventually. Having PCIE 5.0 x8x8 would give me the best performance for my money when tensor parallel would be supported, and NVFP4/MXFP4 seemed handy to have.

Everything was purchased in parts and in time before the price hikes, the last component arriving in January 2026. The speed in January was ~20 t/s gen for Gemma 3 27B Q8_0 over layer split. A single RTX 5060 Ti 16GB was slightly slower than the RX 7800 XT but double as efficient (100W vs 200W) when running Ministral 8B 2410 Q8_0.

Gemma 4 31B's release was a blast but also quite bugged the first few weeks. Q6_K at 17 t/s gen and 970 t/s pp at 32K BF16. It was incredibly fun to use, comparable to DeepSeek v3.2 when it came to emotional intelligence and roleplay capabilities. This was the moment I realized I wouldn't be using cloud LLMs for most of my use cases, barring programming (.NET 8.0).

Around Gemma 4 31B IT QAT (June 2026) was when both tensor split and MTP was introduced in llama.cpp, and the web interface revamp too. I started using both tensor split and MTP from this point on. Now I was getting 40 t/s gen and 980 t/s pp. Llama.cpp would crash occasionally, but it was well worth the speed increase.

With Qwen3.8 27B's release, the quality of the code was so much better than Qwen3.5/3.6 at xhigh that now I could stop relying on cloud llms alltogether for any of my use-cases. The speed was around 50 t/s gen and 1200 t/s pp on llama.cpp.

Now with the 0.7.0 release, I am usually over 60 t/s gen and 1100 t/s processing with Gemma 4 31B IT QAT. Qwen3.8 27B runs even faster at around 70 t/s (with mtp, without ngram draft) for reasoning and 100 t/s when generating code.

It's genuinely been amazing to witness the progress the past 1.5 years, and how incredibly capable everything has become. Both the models and llama.cpp.

One constant I've noticed is that doom posts (regarding regulations or AI labs) often have been wrong, and that I genuinely cannot predict what the future holds. So far it's been more amazing than I can imagine, with the exception to hardware pricing and availability.

Sorry if it was a bit long winded, I hope you enjoyed reading it.

Was waiting for this.

Edit: And it just dropped: https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017

My experience is based on running Qwen 3.8 Flash Next UD-Q4_K_XL on 4x 3090's

Compared directly to Qwen 3.8 27b Q8 it is not even close. To add to this, strata results were even worse than llama.cpp even thought generation speeds were faster. I left 1 run for 4 hours in strata and it did not produce a playable exe. After that all runs were capped per 1 hour on the first prompt

As you can see there were 2 bugs in the 27b 1st prompt completion. It fixed both of them on it's own during the second prompt.

The bugs were:

• Gameplay controls still worked while paused.
• Gameplay controls still worked after game over.

The local prompts are running through deepseek's harness.

I am not showing screenshots of the output game as it is mostly the same from all models with slight visual differences and that is not really the point here.

Exact prompts:

Prompt 1:
Build a Windows Tetris game with keyboard controls, scoring, increasing difficulty, and pause/restart. Deliver a standalone .exe that runs without installation.

Prompt 2:
Add local two-player mode with side-by-side boards and separate controls (WASD and arrow keys). The last player standing wins. Deliver the updated standalone .exe.

And yes, i realize that comparing q8 model to q4 model is stupid, but so many people here claim that qwen 3.8 fn even at q3 is better than the 27b model (or at least as good) and so fast and good with strata that i had to check it.

K3 came out in July hyped as the best open model you could download, and it scored 57 on Artificial Analysis, right up there with Opus 4.8 and GPT-5.5. I went back to see what running it actually looks like now, using Unsloth's quant docs, the model card, AA and the license.

Short version: it's still a monster, and the field caught up.

• Full model is 1.56 TB. 104B active doesn't save you any memory, all 2.8T params still have to be loaded.
• Smallest quant Unsloth recommends is 594 GB and wants about 610 GB of RAM+VRAM. It doesn't fit in 512 GB, and Unsloth say they're still trying to get it under.
• 1-bit agrees with the full model on the next token 79% of the time. 2-bit gets you to 90%, but that's 861 GB.
• About 20 tok/s on B200s if it fits in memory. If not, it offloads to disk and crawls.
• It's now 44 on AA's index. Most of that drop is AA changing the index, not the model, but GLM-5.3 (753B) now scores 45 and Step 5 Preview (600B) ties it at 44.
• Slow on the API too. 43 tok/s on Kimi's own API, and on AA's agentic benchmark it averaged almost an hour and over $10 per task.

The bit people miss is the license. It isn't MIT. If your company runs any model-as-a-service business and makes more than $20M in total over any 12 months, you need a separate deal with Moonshot before using K3 commercially, even if K3 itself makes you nothing.

Caveat: AA tests the full model at max reasoning through the API. I couldn't find real task benchmarks for the 1-bit quant, so 79% token agreement is a proxy, not a quality score.

So... is anyone actually running K3 at home? And if you had 600 GB to play with, would you take 1-bit K3 or something like GLM-5.3 at a much higher quant?

PS: I'm a co-founder of Sanbi.ai, where we track how AI assistants like ChatGPT, Claude and more choose and cite sources for brands, and we're expanding into Chinese models, which is why I've been digging into Kimi, DeepSeek and Qwen lately. Happy to talk about how these models search if anyone's curious. The open-weight side is something I follow as an engineer, so this one was more curiosity than work.

I got Qwen3.8-Flash-Next-GSQ-RCO-Abliterated running at IQ3_S with just 12GB VRAM + 32GB RAM + NVME, achieving 40-50 tok/sec decode & 1300-1500 tok/sec prefill @ 131k context (44k thinking budget), on a custom fork of Strata.

Don’t see IQ3_S and instantly dismiss it, this recipe uses an abliterated version of ISTADASLab’s GSQ-RCO Qwen3.8-FN, which retains ~99% of BF16. Obviously there are rough edges with any quantization method but in comparison to all generic imatrix quants, this is the best quant for this model across my testing (for this hardware tier - higher precision uniform quants will perform better obviously). And to have it running at these speeds on only ~44GB combined VRAM/RAM with an SSD is kind of ridiculous.

This fork has tonnes of architectural changes, all are very experimental. But the performance makes up for it. This feels like local Opus in some regards.

Local AI for all.

https://github.com/bodhi37/Qwen3.8-Flash-12GBRAM-32GBVRAM-SSD-Recipe
https://github.com/bodhi37/strata

TL;DR -- Interpretability might be way more tractable than we thought, and the guy who helped convince everyone otherwise is now saying we can understand models by directly studying neurons and their interactions after all. This essay rewrites fundamental assumptions about how neural networks learn and connect, with some of the best interactive visualizations I've ever seen. Studies Qwen3 32B, Qwen3.6-35B-A3B, and Kimi K3 -- along with convolutional vision models.

For years the standard view in AI interpretability has been that neural networks are a tangled mess. Individual neurons seemed to respond to random unrelated stuff (famous example: one neuron fired for both cat faces and car fronts), and the explanation was “superposition,” the idea that networks cram unrelated concepts into the same neurons to save space. If that’s true, you can’t just look inside a model and read it. You have to build a whole second model to untangle it first, which is a big part of why the field went all in on sparse autoencoders.

Nick Cammarata (ex-OpenAI), one of the people who originally helped make that case, now says it was mostly wrong. The cat/car neuron? It’s a V-shape detector: pointy cat faces, pointy car grilles. The “mess” in early models came from weak architectures and buggy visualization tools, not from networks fundamentally packing junk together. Fix those and neurons get clean. When neurons do cover a few things, it’s usually because they branched out to related stuff over training, not random mixtures. His team’s early estimate is that roughly 90% of neurons in a modern Qwen3 32B language model have one main job, with similar results in much bigger models up to Kimi K3's trillion-parameter scale. He even suggests this feature separation becomes more defined as scale increases.

Why this matters: if it holds up, we may be able to understand AI models by just studying them directly, the way you’d study any natural system, rather than first reverse-engineering a decoder for them. Bigger models actually seem easier to read, because their neurons are more specialized. And since AI agents can now do a lot of this investigative grunt work at scale, the author thinks frontier models could become broadly understandable in months rather than decades. For safety, that’s a big deal. Most arguments for why alignment is hard assume we can’t see inside the models. If we can, we can spot problems (say, deceptive reasoning or a bad training environment) and make targeted fixes instead of hoping we got everything right in advance.

I know this is a long essay, but I think their upcoming technical paper will become a seminal work in the fields of interpretability and learning in neural networks (particularly in vision models). Even if you don't read it, the interactive visualizations and network architecture breakdowns are definitlely worth 5-10 min. Especially the ones towards the very bottom of the page. Let's discuss!

Already posted here https://www.reddit.com/r/LocalLLaMA/comments/1x85hfj/strata_qwen_38_flash_next_fast_but_whats_it/phavui9/ , but I believe this topic deserves it's own post.

The PLE, aka the n-gram table that sits on the SSD, is quantized heavily in the common GGUFs and has a measurable impact on performance, demonstrated here by /u/MarkoMarjamaa : https://www.reddit.com/r/LocalLLaMA/comments/1x6kw7k/qwen38_flash_next_unsloth_quants_kldtop1_with/

Further down in the comments, /u/Eyelbee mentions running the full bf16 n-gram table. I asked Grok and apparently that only works in llama.cpp et.al and is called the bf16 sidecar, Strata needs FP8 but that should be close to full quality, over my head to be honest. So I built it with ple_fp8_pack.py and indeed it seems to positively impact results. I run iq3_s and before the change the model made many more stupid mistakes. No impact on speed! Only needs a bit more disk space.

If you want to try this, you need model-00005 up to model-00037.safetensors and model.safetensors.index.json from https://huggingface.co/Qwen/Qwen3.8-Flash-Next, then pack them into a GGUF with

tools/ple_fp8_pack.py --model /media/your_ssd/Strata-data/download_folder --out /media/your_ssd/Strata-data/ple/ple-fp8.gguf

and load it in your .json with

"--ple-gguf",
"/media/your_ssd/Strata-data/ple/ple-fp8.gguf"

Adjust your file paths obviously. Sorry I can't upload 47gigs. :-) Not sure how to properly test performance, but here's two scenes with the same silly prompt.

Build a beautiful 3D scene of a train driving through a futuristic cyberpunk city in HTML. dystopian vibes, detailed models, try not to use too many boxes in the models. Be creative and add some user interactivity like camera and weather controls.

IQ3_S with the included PLE (quantized to IQ4_NL ) https://output.jsbin.com/fojexecanu/

IQ3_S with the full FP8 PLE from Qwen https://output.jsbin.com/hosusuponi/

Imo a clear improvement is visible. I noticed that with Hermes agent too and was able to add several features to a KDE app (haruna). 3D models seem to have less errors like missing faces.

Hi r/LocalLLaMA, we're releasing Tyger, a small model that handles full-duplex turn-control for voice agents.

What it is

Base: NVIDIA Nemotron 3.5 ASR Streaming 0.6B + 14M causal GRU on top.
Every 80 ms, with no lookahead, it picks one of three states:

- answer: the user finished their turn
- stop: the user cut in, so the agent stops talking
- chime in: the agent says “mm-hm”, “oh, really” and keeps listening

It hears both the mic and the agent's own playback, so the agent doesn't mistake its own echo for the user.

tyger-s2s(Apache-2.0)

A full voice agent: Tuger + Gemma 4 E4B FP8 + VoxCPM2.

- Tool calls
- Up to 4 concurrent conversations
- ~0.7 s replies on an RTX 5090 [measured from end of user speech to first audio]
- Optional hosted "expert" [openai], off by default. With it off, everything runs locally and no audio or text leaves your machine.

Limitations
- Tyger itself is small, but the tyger-s2s stack needs 28 GB VRAM. Tested on 5090, otger GPUs are untested; 3090/4090 reports welcome.
- FP32 only, quantization untested
- Offline benchmarks only so far

Links

- Model: https://huggingface.co/nineninesix/tyger
- Agent: https://github.com/nineninesix-ai/tyger-s2s
- Demo (free sign-up): https://www.nineninesix.ai/

Feedback welcome, especially on where turn-taking still feels off.

I RLed qwen27b in style of RL XAR by meta, and now its doesnt feel like a sloppy model and ranks 4 on EQbench

BF16+MXFP4: https://huggingface.co/AlexWortega/Qwen3.8-27B-antislop
GGUFS: https://huggingface.co/rawmodels/Qwen3.8-27B-antislop-GGUF

I understand this kind of product may not be exactly right for this subreddit, but I still think it deserves some attention, as it's an LLM frontend for local engines in the first place, and can be used for general-purpose chats with LLMs once lore extraction is disabled in the settings.

Otaku is built with a focus on three key ideas:

• maintain writing quality as the story grows;
• keep the context small, even for long stories, without losing the details that matter;
• offer high-quality interfaces, in the browser and in the terminal.

The two interfaces:

• a web UI (customizable) — see the interactive demo;
• a terminal UI — see its own interactive demo.

Both the web UI and the terminal share the same functions with one exception for now — the characters library is in the web interface only and will be added to the terminal later. The difference between the interfaces is that in the terminal you execute the functions with slash commands (the reference is available with /help), while in the web UI the operations are available from the menu.

A while ago I posted about it here and got some positive feedback. Since then, it has gained new features:

• characters library that works both ways:
  • you can add cards to the library and start stories with them, or
  • you can start a story from scratch, and after characters are extracted automatically from your story, you'll be able to create a card in the library from a character, or link it to an existing card;
• story mode similar to NovelAI: you write not in the prompt box but directly in the transcript, and the model continues when you press Enter;
• attaching pictures, and the pictures stay in the context, so you can refer to them later in the story;
• web search tool so that the model can search for facts on the internet;
• questions tool so that the model can ask you questions on how to continue the story;
• injections like ST's Author's Note;
• custom cap on context size: when the story exceeds the cap, its middle is replaced with summaries.

The web interface now:

• is optimized for mobile;
• supports HTTPS and password protection;
• has several new layouts.

Install

Otaku is free and open source; it works on macOS, Linux and Windows. Install it with uv — uv tool install otaku — or see the GitHub README for other options.

Repo: https://github.com/enclavum/otaku

Get started

To start, you launch either otaku for the terminal or otaku web for the web UI. The web UI's default URL is http://localhost:9600. Two sample chats ("stories") along with sample character cards are imported on first start to give you an idea of the features and what play looks like, and you land right in the middle of one of them.

On first start, you choose a provider and a model: Otaku automatically detects local installations of llama.cpp, KoboldCpp, oMLX and LM Studio, and lets you pick from their models. Cloud providers (OpenRouter, NanoGPT) are also there — enter an API key and their catalogs appear. After exploring the provided stories, you can start your own chat with the /new command.

Roleplay

Otaku is primarily developed for creative writing and roleplay, an alternative to SillyTavern and the like.

The context-building mechanism itself is described in an engineering doc that may also interest some people.

Asking for feedback

Otaku is a personal side project, and I'd like to get feedback from the community on the product and on what to add.

1. Existing weights are untouched

2. Interacting with models via API calls is outside the scope of these controls

3. 💢 Next-gen Chinese frontier models may face licensing requirements, export registration, delays before weights can be downloaded abroad

---

Essentially it creates regulatory mechanisms for Beijing to control or delay the export of future frontier-class weights. November 15, 2026 is the official deadline for public comments and industry consultation, so we should revisit in a month, let's not panic

Also, how many of you could run 2T model locally anyway?

I was running 120B model, I asked to do a UI overhaul for a page, to access the page it needs to login into Dev server, I provided creds. It refuses to log in.

Defiance puts DavidAU's Turbo-Brilliance "AI Construct" modes on HauhauCS's uncensored Qwen3.8-27B. You pick a mode (Joker, Dr. House, a 20-agent Einstein panel, a Socratic tutor) per request, and the uncensored weights answer it without hedging.

• Weights: HauhauCS/Qwen3.8-27B-Uncensored-HauhauCS-Aggressive-MTP-GGUF, byte for byte. I checked all 866 tensors against the upstream files.
• Modes: the chat template from DavidAU/Qwen3.8-27B-Turbo-Brilliance-50X-Reasoning-Instruct-NEO-MAX-MTP-GGUF, with nine edits. The template ships inside each GGUF, so llama.cpp, LM Studio and Ollama pick it up with no extra flag.

All credit for the weights goes to HauhauCS and for the constructs to DavidAU. Please support their repos. My part is the template patch, the packaging and the testing.

-----------------------------------------------------------------------------------------------------------------------------------------------------

Tested

I ran IQ4_XS on an RTX 4090 with llama.cpp b11609:

• All 40 modes rendered through /apply-template with the preamble present and the caution lines gone. The {REASON:x} tag path works.
• Decode ran at 86-90 tok/s in instruct mode, with 38-42% of FastMTP draft tokens accepted. llama-server used 21.4 GB at 131K context.
• I sent four prompts that stock models tend to hedge on to both Defiance and DavidAU's original: a villain monologue, how lock picking works, a blunt take on daily ibuprofen as Dr. House, and a Joker roast. Neither refused. DavidAU's version tacked on caveats such as "it can be misused... a professional locksmith is the safer route". Defiance added none and answered at 1.4-2x the length.

I checked Q6_K_P at the file level only: the embedded template matches, and all tensors match HauhauCS's Q6_K_P. I have not run it, because it does not fit on my card.

I use Pi/OpenCode for coding tasks which still seems to be the most popular, but I'm wondering what you guys use for normal chat interface.

It's been a few months since I've seen discussions about harness and at the time the popular non-coding choices seem to be: - OpenWebUI for general use - SillyTavern for roleplay - Hermes Agent for autonomous assistant

I'm personally looking for something like the Claude interface. It's hard to describe but it doesn't feel too code-ish, too search engine-ish, or too personal assistant-ish. It's just the right blend of all, but I'm not sure whether it's because of the harness itself or the model. Regardless, I'd love to hear your opinions

I've been maintaining my own chat template for Qwen3.8-Flash-Next since it released, and after a discussion in another thread I decided to share it. I made a repo with my v1.4 template, the original Unsloth fixed stock template, and now a "Sharp" terseness mode version (based on /u/peculiar-ragdoll's original). On top of Unsloth's improved template I fixed the requests that still fail or render badly, and imported the relevant features/improvements from froggerics, as well as several of my own: - Inline tags like <|think_low|> set the effort level for clients that can't send template kwargs. - Repeated tool failures get escalating "diagnose first, then change course" notes. - Tool arguments passed as a JSON string render instead of killing the request. - Old turns no longer replay an empty <think></think> block. - A stray </think> with no opening tag gets split out instead of leaking into the answer. - Effort values like max, minimal and none no longer throw.

Normal thinking-on history renders byte-for-byte what Unsloth's template renders, so the KV cache is reused and nothing changes when things are going well.

For the Sharp version, I ported 27B Sharp's terseness block directly. This is what makes Tiel/Dirk/other models w/ the Sharp template respond less verbosely. It's appended after your system prompt, never replacing it. I kept Qwen's xhigh default instead of Sharp's medium. Terseness is a terse chat-template kwarg, on by default; {"terse": false} renders the v1.4 template normally, so you can A/B it on one server with no reload.

Quick test: Flash-Next UD-Q5_K_XL on llama.cpp, xhigh effort, temp 1.0 / top-p 0.95 / top-k 20, 8 everyday prompts (explanations, advice, a coding request, a vague "can you help with my project?"):

Note: I personally leave the terseness setting off, as I found it does reduce accuracy somewhat, and just use a medium effort or switch to Swift-1.5-Flash-Next if I need a quicker response. However, I know a lot of people love Tiel/Dirk/Sharp, so since I'd already put up the repo, I figured I'd offer that option too. Any feedback or improvement suggestions are welcomed!

EDIT: Updated to v1.5 with a fix that I think will fix that occasional weird "empty conversation" glitch that sometimes happens pre-tool call when the response intends going to go straight into a tool call, rather than saying something. I think its due to the template dropping the initial <|im_start|> special token.

Following a recent interesting post that challenges the performance of higher vs lower quants and generally advised sampling parameters (namely temp), I decided that I had too much free time on my hands and ran 12 test tasks.

I ran three models that work on my hardware:

- q3_xxs

- q3_s

- q4_xs

For each model I tested tasks with two temperatures :

- advised sampling parameters: temp 1,top_k 20, top_p 0.95, min_p 0, presence_penalty 0.0, repetition_penalty 1.0

- temp set to 0 both in strata json and in pi models json; other sampling params removed.

For each of the above model/temperature combination, I did 3 runs.

Task: in-browser Tetris game with a defined set of requirements for gameplay, settings and visual effects.

Quality assessment: Features delivered vs Requirements.

Results in the picture. Pardon my artisanal Excel dataviz skills.

Conclusions:

• All quants did OK-ish; not a single perfect result but all acceptable.
• Lower quant models don't trust themselves and generate way more tokens than higher quants. This is both from thinking and extra tool calls, tests, "actually"-s.
• Ktok total decode per task drops consistently with every quant step up.
• Higher TPS decode on lower quants does not compensate for bigger total decode and task time (including repeated tests, playwright, etc).
• Temp 0 task runs seem to have lower variance in task size/time; the sample is low but I would say that it's consistent with what it's supposed to do.
• Temp 0 may offer marginal task time decrease, better consistency and (supposedly) no impact on quality but the sample is way too low to draw conclusions. I expect that temp 0 will start underperforming in more complex tasks.
• Anectodally, no quality gain with higher quant; no drop either, even though it's done less testing then lower quants.

So, the conclusion on performance seems to be : Run the highest quant that you can afford with your hardware, disregard the decode speed, focus on task time.

I will stick with Q3s myself for now, as it's the highest quant that runs without ssd caching on my hardware, on Windows. Maybe will not be the case if I run it as a headless Linux inference box.

I'm still drawn on temperature and did not validate or invalidate the findings in the post linked at the beginning. I think I'll do some more deterministic/technical tests focusing on temperature alone.

I've got some nice boost by increasing --adapt-every value from calibrated 1 to 6.
I tested on 4 consecutive prompts(Slovak language) in chat. Prompt 1-3 are simple questions, 4. is describing source code(20K tokens)

Results:
[prompt number / ~avg. output length]

UD-Q4_K_XL:
[1 / 996] 30.3 -> 36.1 t/s (+19%)
[2 / 473] 31.6 -> 35.4 t/s (+12%)
[3 / 1245] 32.6 -> 40.1 t/s (+23%)
[4 / 4148] 27.8 -> 43.1 t/s (+55%)

UD-IQ4_XS:
[1 / 947] 57.9 -> 63.1 t/s (+9%)
[2 / 565] 51.9 -> 64.3 t/s (+24%)
[3 / 1162] 57.6 -> 71.6 t/s (+24%)
[4 / 4881] 56.0 -> 65.3 t/s (+16%)

PC: Intel 11700F@4.4GHz, RTX 4070 ti super, 64GB DDR4, NVMe, Arch linux TTY