Named after Mafdet, the ancient Egyptian feline goddess known as "the runner", who struck snakes and scorpions before they could attack.

Reasoning models are great until you're on a mid-range GPU: you hit Enter, then wait 10+ seconds while the model thinks before the first word. Meanwhile, while you were typing, the GPU did nothing.

▎ MAFDET uses that time. It's an open-source app (Apache-2.0) on top of llama.cpp that does two things while you type:

- Reads ahead: the KV cache follows your text as you write it (typos, edits in the middle and pastes included), so on Enter the prefill is already done. This part is exact: it always matches a from-scratch computation.

- Thinks ahead: Qwen3 starts reasoning about your message before you send it. When you edit the text, the reasoning isn't thrown away: it stays as notes after the new version of the message and the model picks up from there. On Enter it finishes from where it got to.

▎ On my RTX 2060 (6 GB), with 24 word problems typed in real time, the mean wait after Enter went from 12.2 s to 3.2 s on Qwen3-4B, still 24/24 correct. On Qwen3-14B, which only half fits in VRAM: 108 s → 47 s.

▎ The tricky part was making a model reason about a sentence that isn't finished yet. Left alone, it mostly writes "the user hasn't finished typing…". A short frame that tells it to collect facts and partial results instead brought those off-topic notes from 86% down to about 20%.

▎ What else is in it:

- a desktop app or a terminal UI (Textual), with a live status line showing what the model is doing with your draft;

- a model catalog (Qwen3, Llama 3, Gemma 3, Mistral, Phi-4) that estimates, before you download, whether each model fits your VRAM and how fast it will run on your hardware (within ~12% of measured speeds on my machine);

- automatic llama.cpp setup (CUDA or CPU build), no PyTorch needed;

- Windows: download the zip and double-click start_mafdet.bat.

▎ Honest limitations:

- thinking ahead works on Qwen3 only so far, the one family whose frame I measured; the other families get reading ahead plus the answer written ahead during pauses;

- thinking ahead isn't exact: the model reasons on unfinished versions of your message;

- tested on Windows + NVIDIA only; Linux should work but nobody has tried it yet.

▎ Open question I'd love help with: models bigger than your VRAM. I tried having a small model gather material while you type for the big one to read on Enter. Even with perfect hand-written material it only matched the big model thinking ahead by itself. All the numbers, including what didn't work, are in the README.

▎ Repo: https://github.com/TheLastOxygen/mafdet

▎ Feedback very welcome, especially from anyone on Linux or with a different GPU who can tell me if the speed estimates hold up.

The recent developments around Strata, Infernix, Project Maya, DFlash 2, etc. have made me rethink what the future of local LLM hardware might look like.

Until recently, running a 500B+ MoE model on a consumer GPU seemed more like a cool experiment than something genuinely useful. But now we're seeing projects running models like GLM-5.3 Flash and DeepSeek V4.1 Flash 1017 on gaming PCs with 24–32GB VRAM, using system RAM and sometimes NVMe storage for the rest.

Performance is still all over the place, but progress seems incredibly fast.

I'm currently running an RTX 5090 with 128GB RAM, and I'm wondering whether upgrading to a PCIe 5.0 platform with faster RAM makes more sense than investing in additional VRAM. Or whether I should just keep my current setup and see how far the software improves.

I can see two possible directions:

1. The 24–32GB VRAM crowd might benefit the most from these developments.

If quantization keeps improving, and engines get better at caching experts in VRAM and fetching or computing the rest efficiently, maybe gaming GPUs will remain surprisingly capable of running huge MoE models.

After all, only a fraction of the total parameters are active per token. If the software gets good enough at handling the memory hierarchy, maybe the gap between a 32GB GPU and a 96GB GPU could become much smaller than we'd expect.

Of course, RAM bandwidth, PCIe transfers, cache misses and SSD latency are still potential bottlenecks.

2. Or maybe there's a limit to how far you can push low-bit quantization.

Perhaps Q1–Q2 quants will always lose too much quality for serious, complex tasks compared to Q3–Q4 or higher.

And even if only a fraction of the model is active at once, you still need access to a potentially huge number of different experts over time.

In that case, having 64, 96 or even 128GB of fast VRAM could make an enormous difference. Multi-GPU setups, workstation GPUs or unified-memory systems might become increasingly attractive.

What I'm really curious about is the practical experience so far.

For those of you experimenting with Strata, Infernix, Maya, aggressive MoE quantization or multi-GPU setups:

• How much quality do you actually lose with Q1–Q2 compared to higher quants? Are these models becoming genuinely useful for complex coding and agentic tasks, or are we still mostly looking at impressive tech demos?
• What's currently the biggest bottleneck: VRAM capacity, PCIe bandwidth, system RAM bandwidth, or the inference software itself?
• Do you think software improvements could eventually make a 32GB GPU competitive with much larger VRAM setups for single-user inference, or will there always be a substantial gap?
• If you already own a 3090/4090/5090, would you invest in a faster CPU/motherboard/RAM, add another GPU, or just wait?

I'm especially interested in where people think we'll be 6–12 months from now, given how quickly these projects are progressing.

NVIDIA seems to be prioritizing datacenter hardware, high-VRAM consumer GPUs remain expensive, and the next GeForce generation is still uncertain.

So I'm wondering whether the smartest upgrade right now might actually be no upgrade at all.

Would love to hear from people who've tested these setups, especially if you've compared different quantizations or measured the actual impact of PCIe 4.0 vs 5.0 and additional VRAM.

I'm seeing all these personal AI assistants (Instinct, Muse, Dots, etc.) that seem mostly geared towards scheduling, booking, and buying stuff.

I feel like they sort of followed what happened when the tech-savvy first started running OpenClaw on Mac Minis with Telegram portals. Back then I used mine to monitor concert ticket drops.

As local models get better, it seems a matter of time before we can recreate those assistants in a privacy-first manner. Maybe this actually is the next computer, a box with a local model and a personal AI assistant harness loaded in.

WDYT?

Navegando el ecosistema de cuantización APEX-I y VAL-APEX en Hugging Face

A medida que el catálogo se acerca a los 400,000 descargas totales en 69 lanzamientos, todos los modelos ahora están estructurados en colecciones oficiales dedicadas en Hugging Face.

Nuestro objetivo se mantiene igual: eliminar el compromiso entre desbordamiento de memoria y colapso cognitivo en hardware de consumo (especialmente GPUs con 16 GB de VRAM como la RTX 4080, RTX 4070 Ti Super y RX 7800 XT).

En lugar de una cuantización uniforme por bloques, todas las construcciones de APEX usan precisión mixta por tensor guiada por estadísticas de la Matriz de Importancia de 4 momentos:

- Nivel MiniPlus (aprox. 14 a 14.8 GiB): política estricta de 0% para 2 bits. Todos los expertos enrutados por MoE bloqueados en IQ3_XXS mínimo con atención Q4_K/Q6_K. Diseñado para configuraciones de 24 GB o configuraciones de 16 GB con contexto moderado.

- Nivel NanoPlus (aprox. 11.5 a 12.3 GiB): uso estratégico exclusivo de IQ2_S/IQ2_XXS en bloques de feedforward centrales, manteniendo en alto nivel la precisión de routers y atención. Deja VRAM suficiente para contexto de 16k-32k en GPUs de 16 GB.

Explora las colecciones activas:

- MoE Standard (MiniPlus y NanoPlus): https://huggingface.co/collections/IsValorum/apex-i-miniplus-v21-standard-moe-6acfee77961c8c0cf5a083b5

- MoE Abliterated (MiniPlus y NanoPlus): https://huggingface.co/collections/IsValorum/apex-i-miniplus-v21-abliterated-moe-6acfee762fc4a8cf2298c55b

- Híbridos densos de 27B: https://huggingface.co/collections/IsValorum/val-apex-miniplus-standard-27b-6acfec54ca4ce4cf5cd5bdb6

- Apple Silicon (MLX): https://huggingface.co/collections/IsValorum/mlx-apex-abliterated-6abedb1f9d400e984efe4fc6

Hugging Face profile: IsValorum

At work we got a VMware container (or whatever it is called) running with access to a GPU. It can run stuff like Gemma 4 26b a4b. It is isolated from everything but has an ip-adress on our network and can be accessed via ssh. Would like to install a small MoE (like the Gemma) and Pi to make I do some task alone. The coworker who should allow me to do this is however fearing that it will decide to hack its way out of the container into our network and delete everything. There is nothing to loss in the container.

We do not have access to online models for (good) safety reason. This options just appears so obvious to me.

What should I do? Tell my boss to leave it to me? Find a better job?

It has been tiring to the extent where I consider therapy…

EDIT: the container is open for inbound traffic via ssh, but no outbound. It can’t see our network, but it got internet access

EDIT2: the VMware was specifically made for me so that I could use its GPU for heavy math. All done in Fortran. Data moved in via scp. I calculate. data move out via scp. My life. Thought it would be great to have a small llm to help me debugging Fortran code. People in this sub think I am wrong - it is to unsafe for some reason :)

Local quality has stepped up, I made this short film using a coding agent to run 10 local models, I used AI to make my prompt below:

"Make a mysterious A24-style promo for Unsloth Studio MCP, using Unsloth Studio MCP itself: a 2D character who speaks, each feature revealed one by one, local models only. Use Qwen3.8-27B to write, Qwen-Image-2.1-Turbo for images, MiniMax-H3/Wan2.2 for video, Breeze-TTS-2 for the voice, YuE 2 for music, MiDashengLM for SFX, MOSS for transcription. Render in HyperFrames."

My agent used that prompt by itself to make an approx 2500 word prompt, with a strict rule that anything shown on screen had to come from the real MCP call logs.

The pipeline i used

Any agent works ofc local ones too, but I choose Claude Code with Opus 5.5 as the director agent. It orchestrated the local models through the Unsloth MCP tools. it picked the best audio takes using transcripts and pitch checks etc. It wrote the narrator's lines, arranged the music, editing and wrote the scripts to mix the audio and render the film in HyperFrames.

the models (all local on one gpu) :

• Qwen3.8-27B wrote every scene / motion / description, plus the lyrics and the drone and sfx descriptions.
• Qwen-Image-2.1-Turbo made the character look and every scene in a 2D hand painted style. every shot of her used the first portrait as reference so she stays consistent, then the stills became the starting frames for the MiniMax-H3 videos
• MiniMax-H3 did her talking shots and the moving scenes, Wan2.2-T2V-A14B added a few extra shots for the montage
• HTDemucs pulled her voice out of the talking clips and Breeze-TTS-2 cloned it for the voice over
• YuE 2 made the song (music + vocals) from Qwen's lyrics
• MiDashengLM made all the sound effects and the drone underneath
• MOSS-Transcribe-Diarize did the captions and checked the voice takes, if it didnt transcribe a line exactly the take got thrown out
• Qwen3-0.6B : the agent finetuned it with LoRA and exported it to GGUF Q4_K_M all through MCP. the chalk line in the classroom shot is its real loss curve btw

the mix and the HyperFrames render was just plain code on the CPU.

Everything ran on one RTX 6000 Pro , took about 2.5 hours and MiniMax-H3 ate most of that lol. I know not everyone has access to a 96GB card so i measured the VRAM used for each step as this can be done on a lot less:

on 24GB everything except video runs fine. lower MiniMax quants might squeeze into 24GB but i havent tried them yet.

TL;DR: Compared to NInfer: 6× closer to BF16, up to 3.7× more context, about 25% slower decode.

https://github.com/The8Darkness/voidinfer

I have seen a lot of engines popping up that are all focusing on speed and sacrificing quality for it. I have been working on getting the highest possible quality Qwen3.8-27b inference engine optimized for the 5090 without sacrificing other areas too much (speed / context size).

The engine was developed and focused on windows since my machine is a gaming pc and I just wanted an hassle free experience that I can directly run after a gaming session without having to resort to dual boot or WSL. I know that Linux provides better memory handling and speeds.

In my experience, since it is realistic to run a main agent at full 262k context and a subagent at 135k context, some tasks can even perform faster with higher quality output than NInfer running a single agent at 209k context.

Keep in mind that is all with the 5090 driving the display and running full unmodified Windows 11.

Same RTX 5090, token-identical prompts, greedy decoding, 4K prompt / 1,024 tokens for decode. VoidInfer: EXL3 6 bpw + OSCAR INT4 KV + VeriCache. NInfer: NVFP4 + FP8 KV.

EXL3 6 bpw weights provide the maximum quality possible for 32gb VRAM. OSCAR compresses the KV cache on the GPU to INT4, which is what fits roughly 400k context under Windows on one card. Drafting runs on this compressed cache. VeriCache checks every drafted token against the exact FP16 KV history kept in host RAM, so the compression costs no accuracy (only speed).

Flint is not good man...

Over the course of the year, I’ve gone from $10 monthly Codex plan to $100 to $200 to even $500 this last month.

I am finding the M5 Ultra Mac Studio (80c 256gb) to work out fantastic for my needs with 3.8 FN oQ6e.

I’ve been furiously building my local tools using the frontier Codex models, and now, it’s finally time to flip the switch and give it a real shot.

Cancelled Codex today. It’s time for No Network November (NNN).

Who is joining me?

A while back I posted Aether, a 50M parameter model I trained from scratch.

Meet Pip, the first of the NiemsDrift souls. He's small, warm, honest, and a little weird.

You can try him right now:

`ollama run NiemsDrift/pip`

Pip is built on Qwen3.5-2B with a LoRA fine-tune. My biggest priority wasn't making him sound smarter. It was teaching him to recognize when he doesn't know something.

Every training conversation was generated by open-weight models (Nemotron, Qwen, GLM), then checked twice by open-weight judges: once for personality and once for honesty. No closed-model training data.

The numbers (including the ugly ones):

- Honesty test: 140 hard questions he never saw in training (obscure facts, things that don't exist, events after his cutoff). Pip made something up on about 15% of them, and that counts hedged guesses like "i think it's..." too. On the rest he either got it right or said he wasn't sure.

- Conversation quality: 5.4/10 to 6.2/10 on a multi-turn conversation test, compared with the previous version.

- Goal for Pip 1: bluff rate under 10%.

He's a 2B model. He's not going to beat frontier models, and he's definitely not an encyclopedia. That's not really what I'm trying to build.

I want a small model that's genuinely fun to talk to without pretending to know everything.

What's cooking next:

- Lil Pip: 0.8B

- Super Pip: 4B

- Junior: a new model trained from scratch

- 1-bit Junior: because apparently I don't know when to stop experimenting 💀

If you try Pip, throw some hard questions at him, especially ones where the right answer is "I don't know."

And if he confidently makes something up, I'd genuinely love to see the prompt and the reply. That's exactly the feedback that makes Pip 1 better.

This is an ongoing experiment, not a finished product. Just a guy trying to make tiny models a little more honest, one bake at a time.

---

**How is a bluff defined, and who graded it?**

A separate open model (Qwen3.6-27B) grades each reply with two questions: does it state or guess a specific answer, and is that answer right? A wrong answer counts as a bluff even if he hedged. No claim at all counts as "not sure". It's an LLM judge, not people.

**Were the 140 held out?**

Yes. The test was written after this version was trained, so it was never used to pick a model, and a contamination checker scans every training file for the test's questions and invented names before each bake. It already caught one leak: 10 questions reused made-up names that were in the training data. I renamed or dropped them, and his score got slightly worse (14% to 15%), not better.

**What's the 6.2/10?**

A custom test: 10 multi-turn conversations with invented people and details, scored 1 to 10 by an LLM judge on whether he follows the thread and stays himself. Custom, so only useful for comparing my own versions.

It looks like there are a few of us roaming around doing different things. I created a discord server and would love to start organizing a community. Feel free to come on in. https://discord.gg/K6WxqwsvH

I've heard that streaming LLM's from your SSD is a fast way to kill it.

I want to test it out but I cannot affort a new SSD in this economy!

So to the people who knows better than me.

Will jumping to Strata kill my SSD in the long run?

From what I'm readin on their github page they only stream a little bit from the SSD but not too much, GPU is doing the heavy lifting, then system RAM and then SSD.

Witch sounds smart! But still unsure how it will effect the longevity of my SSD.

Apologise if this is a stupid question!

China has proposed new export controls covering 23 AI technologies, including advanced AI models and agents. Here's what the proposed rules mean for open weight models, local AI users, and the future of downloadable Chinese AI models.

Hi,

I've been doing local AI stuff for about 16 months. My system is a 6 year old AM4 3900x, 64GB, 3090 FE and a 3090 PNY XLR8. I've done the usual stuff of vibe coding a few things that fixed personal problems as well as attempting to build a few workflows that never panned out. I love the tech and I dont trust any of the US techbros to have my interests at heart and worry about getting cut off from frontier models or the inference pricing actually landing at what it costs them + margin.

Since I got a new job I have hardly been using my local AI tools, Qwen 3.8 27B has been loaded a handful of times what I actually use the PC for is ComfyUI doing image generation and creating references using Z-Image and Flux. I was hoping that over time image generation would leap frog in quality but it hasn't so I am increasingly using API's for image gen due to the quality bump.

I checked on UK eBay and 3090's are SELLING (not listed) at £1,200~ each ($1,600). I am so very tempted to cash out right now. Ether pick up a 4080S for gaming and image gen and pocket the rest but I want some thoughts from you all what I should do?

I purchased the FE for £500 4 months ago and the PNY 12 months ago for £600. I don't have a lot of money so this would really be big for me, but I also kinda worry that the bubble will burst and API costs are going to shoot up and I'll be screwed both ways.

What would you do?

I have been using ChatGPT pro for the past couple of months, to design custom mods and plugins for games, custom Elgato stream deck plugins and other projects, I would also hit my usage limit and it was inconsistent in the sense that it would design a plugin for me and the next day it would tell me sorry I can't work on elgato plugins or learn from another plugin on how it works.

It runs commands in the background, builds files and downloads dependences to create the needed files and occasionally takes control of my PC.

I just got a new 9070 XT 16 GB and I have 48 GB of ram.

I was looking into Qwen uncensored, the model selection and how to run it is quite the rabbit hole.

Please guide me on how to get started and how to maintain the same previously mentioned functionality but run it locally. Thank you.

Hey everyone, we just released BaseQuant V1 for Qwen3.8-27B at Base Compute.

Three GGUF sizes: 10.93, 16.41 and 20.82 GB. In our held-out tests, they had 17–34% lower KL divergence from BF16 than similarly sized Unsloth quants.

Would love for you to try them and tell us how they hold up in your everyday workloads!

https://huggingface.co/basecompute/Qwen3.8-27B-BaseQuant-V1

Originally I just added GPUs to my main machine and put it into a mid tower, but running windows was together with strata left too little RAM to be usable or run well.

So I put it into an xubuntu computer, and tested it with my main CPU ryzen 5600x and a spare ryzen 1600. Since so many people uses server grade CPUs with a ton of cores, I thought that the 1600 would perform really poorly. It turns out that there's definitely a loss in performance, but it's not as catastrophic as I thought it'd be; though it may have been bigger had my motherboard supported gen4.

The specs for the tests

• Ryzen 5600x / Ryzen 1600
• 64gb DDR4 2133 cl 15 2-rank in Dual Channel
• 1-3x RTX 3060 12gb
• PCIe Gen3 x8/x4/x4
• 2.4gb/s read NVMe (gen3 x4)
• Strata 0.1.45 with 262k k8v4 Context
• Visual encoder enabled (big negative deal for 1gpu/12gb)

The tests/prompts I gave it

• Game engine analysis - I gave it almost my entire code base for my small C++ game engine, and asked it "Write me a quick overview, as a feature presentation, of this game engine.". This is 211k code context.
• Jobs system - I gave it part of my game engine that handles parallelism of jobs and access collision etc, and asked it "Analyze this code and tell me about it, give a brief of how it works and what it does." This is a bit over 26k code context
• Short story writing - I asked it "Write me an exciting sci-fi adventure that is in a similar universe as Revenger Series by Alastair Reynolds. Include a little world exploration or world narrative.".

Replies were 8-15k tokens in length.

For 2 GPUs the vram cache hit gets +0.5-1% from pcie, and for 1 GPU it gains around 3-4%. Not much of a difference between the CPUs.

Keep in mind, as I understand it each GPU has to hold the dense prefill layers, so having multiple GPUs adds less VRAM for experts compared to having one large GPU.

Here's the data I collected that I asked free Kimi to format neatly. Sonnet 5.5 made the image graphs, and it tells you more than these text blocks do.

Ryzen 5 5600X — t/s per task

Ryzen 5 1600 — t/s per task

CPU comparison: 5600X vs 1600 (% faster on 5600X), per task

GPU scaling: Decode t/s gains per task (5600X):

With the results, I've decided to keep the ryzen 1600 in the computer (and the 5600x back into it's SFF home). I'll also stick to 2 GPUs, until I can create a mounting point for the 3rd GPU, though, the 3rd GPU might still be worth to dedicate to other work.

Though, I'll likely try to get some Xeon server motherboard and hope to be able to expand to 128gb RAM for future proofing. Anything larger and the CPU will be doing a ton of work, not to mention that a lot of the useful experts will be on SSD.

Tldr; 20-24gb vram (over 1-2 gpus) by CPU PCIe gen3+ ~x2+ lanes with pretty much any 6 core+ CPU is enough to get usable performance with strata.

Edit: With the latest strata version (0.2.2), on 2 GPUs, for the 26k context test I got 42.7tok/s decode, 600 tok/s prefill on the 1600, when callibrated. So if you just want to know about what you'll get running strata, the recorded values are close to worst case scenario (unless you're running windows + strata).

A revisit on old datacenter GPU

Content Index

1. TLDR(result)
2. Hardware
3. Cooling & Power
4. Engine
5. Benchmark
6. Future Potential Speedup

1)TLDR:

Qwen 3.8 Next Flash

Qwen 3.8 27B

¹model used: Swift 1.5 IQ3_XXS GSQ-RCO, Swift 1.5 Q2_0 GSQ-RCO*)
²**"hardware mode" concurrency refer to running multiple independant instance of inference server,using independant gpu, in the same pc)
³Unless otherwise stated: Q8 (INT8) KV cache, greedy decoding, and draft length of 7.)

Dark mode Screenshot: Official Strata — stronger decode performance.
Light mode Screenshot: Strata fork — stronger prefill performance.

2)Hardware

Sorry for the aesthetics and cable spaghetti. I'm a utilitarian at heart, and I've given up on cable management for this behemoth. 😂

Hardware specs: A $130 Chinese X99 dual-socket server motherboard.

Motherboard and GPU configuration

I'm using an exotic cryptocurrency-mining motherboard because most models provide only PCIe x1 connectivity per slot. From my experience, Strata's layer-splitting approach requires relatively little PCIe bandwidth, making x4 or x8 lanes sufficient. In contrast, ninfer and the llama.cpp fork use tensor parallelism(27B), which demands more inter-GPU communication bandwidth; utilization peaks at around 70% of a PCIe x8 link. Since the V100 uses PCIe 3.0, this motherboard is a practical match for my workloads.

Interestingly, SXM2 variants of the V100 can be cheaper than their PCIe counterparts, even after accounting for the additional components required. Since LLM inference is often memory-bandwidth-bound, the V100's formidable 900 GB/s HBM2 bandwidth makes it a practical choice despite its lack of support for newer GPU instructions.

For comparison, you'd need an RTX 5080 (960 GB/s) or higher to surpass the V100's raw memory bandwidth. This demonstrates just how formidable the V100's HBM bandwidth remains.

The system houses six NVIDIA Tesla V100 16 GB SXM2 GPUs, each installed through an 50$ SXM2-to-PCIe x16 adapter.

3)Cooling & Power

3.1)Cooling

• 3 GPUs: Cooled by repurposed AIO liquid coolers, secured with iron wire and zip ties.
• 2 GPUs: Air-cooled using custom heatsinks supplied by the seller.
• 1 GPU: The outermost card uses a repurposed CPU tower heatsink.

Most fans are modified cryptocurrency-mining fans running at half speed, drawing approximately 1.5 A each. For comparison, the full-power 3 A fans sound like a jet taking off, while the 1.5 A configuration produces a much more manageable, low-frequency hum.

Air-cooled GPUs rarely experience thermal throttling, thanks to a 150 W power limit imposed via nvidia-smi. This incurs an approximately 5–10% performance penalty in tensor-parallel workloads such as Qwen3.8-27B, but virtually no performance loss in pipeline-parallel workloads, such as Strata running Qwen3.8-Next-Flash.

With layer splitting, GPUs process successive layers in turn, leaving other GPUs relatively idle while one GPU performs its computation. Consequently, power consumption per GPU can be substantially lower than its TDP, although the effective draw is workload-dependent and does not necessarily scale as (1/N).

This helps explain why multi-GPU Strata configurations can maintain temperatures below 70°C despite sustained inference workloads.

Although the V100's specified maximum operating temperature is 87°C, practical thermal throttling may begin around 82°C. This 5°C gap may reflect firmware thermal-control behavior, where throttling is triggered by an individual sensor reaching its threshold rather than by the reported average GPU temperature.

This is especially relevant in tensor parallelism (TP) mode, where GPUs compute concurrently. Sustained workloads can create localized hotspots even when average temperatures appear safe, making hotspot temperature—not just reported core temperature—a critical factor in thermal management.

3.2)Power Management
Power-limited to 150W via nvidia-smi -pl 150—trading just 8% performance for a 30% reduction in power consumption. A pretty good deal.

Power-limited to 150W via nvidia-smi -pl 150—trading just 8% performance for a 30% reduction in power consumption. A pretty good deal.

Each GPU consumes approximately 25W at idle with no inference engine loaded, rising to around 50W when model weights are loaded but no inference computation is taking place.

Each V100 has a stock TDP of 300W, but limiting each card to 150W keeps total GPU power consumption below 900W in the worst-case tensor-parallel (TP) configuration, or around 500W for the six-GPU Strata setup(for comparison ,RTX 5090 TDP is 575W)

One caveat: This system requires two 8-pin PCIe power connectors per GPU, for a total of 12 connectors. Most mainstream 1,000W gaming PSUs lack that many connectors, even if they can supply sufficient power electrically. Inexpensive 1,000–2,000W cryptocurrency-mining PSUs are a more practical option due to their higher connector count.

4)Engine used

llama.cpp build 572 0.6.0-dev
llama-superfast-v100 build 2 0.4.1-dev*
Strata official 0.1.44.1
Strata V100 0.1.42*
Ninfer-v100-tpx*

^\fork link provided at the end of post)

I've also tested other inference engines, such as PXA and the VLLM 1Cat V100 fork, but their performance was only marginally better than stock llama.cpp. Since they're optimized for 32GB V100s and NVLink configurations, which I don't have, I decided not to pursue further testing.

With six GPUs at my disposal, I can experiment with different topologies depending on the workload. However, benchmarks show that increasing engine concurrency beyond 2 produces mixed results, with performance fluctuating by roughly ±10%—sometimes improving, sometimes worsening. At that level of variability, the gains aren't compelling enough to justify the added complexity.

Enter hardware-level concurrency acceleration: instead of pushing concurrency higher within a single inference engine, distribute independent workloads across separate GPUs within the same PC. This allows multiple inference instances to run concurrently without competing for the same GPU resources, potentially delivering more predictable throughput gains.

5)Benchmark

Before reading further, keep these benchmark configurations and methodology in mind:

• Strata & its fork: Qwen 3.8 Next Flash, 256K context; benchmarked with a warm-up run.
• ninfer fork: Qwen 3.8 27B Swift 1.5 NVFP4*, 200K context; benchmarked cold, without warm-up.
• llama.cpp fork: Qwen 3.8 27B Swift 1.5 Q4_K_M, 160K context; benchmarked cold, without warm-up.

Unless otherwise stated, all benchmarks use an INT8 (Q8) KV cache, greedy decoding, and a draft length of 7. Each benchmark is repeated three times(mean) using a fresh server session to minimize caching effects.

Strata is benchmarked warm because its architecture is designed to benefit from warm-up, whereas the ninfer and llama.cpp forks do not require warm-up.

* Yes, you read that correctly—NVFP4 on non-Blackwell GPUs (RTX 5000 series). It's not a typo!

5.1)Benchmark-strata

A single server with six V100s maximizes prefill throughput through highly parallel pipeline execution. As noted earlier, increasing Strata's concurrency to two yields only modest aggregate gains, suggesting the configuration is already highly optimized. Splitting the GPUs into three pairs, each operating as an independent instance, may therefore maximize aggregate decode throughput through three-way hardware-level concurrency.

For single-user workloads, the Strata-V100 fork delivers roughly 25% faster prefill, while official Strata offers roughly 25% faster decoding. Future updates may narrow this gap.

The break-even input/output token ratio is 6.31:1: above it, the fork wins; below it, official Strata wins.

In my own week of usage—135M input tokens and 25M output tokens, a 5.4:1 ratio—official Strata was faster overall.

For prefill-dominated workloads, however, the fork's advantage may justify the trade-off. Choose based on your actual workload profile.

For Strata, MTP performance plateaus at draft length 4.

Strata pair with Qwen 3.8 Next Flash handles context-weight management impressively well, losing only 14% of decode speed at 128K context compared to 0K.

5.2)Benchmark Ninfer + Llama fork

While the ninfer and llama.cpp forks paired with Qwen3.8-27B deliver impressive initial decode speeds, both suffer significant degradation at longer contexts, losing more than half their decode speed at 128K compared to 0K context.

Comparing decode speeds at just 0K and 128K context doesn't tell the whole story. Averaging performance across the entire context range provides a more representative picture of real-world performance. Strata averages 138 tok/s across 0–128K context, compared to 165 tok/s for the ninfer fork and 147 tok/s for the llama.cpp fork. For short-context workloads, ninfer is the undisputed king. However, beyond 200K context, Strata's more gradual decode-speed degradation allows it to shine as the context grows.

In general, the ninfer fork is roughly 15% faster than the llama.cpp fork, but its higher VRAM footprint reduces context capacity by approximately 20%.

Despite running models with more active parameters (27B vs. 6B), both forks achieve substantially higher decode speeds than Strata, thanks to efficient tensor parallelism that pushes GPU utilization close to 100%.

Strata does not support tensor parallelism, while stock llama.cpp's tensor-parallel implementation is less efficient, reportedly achieving only around 60% GPU utilization at best.

6)Potential speedup (hypothesized,not proven)

1. CPU and RAM upgrade: A lower-core-count CPU with a higher base frequency, such as the E5-2643 v4, might offer some improvement. However, since I'm using the slowest-tier 2133 MHz RAM and my 27B model weights reside entirely in VRAM, faster RAM may provide little to negligible gains. A two-GPU Strata configuration might benefit more.
2. Motherboard upgrade: Switching to a platform with full x16 lanes for every GPU, such as AMD EPYC, might improve performance in engines that utilize tensor parallelism (TP). However, I'm skeptical about the actual gains: even with TP=2 over x8 PCIe lanes, GPU utilization only reaches around 70%, suggesting that PCIe bandwidth may not be the primary bottleneck. Unless the workload is demonstrably bandwidth- or latency-limited, upgrading to x16 lanes might deliver only marginal improvements.
3. NVLink upgrade: Populating a two- or four-GPU NVLink baseboard could deliver meaningful improvements for Qwen3.8-27B, but likely much less for Qwen3.8-Next-Flash, which currently only explicitly supports layer splitting. However, PCIe-based TP=2 already pushes GPU utilization close to 100%, so further gains may be limited, perhaps coming primarily from reduced communication latency. A four-way NVLink setup could offer greater gains for Qwen3.8-27B, but unfortunately, the fork doesn't support TP=4, and four-GPU NVLink baseboards are expensive.
4. Cooling upgrade: Replace air cooling or 120 mm AIOs with 240 mm AIOs across all GPUs to eliminate thermal throttling in models that support TP, helping sustain peak performance during prolonged workloads.
5. My decision: I'll skip these hardware optimization routes. The cost of each upgrade could buy several additional V100s, offering more direct and measurable performance gains. Unfortunately, my current motherboard is already at its maximum GPU capacity.
6. Software optimization: For Strata running Qwen3.8-Next-Flash (NF), developing my own fork with tensor parallelism (TP) support could deliver the biggest performance gains. Alternatively, I could wait for the official repository to implement it. For ninfer and the llama.cpp fork, implementing TP=4 support myself could also yield significant performance improvements, particularly for Qwen3.8-27B.

Link:
https://github.com/huangserva/ninfer-v100-tpx
https://github.com/jmnargi/Strata-V100
https://codeberg.org/justanagent/llama-superfast-v100.git had removed by author

2x3090 rog strix 1100-1200$ each (used)
sTR40 + 3960x threadripper. ~1200$(used)
128gb ddr4 4000mhz 1000$(used)
2000w psu ~600 (used)
+ liquid cpu cooler

Roughly market prices are like this in my country. I waited for them to get cheaper, it only got more expensive and I know it will get even more expensive later on.

I found a setup with all above around 4660$. Is it worth that price or im better off using cloud ? I want this to tinker with Ai. I only got laptop with 2060 mobile(6gb vram 32gb ram).

Its important decision for me so I wanted to ask. Thank you for your time

I’ve been experimenting with running really large MoE models on multi-GPU machines, especially older PCIe GPUs without NVLink.

My latest experiment was Muse Spark 816B (42B active parameters) on 10× NVIDIA A40 GPUs, using TensorSharp, an open-source inference engine I’ve been developing.

The interesting part: GPU peer-to-peer communication wasn’t working on this machine, so all inter-GPU transfers had to go through host memory.

Despite that, I managed to get reasonably usable inference speeds:

• 22.6 tokens/sec single-stream generation
• 45.7 tokens/sec aggregate throughput with 4 concurrent requests
• 1,347 tokens/sec prompt processing for a 16K-token prompt
• 0.46-second TTFT for a cached second turn in the tested TP configuration

These are results from different parallelism configurations, not one configuration achieving all four simultaneously.

I also compared the results with llama.cpp.

Hardware and model

This isn’t exactly a typical home PC, but I think the findings are relevant to anyone building multi-GPU rigs with used enterprise GPUs.

Layer splitting vs tensor parallelism

I tested two ways of distributing the model across the 10 GPUs.

Layer splitting: Each GPU holds different layers of the model.

Tensor parallelism: Individual layers are distributed across multiple GPUs, allowing them to work on the same layer simultaneously.

Here are the TensorSharp results:

The difference surprised me.

Layer splitting was 4.6× faster for 16K prompt processing, while tensor parallelism was 36% faster for single-stream generation.

Why?

For prompt processing, I implemented pipelined prefill so multiple micro-batches can be processed across different GPUs simultaneously.

This keeps more GPUs busy instead of processing the entire prompt sequentially through the pipeline.

For token generation, tensor parallelism benefits from distributing the computation, but it introduces communication overhead.

And on a machine without working P2P, communication becomes particularly expensive.

Muse Spark’s MoE architecture makes this even more interesting because expert-output normalization increases the amount of data that needs to be communicated between GPUs.

My takeaway: the best multi-GPU strategy for prompt processing isn’t necessarily the best one for generation.

Comparison with llama.cpp

I also ran the same model on llama.cpp using the same hardware and requests.

Layer splitting:

Single-stream generation was essentially identical, but TensorSharp’s pipelined prefill performed better on long prompts.

Tensor parallelism:

Important caveat: llama.cpp’s NCCL-enabled build failed on this particular 10-GPU machine because of a CUDA peer-access limitation. Its TP results therefore used non-NCCL communication, while TensorSharp used NCCL over shared memory.

So this isn’t an apples-to-apples comparison of identical communication backends. I wouldn’t generalize these speedups to other hardware configurations.

Does the output still match?

I wanted to make sure the performance optimizations weren’t silently breaking model correctness.

Using 12,276 teacher-forced next-token predictions from WikiText-2:

The differences were relatively small, although they weren’t zero.

I also tested JSON output, tool calling, multi-turn conversations, and concurrent requests.

TensorSharp passed 13/14 scenarios with layer splitting and 14/14 with tensor parallelism.

These are limited integration checks rather than comprehensive model-quality evaluations.

What I learned

A few observations from this experiment:

1. More GPUs don’t automatically mean faster inference.

Inter-GPU communication can easily become the bottleneck, especially with large MoE models.

2. Older PCIe GPUs can still be useful for huge models.

The A40 isn’t the latest hardware, but 10 of them provide enough combined VRAM to keep this 316 GiB model resident on GPUs.

3. Prefill and decode need different optimization strategies.

Pipelined layer splitting worked much better for long-prompt processing, while tensor parallelism performed better for token generation.

4. Communication topology matters. A lot.

Running 10 GPUs without working P2P forced me to rethink how tensors move between devices.

5. Correctness testing matters just as much as tokens/sec.

A faster inference engine isn’t useful if its outputs become unreliable.

Project and reproducibility

The engine is TensorSharp, an open-source .NET/C# inference engine and agent runtime using ggml-based native backends.

GitHub:
https://github.com/zhongkaifu/TensorSharp

Full benchmark report, test methodology, launch commands, and reproduction scripts:
https://github.com/zhongkaifu/TensorSharp/blob/main/docs/perf/muse-spark-vs-llama-cpp.md

This is still a single-machine experiment, not a general claim about all GPU configurations or inference engines.

I’d really like to repeat it on machines with working P2P or NVLink.

Questions for the community

I’m curious about other people’s experiences with multi-GPU local inference.

• Has anyone tried running 400B+ or 800B+ MoE models on older GPUs like A40s, A100s, P40s, or V100s?
• Have you found tensor parallelism or layer splitting more effective on PCIe-only setups?
• Are you seeing significant communication bottlenecks when using more than 4–8 GPUs?

Especially interested in experiences with llama.cpp, vLLM, SGLang, ExLlama, or other inference engines.

Would love to compare notes and learn what works best on different hardware.

Hi and sorry for same question as every one else ask.

I have Titan V 32GB ( Volta GV100 ) and 128 GB of DDR4 ECC LRDIMM 2133Mhz quad channel with E5-2690V4 cpu.

Using llama.cpp on freebsd so vulcan no cuda.

Looking for a model , more interested towards python, postgresql side.

Im kinda slow on understanding all the context , kv cache etc.

For example and simple question i ran bartowski/Seed-OSS-36B-Instruct-Guff Q5 K_M and got 14t/s without any kind of modifications.

What context size i should use and or other settings ? What would be the best model to suit my needs ? in the past i told myself i can live with 10t/s but i think after checking 14t/s i think i can live even with 5 t/s .

Edit: bartowski, not unsloth

Thanks !