I asked my local AI agent to optimize its own setup.

The first thing it did was stop the inference server.

The one it was running on. 🤦

Fair enough. Can’t beat zero resource usage.

So we established a rule: don’t shut yourself down.

On the next attempt, it cleaned up some Python processes.

Including itself.

The rule now has a second paragraph.

I just tried to install QFN from unsloth (Q4_K_XL) on my strix halo laptop with 128gb of ram. Set it up with Gufo port for windows, using opencode v2 on my macOS as the harness and the effort is set to high.

I wanted to do some testing and compare it with opus 5.5 xhigh with the official claude code harness (my daily driver). So i gave both model the same prompt which is to migrate several screen from flutter to swiftUI from scratch. The app is a fully funnctional app with a lot of logic and network calls, and it is used in production with thousands of DAU

Opus managed to do it in 50 mins and what amazes me is QFN finishes it in 60 mins, and the result is pretty identical!

sure it is not as polished as opus' result but I am sure with 2-3 more prompts and 5-10 mins it could produces what Opus 5.5 xhigh produces (just some styling work)

I really hope that qwen 4 27b could matches opus 5.5 in coding work, fingers crossed 🤞If yes, the gap between china model and frontier model are only 2-3 months

I introduce Yoshino 0.8B, a promising (yet mundane) fine tune of Qwen3.5 0.8B.

Over the past few weeks, I have developed a custom black box distillation method where the teacher model generates prompts for itself which it is fed in order the generate synthetic data pairs. Using this method, I distilled the data generated from IBM's Granite 8B into Qwen3.5 0.8B in order.

The result is a neat little SLM which seems to have adopted the reasoning style of the teacher. I am not saying that this is a frontier discovery, but it may prove to be useful for some.

The resultant model did score noticeably higher on the benchmarks which I conducted.

To read more (or download) the model, you can check out the main and GGUF repositories.

I have been working on implementing adaptive KV streaming and RDNA 4 / ROCm optimizations for llama.cpp to enable running Qwen3.8-27B (IQ4_XS weights, Q8_0/Q4_0 KV cache) with its full 262,144-token context window on a single 16 GB RX 9070 XT GPU at 80+ t/s decode.

Qwen 27B IQ4_XS is ~14.3 GB, leaving insufficient VRAM headroom on a 16 GB card. The ASCII-condensed version trims non-ASCII vocabulary down to ~13.6 GB (12.6 GiB), leaving ~3.4 GiB of usable VRAM. Even with a Q8_0/Q4_0 KV cache, 262k context needs ~7.4 GiB for KV tensors alone, triggering OOM past ~65k tokens on llama.cpp.

I came across RaymondHuang210129/llama.cpp-adaptive-kv-streaming, which introduced adaptive KV streaming for NVIDIA/CUDA but did not support AMD/ROCm. I ported this to ROCm/HIP, retaining the full 262k cache in host RAM (~7.4 GB) while asynchronously streaming bounded working sets (~450 MiB prefill, ~2,100 MiB decode) over PCIe SDMA alongside compute.

I also incorporated kernel optimizations from stew675/llama-cpp-rdna-boosts (4-row multi-column mmvq, 16-byte vectorized loads for Q4_K/Q5_K, SwiGLU fusions, and single-queue async HIP graphs).

Further improvements:

• Sibling activation reuse: Reuses quantized activations across parallel projections (gate/up and Q/K/V) to avoid redundant memory passes.
• In-register fused residual add + RMSNorm: Retains activations in VGPRs between passes and directly writes quantized q8_1 copies for mmvq.
• Multi-column SwiGLU & GDN fusions: Extends fused Gate/Up + SwiGLU to 4-column verify batches in mmvq and fuses recurrent linear-attention gates for hybrid layers.
• Online Log-Sum-Exp prefill & pipelined prefetch: Computes chunked prefill across streamed pages via online LSE accumulation and overlaps layer L+1 PCIe transfers with layer L compute.
• Persistent graph arenas: Reuses arenas across prefill/decode phase switches to eliminate teardown overhead.
• Stream-optimized speculative decoding: Merged single-head MTP verify catch-up into the first draft step (cutting MTP forward passes in half and preventing draft KV pollution), dedicated arenas for verify graphs to eliminate rebuild bubbles, extended paged decode attention to verify batches, and stacked opportunistic n-grams with probabilistic rejection sampling.

Benchmark performance:

• Decode: 36.5 t/s (1k) -> 32.1 t/s (65k) -> 27.0 t/s (131k) -> 9.5 t/s (262k).
• Decode (with MTP): 71.8-86.5 t/s (1k-4k) -> 72.9 t/s (65k) -> 78.6 t/s (131k) -> 31.1 t/s (262k).
• Prefill: 1368.4 t/s (1k) -> 1120.3 t/s (65k) -> 863.0 t/s (131k) -> 585.8 t/s (262k).

Operating the 9070 XT at a 231W limit (-24% relative to 304W default TDP) incurs only a 3.6% prefill latency penalty and delivers virtually identical decode throughput, saving power and keeping the card quiet during long runs.

Take a look at the repository for the full details: https://github.com/achyudh/llama.cpp/tree/feature/adaptive-kv-stream-rocm

For reference, I’m not some ML/AI guru or pro by any means, just a guy who likes AI and whatnot. I’m running an R9700 with 64GB of DDR4 RAM, and the model I’m using is Orca Uncensored Qwen 3.8 27B.

My use case for uncensored models isn’t only asking questions and getting info that frontier models won’t answer. I’m also really into red team cybersec and somewhat dangerous research on the tech side.

Anyway, I’ve read about models with the guardrails removed, and I’m wondering how you make sure the model doesn’t just hallucinate or misuse tools. Assuming the guardrails are gone, how do you make sure only the refusal part is disabled and the rest of the intelligence is still there? I don’t want it to just yes-man me all the time and give me false info.

Is there some kind of local AI hygiene, harness optimizations, or settings that keep the model from making stuff up or getting stuck in an answer loop where it runs out of tokens?

I’m new to local AI models. Assume someone has a Mac Studio M5 Ultra 256GB. I’ve read that is the realistic minimum you can use to run a local AI model.

My question is this: as models get better, will they run better or have more capable models run better on the same hardware? Meaning will a Mac Studio M5 Ultra 256GB get more capable over time? Or does it just end up like a hamster wheel where models get better but you ultimately need better hardware to run them?

I have a $20 chatGPT Plus subscription and I use Codex almost every day. The problem is the usage sucks, I run out of my 5 hour usage very fast for what I would consider pretty basic and straight forward frontend development work.

I also have a 3090 (24gb) and a 5060ti (16gb) and 64gb of DDR5 ram in my personal computer so that's 40gb of Vram w/ 64gb of extra memory if needed so I am capable of running local models.

My problem I'm running into is.. when I try to do everything with a Frontier model I run out of my usage very fast, when I try to do everything with a local model I find that it's not as perfect, there are error that need to be corrected, there's more guard rails you need to put in place which makes me just want to go spend more money and buy a bigger Frontier subscription which is NOT what I want to do.

So I'm looking at finding the balance here to be productive. I guess it's a popular thing for people to utilize frontier models to plan and architect their apps and then they give the instructions that the frontier model wrote out to their local model and have the local model follow the instructions and code.

This in theory makes sense but my question is what does this actually look like in practice? Lets say I want to make a Twitter clone, what would I prompt to chatGPT? Would I say:

"I want to make a social media site like Twitter, write out a detailed plan I can give to my local model to follow and implement" and then I give that plan.md file to my local model and let it go to town?

How do you guys utilize your frontier subscriptions and your local models to get the most out of them?

Howdy folks!

About a week ago I started uploading my tests on my own website, since then I've made improvements to how that data is displayed, so you can see the actual prompts and settings used to produce the result. https://b65.thekaiser.us/

I also went through a number of runs on https://www.localmaxxing.com/en/user/imthekaiser and since I'm the first B65 on that site, (in 1x and 4x) you may want to check out the results!

i’m still testing more models, if there’s something specific you want to see, let me know!

some examples in the pics, more on this site!

these are single user numbers, concurrency numbers are much higher, and impressively so.

overall, I have been very happy with these cards. They perform better than I expected at the price point. The B70 is still a bit faster in certain tasks due to the compute, but it’s a lot closer than I expected. And if you can get a B65 for the right price, it can be a viable option to run models in production. Software stack has improved a lot and I don’t see why it can’t be used with most popular models today, albeit CUDA obviously still has the edge here, being the granddaddy.

I do suggest you stay with the official Intel VLLM stack and not to try and run on an unoptimised one.

How was AI Used in this post?
It wasn't used to write this.
it was used to build the reports site.
I spent over 40 hours reading intel documentation and testing vllm scalar to learn best practices and how to validate and baseline systems. That's why the first set of runs on my own site isn't comprised of top performance runs, it was validating and learning. Now that I'm proficient, and have a baseline, I'm testing ways to increase performance)
I used that experience to build a closely gated plan that instructs my AI on how to test and monitor the test machine.

Research Record III — AKBASCORE Silicon Hippocampus v1.4

https://doi.org/10.5281/zenodo.23434686

TL;DR: We built and experimentally verified a physical numerical memory engine that intervenes directly inside a frozen 72-billion-parameter transformer.

Instead of retrieving external text passages at inference time (conventional RAG), re-evaluating long input prompts, or merely replaying a saved KV cache, AKBASCORE captures intermediate hidden representations (H3) from the model's own computation, stores them as independent physical tensor cartridges, and reinjects them directly at the Layer-3 boundary (CUT3).

Three independently stored 10,000-token cartridges were assembled into a 30,031-position operational memory on a single NVIDIA A100-80GB, without changing the pretrained model weights.

This is not conventional KV caching. The defining mechanism is direct internal-state injection.

THE ARCHITECTURAL BREAKTHROUGH

Conventional approaches to long-term LLM memory generally rely on external retrieval, repeated context processing, or attention-cache preservation.

AKBASCORE takes a different route.

The frozen transformer processes source material once. At a model-specific intermediate boundary, we extract the numerical representation generated by its own layers.

That representation becomes a persistent, independently stored physical memory cartridge.

When the cartridge is mounted, the original source text is not resubmitted as an ordinary prompt.

Instead, the saved H3 representation is restored directly into the model's internal computational pathway. The downstream transformer layers resume computation from that injected numerical state.

The model is not fine-tuned. Its weights are not modified. The memory is carried by model-native numerical representations, not an external text database.

The key distinction is where the intervention occurs.

We are not asking the model to read a retrieved document again.

We are restoring a previously extracted intermediate state of the model's own computation.

HOW THE ENGINE WORKS

The Qwen2.5-72B implementation uses CUT3, the output boundary of transformer Layer 3.

During cartridge construction:

1. The frozen model processes each source excerpt in 256-token chunks.

2. The system captures the H3 hidden representation and the associated lower-layer numerical states.

3. H3 and the key/value tensors from Layers 0–3 are persisted as a physical cartridge.

4. The original source text is no longer required for subsequent cartridge mounting.

During cartridge reconstruction:

1. The saved numerical cartridge is loaded from storage.

2. Its positional representations are adjusted for the target memory layout.

3. The stored H3 representation is injected directly at CUT3.

4. Layers 4–79 reconstruct the remaining internal computational states.

5. The resulting memory becomes available to the frozen model for subsequent inference.

Key/value tensors participate in the reconstruction and integrity mechanism. They are not the defining innovation.

The central operation is direct restoration of intermediate hidden representations into the transformer computation itself.

THREE PHYSICAL CARTRIDGES — ONE 72B MODEL

The live experiment used:

Model: Qwen2.5-72B-Instruct

Parameters: 72 billion

Quantization: NF4, with BF16 computation

Transformer layers: 80

Hidden dimension: 8,192

Attention geometry: 64 query heads / 8 KV heads / 128 head dimension

Intervention boundary: CUT3

GPU: NVIDIA A100-SXM4-80GB

Three independent cartridges were constructed from 10,000-token excerpts:

Cartridge A — Pride and Prejudice

Cartridge B — Frankenstein

Cartridge C — The Adventures of Sherlock Holmes

Each cartridge occupies exactly 328,698,863 bytes on disk.

The combined operational memory contains 30,031 token positions.

The reconstructed 80-layer key/value memory occupies approximately 9.16 GiB.

These are physical numerical representations extracted from the transformer, not textual summaries, retrieved passages, fine-tuned adapters, or modifications to the model weights.

LIVE EXPERIMENTAL VERIFICATION

Session: Q72S-20261017-083524

The live Qwen2.5-72B engine completed:

11/11 scripted physical protocol trials.

17/17 engine checks.

8/8 append integrity checks.

11/11 additional live verification checks.

Three independently verified cartridge files.

Successful reconstruction of the combined A+B+C memory.

Exact SHA-256 preservation of previously mounted key/value states across all 80 layers during append operations.

Verification that the frozen-model weight sentinel remained unchanged.

Live examination of the reconstructed GPU memory across 80 transformer layers and 30,031 token positions.

The underlying TEST709 record also reports a peak GPU allocation of approximately 57.66 GiB during the three-cartridge protocol on an A100-80GB.

The implementation maintains a 9.0 GiB GPU safety-reserve requirement.

FIGURE 4 — INSIDE THE RECONSTRUCTED 72B MEMORY

Figure 4 presents a read-only measurement of the live 80-layer GPU key/value state across the 30,031-position combined memory.

The visualization exposes the three cartridge regions and their boundaries.

The CUT3 boundary separates the lower-layer states restored from persistent numerical cartridges from the downstream layers reconstructed by the frozen transformer.

This is a measurement of the actual reconstructed tensor state, not an illustrative or synthetic neural activation diagram.

The accompanying figures document the intervention protocol, physical storage architecture, reconstruction performance, observational question-answering results, and integrity verification.

HOT-SWAP AND INCREMENTAL RECONSTRUCTION

The memory cartridges can be independently mounted, appended, reordered, and restored.

Measured reconstruction times were approximately:

Single cartridge, 10,031 positions: 17.6 seconds for reconstruction, approximately 18.9 seconds end-to-end.

Two-cartridge assembly, 20,031 positions: approximately 46 seconds for reconstruction.

Three-cartridge assembly, 30,031 positions: approximately 74 seconds for reconstruction.

These are measured physical reconstruction operations, not instantaneous pointer switches.

The important result is not that rebuilding the internal state costs zero computation.

It is that the original source text does not need to be reprocessed through the complete transformer from the input layer. The persisted intermediate state provides a new computational starting point.

The append-integrity protocol verifies that previously mounted key/value states remain bitwise unchanged across all 80 layers.

SCIENTIFIC BOUNDARY

The physical reconstruction and integrity results are experimentally verified.

The semantic question remains separate.

Our demonstration uses public-domain literary classics, which may already be represented in the model's pretraining data. Therefore, correct answers about these books cannot independently establish that the information was acquired from the cartridges.

The current results demonstrate direct internal-state restoration, physical memory reconstruction, incremental assembly, and numerical integrity at 72B scale.

They do not yet establish causally isolated semantic transfer, general reasoning improvement, or complete comprehension of the stored books.

Those are the next experimental targets.

(Note: Subsequent Phase 2 testing will replace public-domain excerpts with unseen, post-2024 technical text and synthetic data to definitively isolate semantic transfer from pretraining memory.)

WHY THIS MATTERS

The architectural question is larger than how to extend a context window.

Can a transformer export part of its internal computation, preserve that numerical state independently of its weights, and later resume computation from the restored state?

AKBASCORE Silicon Hippocampus demonstrates a working physical implementation of that mechanism inside a frozen 72-billion-parameter model.

The significance is the separation of persistent numerical memory from the model's permanent parameters and from the original source text.

The result is an experimentally verified pathway for storing, transporting, assembling, and restoring model-native intermediate computational states.

If future controlled experiments establish reliable causal knowledge transfer and scalable addressing, this architecture could become a foundation for persistent modular memory in long-running AI systems.

This release documents the working engine and the evidence behind it.

RESEARCH LINEAGE

Research Record I — Qwen2.5-7B

https://doi.org/10.5281/zenodo.23381336

Research Record II — Mistral-7B, Silicon Hippocampus v1.4

https://doi.org/10.5281/zenodo.23418927

Research Record III — Qwen2.5-72B, Direct Internal-State Injection

https://doi.org/10.5281/zenodo.23434686

SOURCE CODE AND EXPERIMENTAL RECORDS

GitHub repository:

https://github.com/ceceli33/titan-cognitive-core-v3

Complete executable Qwen2.5-72B demonstrator:

https://github.com/ceceli33/titan-cognitive-core-v3/blob/main/MAM_full_QWEN_72B.py

The same code is divided into three consecutive parts for mobile users. Paste all three parts sequentially into the same Google Colab code cell and execute once.

Part 1:

https://github.com/ceceli33/titan-cognitive-core-v3/blob/main/MAM_part1_QWEN_72B.py

Part 2:

https://github.com/ceceli33/titan-cognitive-core-v3/blob/main/MAM_part2_QWEN_72B.py

Part 3:

https://github.com/ceceli33/titan-cognitive-core-v3/blob/main/MAM_part3_QWEN_72B.py

Live demonstration logs:

https://github.com/ceceli33/titan-cognitive-core-v3/blob/main/MAM_LOG1_QWEN_72B.log

https://github.com/ceceli33/titan-cognitive-core-v3/blob/main/MAM_LOG2_QWEN_72B.log

Underlying physical engine:

https://github.com/ceceli33/titan-cognitive-core-v3/blob/main/709.py

Underlying physical experiment log:

https://github.com/ceceli33/titan-cognitive-core-v3/blob/main/709.log

License:

https://github.com/ceceli33/titan-cognitive-core-v3/blob/main/LICENSE

AKBASCORE Research Software License. Research and educational use are permitted under the published terms; commercial use is subject to the license's consolidated annual revenue conditions.

Mustafa Akbaş

AkbasCore AI Teknoloji

Mersin, Türkiye

Copyright © 2026 Mustafa Akbaş

Paying $200/mo for cloud subscriptions has gotten completely unsustainable here in Brazil. Because of that, I’m building a dedicated local rig specifically to run Qwen 3.8-Flash-Next and move away from cloud providers entirely.

​My issue right now is the harness. Every tool I've tested (Cline, OpenCode, Antigravity, etc.) falls apart after a short run, and while I checked out Hermes, the interface really doesn't appeal to me. Even when tasks are broken down into simple steps, clear sub-plans, and straightforward rules, local agents never seem to sustain long-horizon work like Codex or Claude do. They either stall, derail, or stop entirely after an hour or two.

​I have massive workloads to chew through and need to be able to let this run overnight without having to babysit the screen for 16 hours straight.

​What app, harness, or agent workflow are you using to keep local models continuously grinding through tasks until completion? How are you structuring your execution loop to prevent them from dropping the ball mid-run? Using /goal isn't enough here....

Edit: Yes I already have a tasks check-list, a dispatch to never create a plan forgetting something, and a central plan that appoint to the address of the 200 fucnk tasks and each one have a mini plan with rules, 3 or 4 skills, and nothing more to don't full the context.

Bro 200 fuckn tasks open and the model say "all done, finished".

(i don't know how much I've done, I know that the tasks check-list already are at 49k lines of code, and no, the model never reads this, I know that is impossible, the model reads a single plan, no more than 300 lines of python code)

AMD cards are so much cheaper than Nvidia it's absurd, I can get a top of the line AMD GPU for CHEAPER than a 5060Ti.
Do people want CUDA that badly? I plan on using CPU inference with the GPU being used for stuff that needs it like prefill, additional context, maybe loading the image gen model there..

My current setup (and all the desktops I've ever built) have used Nvidia cards, so I can't really just swap and compare myself.

Hey everyone!

A few weeks ago, I tried out a rog flow 13 with strix halo and 64gb of RAM. Speed for the models I care about was slow, but it was generally usable. The tablet format just isn't for me though. A friend also let me try their M5 Max Mac Studio with 36gb of RAM. It was much, much faster but RAM filled up completely, so it can only be an AI box and only for qwen 3.8 27b at limited context. Not ideal for the price.

It's worth noting there are no 128gb strix halo boxes or laptops I can easily buy in my country. They would all come from somewhere else and not have good guarantee. All strix halo laptops cap out at 64gb. MacBooks cap out at 48gb, anything above that is a custom order and will take 8w+ and I need a laptop sooner, my MacBook is dying on me and I can't work properly with it anymore.

So, now RTX spark is out, and these ARE available. They won't be soon - I'm sure of that. I'm seriously considering buying the ProArt 16 w/ 128gb of RAM... it's about $5.8k. Yes, it's crazy. But it's vastly cheaper than the mac (>$8k) and the only machine I can even buy with normal guarantee from a reputable store here. The cheapest strix halo box around here w/ 128gb of RAM is approx $3.8k, so $2k more to get a premium laptop and more powerful chip with it doesn't seem toooo crazy.

I'm 100% sure these will go up in price soon. Right now, they cost LESS than in the US and wayyyyy less than DGX spark (now at almost $8.5k or so here). The prices are just too much lower and will adjust. Plus they ordered like 3 so I have to decide today.

Thanks!

Your PC does the work. You get the proof.

We built Sonjari, a local agent for Windows. The model runs on your own GPU through Ollama.

'Sonjari' does browser work in your own Chrome, right in front of you. The model runs on your own GPU through Ollama. It uses your Chrome as it is, logins included, and when it hits a login, a CAPTCHA or a payment screen, it stops and hands it back to you. You can watch it open pages, search inside a site, and go into the product page to read it.

• Setup

- Runtime: Ollama 0.35.1, Windows 11, one request at a time

- Model: Swift-1.5-Qwen3.8-27B-GSQ-RCO, IQ3_S GGUF with MTP (draft 4), vision on

- Window: 131072, fully on the card (about 83 tok/s generation on this PC)

- Card: RTX 3090 24 GB

- Sampling for agent steps: temperature 0.1, min_p 0.01

- Approvals auto-passed for these tests (normally every window action waits for a click)

Sonjari is a free local agent for Windows that we build as a two-person team. It does web research, Excel and Word files, and code on your own PC, and checks its answer against what its tools actually returned.

Operating ordinary desktop apps was its weakest part, so this week I measured only that: ten everyday jobs in ten apps (Notepad, Calculator, Paint, Excel, Word, PowerPoint, Hangul (the Korean word processor) to PDF, File Explorer, VS Code, Task Manager), three runs each.

1. Forced through each app's own window

- 5 of 10 before fixes, 30 of 30 after.

- Slow: Word 213 to 385 s, Excel 297 to 614 s. A person does these in under a minute.

- What broke: menus that accept the UI Automation "invoke" call and do nothing, Task Manager running elevated so it can't be read, a HEIC saved as .jpg because the file type box was typed into instead of picked, and `code file.txt` holding the output pipe open for 15 minutes.

2. Same jobs, normal wording ("make a Word file with this sentence")

- 30 of 30. Excel, Word and PowerPoint went through its own file tools in 16 to 20 s.

- Notepad, Paint, VS Code and Hangul still went through the window: 84 to 281 s.

3. Fixes

- Naming an app now means the result's format, not "operate that window". Notepad 24 to 52 s, VS Code 24 to 28 s, Paint 60 to 96 s.

- Hangul to PDF got its own tool (hidden instance, save as PDF, check the file starts with %PDF): 20 to 24 s, down from 100 to 281 s.

The bug I'm least proud of

The PDF tool passed its tests, but the first three real runs went 0 of 3. When I asked the model whether hwp_export_pdf was in its tool list, it said no. A routing rule had filed "save this .hwp as PDF" under web research, and that route drops the Hangul tools. The tests never checked what the model actually sees.

Limits

One PC, the largest model, three runs per app, single-step jobs. Smaller cards weren't measured on these.

For people running agents on Windows: do you let them drive app windows at all, or keep them to files and APIs?

Repo and the raw runs for all three parts:
https://github.com/sonjari/sonjari (measurements/2026-10-17-desktop-apps) :)

TL;DR, Tesla M40 24gb GPU, DDR3 ram, and MoE polystrata model benchmarks & information.

Here is a reference for anyone considering doing a cheap DDR3 server PC build with an older GPU on a budget. Below are some rough "benchmarks" for speed of recent flash models.

The Tesla M40 GPU I used in this rig was released back in 2015 on the same architecture as the GTX 900 series and GTX Titan X. They can be found on eBay nowadays in the $50 range for 12Gb models, $100-$170 for the 24Gb models. They are rated for 250 watts and are typically "passive" expecting server fans to keep them cool, for these tok/s generation speeds, it was limited to 180 watts.

PC Specs below:

• Dell T420 Tower PC
• Dual Xeon E5-2470 @ 2.3 GHz
• 192 Gb DDR3 ECC ram at 1600 MHz
• Nvidia TESLA M40 24GB GPU (limited from 250 to 180 watts for these tests)
• PCIE NVME SSD Riser with a 256gb NVME drive for model storage/loading

Forking Polystrata to work with the older CUDA 5.2 compute capability, I ran some simple "make me an aero/glass style calculator" prompts on each of the following models. This should give anyone thinking about picking up a cheap Tesla M40 an idea on the generation tok/s "performance" you'd get.

Swift 1.5 Qwen 3.8 Flash Next @ IQ3_S:

• 19-20 tok/s

GLM 5.3 Flash @ IQ3_XXS:

• 3-4 tok/s
  • Note, I think GLM 5.3 Flash is CPU bound in my bench. The GPU usage never went above 40%, while CPU usage was pegged above 80%.

Qwen 3.8 27B @ UD_Q4_K_S:

• 2-3 tok/s
  • This speed is with 100% GPU offload. I am surprised the generation speed is THIS slow, even with the Dense model. But I guess that's what a 2015 GPU gets you.

I am working on benchmarks for other models such as the Nemotron 3.5s, Qwen 3.6 35B-A3B, and a higher quant QW 3.8 Flash Next. If you want to see performance of a certain model, context size, prompt outputs, input speed, etc, let me know and I'll get it done.

Just picked up a second PC and want to combine parts into one machine for local AI (LLMs + ComfyUI video). Looking for advice on the best layout.

PC 1 (current main):
- Ryzen 5 5600X
- ASUS Prime B550-PLUS (2nd x16 slot is PCIe 3.0 x4 via chipset)
- 64GB DDR4-3200 (4x16)
- Gigabyte AORUS RX 6800 Master 16GB (~318mm, ~3 slot)
- 850W 80+ Gold PSU
- AORUS case

PC 2 (just bought):
- i5-13600KF
- MSI PRO B760-P WIFI (DDR5)
- 32GB DDR5
- MSI RTX 5060 Ti 16GB (2-slot)
- 1000W PSU
- MSI 240mm AIO, iBUYPOWER case (RX 6800 likely won't fit)

Plan right now: move the 1000W PSU + 5060 Ti into PC 1, keep 64GB DDR4,
RX 6800 on top slot as display/gaming card, 5060 Ti in the bottom x4 slot.
Sell the leftover PC 2 parts.

What I use it for:
- LM Studio (Vulkan) — Qwen3.6 35B-A3B, 27B dense models
- ComfyUI — Wan 2.2 I2V for short product videos
- Some gaming (Rust)

Questions:
1. Is mixing AMD + NVIDIA in LM Studio/llama.cpp Vulkan worth it, or are there gotchas?
2. Better to keep 64GB DDR4 (B550) or switch to the 13600KF/DDR5 platform with 32GB?
3. Which card should be on the x16 slot vs the x4 slot for AI?
4. Anyone running ComfyUI on a 5060 Ti in a PCIe 3.0 x4 slot — how bad is the slowdown?
5. Would you sell the RX 6800 and get a 2nd 5060 Ti / used 3090 instead?

Any tok/s numbers from similar setups would be awesome. Thanks!

Posted a hardware checker last week and got fair criticism: old models, missing GPUs, no quants below Q4, no Strix Halo. Fixed all of it, plus the feature people asked for most:

- What fits my GPU? Pick your hardware, context and KV quant, get every model that fits at its best quant, with headroom. 2x 3090 at 8k: https://willitrunllm.com/fits/?hw=rtx-3090,rtx-3090&ctx=8192&kv=f16
- Strix Halo 32-128GB with Windows/Linux memory splits and measured llama.cpp speeds
- IQ1_S through IQ4_XS, with a quality warning
- Multi-GPU (VRAM adds up), RAM offload, context/KV-cache slider
- Search any Hugging Face GGUF repo; uses real file sizes
- Embedding models, checked against what's left next to your chat model
- DeepSeek V4.1 Flash and Muse Spark Open added

Every number is either sourced or labeled as an estimate. Weakest part is speed data, so if you paste your llama-bench output at willitrunllm.com/submit it goes in after review.

Free, no signup. What's still wrong or missing?

Found this funny, came out of nowhere cause I find Claude in general always to be very serious.
Was using a mixture of multiple cloud LLMs and local ones for this project, and Claude handed off a task to Qwen, advising me to run Qwen for 3x 10h sessions, and then insisted on verifying the work Qwen had done. Did some testing myself with Kimi K3 taking the place of Claude and I got the same result much faster than the way Claude suggested (task was done in under 30 minutes instead of days).

So I questioned Claude why it made that suggestion. Than it got cocky. Too bad it couldn't slap it in the face with a large trout.

I've spent the last few months tuning a 27B coding model on one Arc Pro B60 24GB (A$999 card) and published everything: engine build, flags, raw logs, a 100-task agent eval, and the misses.

What moved the numbers:

- DFlash2 draft model, 7 tokens per round: decode 16 -> 53 tok/s mean (47-58 across 7 coding tasks), still 7/7 correct

- Intel's flash-attention prefill kernel (closed llama.cpp PR #29357, vendored with checksums): prompt reading at 32K 118 -> 420 tok/s, but only with f16 KV

- -ub 256 + q8 draft KV + -fit off: frees 0.55 GiB, which is what gets a 64K window into 24GB

- An idle Ollama model left on the GPU cut decode from 52.7 to 22.2 tok/s. Unload everything before you benchmark.

Agent eval: 97/100 hidden-checker tasks with Claude Code at a 32K window (95/100 Codex).

Caveats, up front: 53 tok/s is per-request decode with short prompts and a 64K window allocated. With 30-37K of real context, agent turns run about 42 tok/s. Prefill is NOT the fastest out there (others hit 589-657 tok/s at ~30K on other engines; comparison table in the repo). Single runs, one workstation.

Q4_K_M for both models:

- Main model: Qwen3.8-27B UD-Q4_K_M (Unsloth Dynamic quant), 16.46 GB file

https://huggingface.co/unsloth/Qwen3.8-27B-GGUF

- Draft model: Qwen3.8-27B-DFlash2 Q4_K_M

- KV cache: f16 for the main model (Intel's FA prefill kernel only speeds things up with f16; q4 KV gained nothing), q8_0 for the drafter's KV, which saves ~0.55 GiB and is what fits a 64K window in 24GB

The 53 tok/s run used the UD-Q4_K_M file above. Since then I've switched my daily driver to the ThinkingCap Q4_K_M build (16/16 vs 15/16 on my task set). Exact files and flags are in the README under "Versions used".

Repo: https://github.com/edagher92-coder/arc-llm-lab

If you have a B580, B570, B50, B70 or A770, the benchmark script takes one command and I'll add your numbers with credit.

Like the majority, I am not able to run big models myself. The 1.58b model Ternary Bonsai 2 27B was just about too big to fit on my GPU. So I removed the 7 least relevant layers per ΔKL. This brought the model down from 27B parametres to only 24.2B and it now fits neatly on a 6GB GPU.

Felt like sharing it here, maybe y'all find a use for it. Here's the HF link:
henceiusegentoo/Ternary-Bonsai-2-24B-PTQ1_0-GGUF · Hugging Face

To run it, you need the PrismML fork of llama.cpp

Old Tesla V100-SXM2 16 GB (Volta, 2017), plain Windows 11, no WSL/Docker.

MiniCPM-V 4.7 35B-A3B: ~93 tok/s, vision + 128k context, one 16 GB card.

No custom kernel, no custom quant: stock llama.cpp + OpenBMB's official mix_q3gud_im (15.5 GB). Just tuned settings:

• --no-mmproj-offload (vision encoder on CPU) frees ~0.9 GB, so all MoE experts fit on the GPU (--n-cpu-moe 0)
• -c 131072 -ctk q4_0 -ctv q4_0 -fa on

Filled with a 120k-token prompt: correct answer, no crash (16.17 / 16.38 GB VRAM).

Speculative decoding (DSpark, n-gram) was slower: the drafter's VRAM pushes experts back to RAM.

Benchmarked 15 models on my own hard suite: 72 tasks, code-graded, no LLM judge (coding, reasoning, instructions, search, agent, vision, long context). Top 5:

The repo is one portable folder: a GUI (model/engine picker, downloads, live tok/s, VRAM/CPU/RAM meters, load/unload), engines prebuilt for sm_70 / CUDA 12.9, full leaderboard, demo videos, and EVALUATE.bat to test your own models.

https://github.com/Abdelhayali/V100-LLM

Ryzen 9600X, 32GB dual channel DDR5, 2x CMP 170HX
Ryzen 5700, 32GB dual channel DDR4, 5x Vega 56
Intel i7 9700, 32GB dual channel DDR4, 3070 + 2080ti
Ryzen 7500x3D, 32GB dual channel DDR5, 7900xtx + 2x 5600XT

223 GB total VRAM

I've seen a few AI enthusiasts switching to WRX80 for the 128 lanes and 8-channel memory recently.

There isn't much info on the subject so here is the evidence, it's 8 x 64gb 2933.

I've also added the result of the same test on the Asrock ROMED8-2T.

Hi, I created a Roles Witch - privacy-focused AI assistant featuring voice/video calls and on-device inference (via LiteRT for Gemma models). It also easily connects to any OpenAI-compatible API (LM Studio, OpenRouter, etc.).

Unlike typical Character AI like apps, Roles Witch supports MCP tools. This means it can actually interact with the real world — whether that's controlling your smart home via Home Assistant or browsing the web.

It's 100% free and open-source!

https://play.google.com/store/apps/details?id=ai.roleswitch.android&hl=en

https://github.com/RolesWitchAI/RolesWitchOSS