I asked my local AI agent to optimize its own setup.

The first thing it did was stop the inference server.

The one it was running on. 🤦

Fair enough. Can’t beat zero resource usage.

So we established a rule: don’t shut yourself down.

On the next attempt, it cleaned up some Python processes.

Including itself.

The rule now has a second paragraph.

I just tried to install QFN from unsloth (Q4_K_XL) on my strix halo laptop with 128gb of ram. Set it up with Gufo port for windows, using opencode v2 on my macOS as the harness and the effort is set to high.

I wanted to do some testing and compare it with opus 5.5 xhigh with the official claude code harness (my daily driver). So i gave both model the same prompt which is to migrate several screen from flutter to swiftUI from scratch. The app is a fully funnctional app with a lot of logic and network calls, and it is used in production with thousands of DAU

Opus managed to do it in 50 mins and what amazes me is QFN finishes it in 60 mins, and the result is pretty identical!

sure it is not as polished as opus' result but I am sure with 2-3 more prompts and 5-10 mins it could produces what Opus 5.5 xhigh produces (just some styling work)

I really hope that qwen 4 27b could matches opus 5.5 in coding work, fingers crossed 🤞If yes, the gap between china model and frontier model are only 2-3 months

I have been working on implementing adaptive KV streaming and RDNA 4 / ROCm optimizations for llama.cpp to enable running Qwen3.8-27B (IQ4_XS weights, Q8_0/Q4_0 KV cache) with its full 262,144-token context window on a single 16 GB RX 9070 XT GPU at 80+ t/s decode.

Qwen 27B IQ4_XS is ~14.3 GB, leaving insufficient VRAM headroom on a 16 GB card. The ASCII-condensed version trims non-ASCII vocabulary down to ~13.6 GB (12.6 GiB), leaving ~3.4 GiB of usable VRAM. Even with a Q8_0/Q4_0 KV cache, 262k context needs ~7.4 GiB for KV tensors alone, triggering OOM past ~65k tokens on llama.cpp.

I came across RaymondHuang210129/llama.cpp-adaptive-kv-streaming, which introduced adaptive KV streaming for NVIDIA/CUDA but did not support AMD/ROCm. I ported this to ROCm/HIP, retaining the full 262k cache in host RAM (~7.4 GB) while asynchronously streaming bounded working sets (~450 MiB prefill, ~2,100 MiB decode) over PCIe SDMA alongside compute.

I also incorporated kernel optimizations from stew675/llama-cpp-rdna-boosts (4-row multi-column mmvq, 16-byte vectorized loads for Q4_K/Q5_K, SwiGLU fusions, and single-queue async HIP graphs).

Further improvements:

• Sibling activation reuse: Reuses quantized activations across parallel projections (gate/up and Q/K/V) to avoid redundant memory passes.
• In-register fused residual add + RMSNorm: Retains activations in VGPRs between passes and directly writes quantized q8_1 copies for mmvq.
• Multi-column SwiGLU & GDN fusions: Extends fused Gate/Up + SwiGLU to 4-column verify batches in mmvq and fuses recurrent linear-attention gates for hybrid layers.
• Online Log-Sum-Exp prefill & pipelined prefetch: Computes chunked prefill across streamed pages via online LSE accumulation and overlaps layer L+1 PCIe transfers with layer L compute.
• Persistent graph arenas: Reuses arenas across prefill/decode phase switches to eliminate teardown overhead.
• Stream-optimized speculative decoding: Merged single-head MTP verify catch-up into the first draft step (cutting MTP forward passes in half and preventing draft KV pollution), dedicated arenas for verify graphs to eliminate rebuild bubbles, extended paged decode attention to verify batches, and stacked opportunistic n-grams with probabilistic rejection sampling.

Benchmark performance:

• Decode: 36.5 t/s (1k) -> 32.1 t/s (65k) -> 27.0 t/s (131k) -> 9.5 t/s (262k).
• Decode (with MTP): 71.8-86.5 t/s (1k-4k) -> 72.9 t/s (65k) -> 78.6 t/s (131k) -> 31.1 t/s (262k).
• Prefill: 1368.4 t/s (1k) -> 1120.3 t/s (65k) -> 863.0 t/s (131k) -> 585.8 t/s (262k).

Operating the 9070 XT at a 231W limit (-24% relative to 304W default TDP) incurs only a 3.6% prefill latency penalty and delivers virtually identical decode throughput, saving power and keeping the card quiet during long runs.

Take a look at the repository for the full details: https://github.com/achyudh/llama.cpp/tree/feature/adaptive-kv-stream-rocm

I’m new to local AI models. Assume someone has a Mac Studio M5 Ultra 256GB. I’ve read that is the realistic minimum you can use to run a local AI model.

My question is this: as models get better, will they run better or have more capable models run better on the same hardware? Meaning will a Mac Studio M5 Ultra 256GB get more capable over time? Or does it just end up like a hamster wheel where models get better but you ultimately need better hardware to run them?

Paying $200/mo for cloud subscriptions has gotten completely unsustainable here in Brazil. Because of that, I’m building a dedicated local rig specifically to run Qwen 3.8-Flash-Next and move away from cloud providers entirely.

​My issue right now is the harness. Every tool I've tested (Cline, OpenCode, Antigravity, etc.) falls apart after a short run, and while I checked out Hermes, the interface really doesn't appeal to me. Even when tasks are broken down into simple steps, clear sub-plans, and straightforward rules, local agents never seem to sustain long-horizon work like Codex or Claude do. They either stall, derail, or stop entirely after an hour or two.

​I have massive workloads to chew through and need to be able to let this run overnight without having to babysit the screen for 16 hours straight.

​What app, harness, or agent workflow are you using to keep local models continuously grinding through tasks until completion? How are you structuring your execution loop to prevent them from dropping the ball mid-run? Using /goal isn't enough here....

Edit: Yes I already have a tasks check-list, a dispatch to never create a plan forgetting something, and a central plan that appoint to the address of the 200 fucnk tasks and each one have a mini plan with rules, 3 or 4 skills, and nothing more to don't full the context.

Bro 200 fuckn tasks open and the model say "all done, finished".

(i don't know how much I've done, I know that the tasks check-list already are at 49k lines of code, and no, the model never reads this, I know that is impossible, the model reads a single plan, no more than 300 lines of python code)

I have a $20 chatGPT Plus subscription and I use Codex almost every day. The problem is the usage sucks, I run out of my 5 hour usage very fast for what I would consider pretty basic and straight forward frontend development work.

I also have a 3090 (24gb) and a 5060ti (16gb) and 64gb of DDR5 ram in my personal computer so that's 40gb of Vram w/ 64gb of extra memory if needed so I am capable of running local models.

My problem I'm running into is.. when I try to do everything with a Frontier model I run out of my usage very fast, when I try to do everything with a local model I find that it's not as perfect, there are error that need to be corrected, there's more guard rails you need to put in place which makes me just want to go spend more money and buy a bigger Frontier subscription which is NOT what I want to do.

So I'm looking at finding the balance here to be productive. I guess it's a popular thing for people to utilize frontier models to plan and architect their apps and then they give the instructions that the frontier model wrote out to their local model and have the local model follow the instructions and code.

This in theory makes sense but my question is what does this actually look like in practice? Lets say I want to make a Twitter clone, what would I prompt to chatGPT? Would I say:

"I want to make a social media site like Twitter, write out a detailed plan I can give to my local model to follow and implement" and then I give that plan.md file to my local model and let it go to town?

How do you guys utilize your frontier subscriptions and your local models to get the most out of them?

I’m trying to find the best local LLM for coding that can handle pretty much anything I throw at it without refusing all the time.

I still want good code quality though, not just some uncensored model that gives bad answers. Something that feels close to Claude/Opus level for coding would be ideal.

Im willing to spend money on hosted GPUs.

Any help would be appreciated!

Edit:
Good computer use skills is a must!

AMD cards are so much cheaper than Nvidia it's absurd, I can get a top of the line AMD GPU for CHEAPER than a 5060Ti.
Do people want CUDA that badly? I plan on using CPU inference with the GPU being used for stuff that needs it like prefill, additional context, maybe loading the image gen model there..

My current setup (and all the desktops I've ever built) have used Nvidia cards, so I can't really just swap and compare myself.

Research Record III — AKBASCORE Silicon Hippocampus v1.4

https://doi.org/10.5281/zenodo.23434686

TL;DR: We built and experimentally verified a physical numerical memory engine that intervenes directly inside a frozen 72-billion-parameter transformer.

Instead of retrieving external text passages at inference time (conventional RAG), re-evaluating long input prompts, or merely replaying a saved KV cache, AKBASCORE captures intermediate hidden representations (H3) from the model's own computation, stores them as independent physical tensor cartridges, and reinjects them directly at the Layer-3 boundary (CUT3).

Three independently stored 10,000-token cartridges were assembled into a 30,031-position operational memory on a single NVIDIA A100-80GB, without changing the pretrained model weights.

This is not conventional KV caching. The defining mechanism is direct internal-state injection.

THE ARCHITECTURAL BREAKTHROUGH

Conventional approaches to long-term LLM memory generally rely on external retrieval, repeated context processing, or attention-cache preservation.

AKBASCORE takes a different route.

The frozen transformer processes source material once. At a model-specific intermediate boundary, we extract the numerical representation generated by its own layers.

That representation becomes a persistent, independently stored physical memory cartridge.

When the cartridge is mounted, the original source text is not resubmitted as an ordinary prompt.

Instead, the saved H3 representation is restored directly into the model's internal computational pathway. The downstream transformer layers resume computation from that injected numerical state.

The model is not fine-tuned. Its weights are not modified. The memory is carried by model-native numerical representations, not an external text database.

The key distinction is where the intervention occurs.

We are not asking the model to read a retrieved document again.

We are restoring a previously extracted intermediate state of the model's own computation.

HOW THE ENGINE WORKS

The Qwen2.5-72B implementation uses CUT3, the output boundary of transformer Layer 3.

During cartridge construction:

1. The frozen model processes each source excerpt in 256-token chunks.

2. The system captures the H3 hidden representation and the associated lower-layer numerical states.

3. H3 and the key/value tensors from Layers 0–3 are persisted as a physical cartridge.

4. The original source text is no longer required for subsequent cartridge mounting.

During cartridge reconstruction:

1. The saved numerical cartridge is loaded from storage.

2. Its positional representations are adjusted for the target memory layout.

3. The stored H3 representation is injected directly at CUT3.

4. Layers 4–79 reconstruct the remaining internal computational states.

5. The resulting memory becomes available to the frozen model for subsequent inference.

Key/value tensors participate in the reconstruction and integrity mechanism. They are not the defining innovation.

The central operation is direct restoration of intermediate hidden representations into the transformer computation itself.

THREE PHYSICAL CARTRIDGES — ONE 72B MODEL

The live experiment used:

Model: Qwen2.5-72B-Instruct

Parameters: 72 billion

Quantization: NF4, with BF16 computation

Transformer layers: 80

Hidden dimension: 8,192

Attention geometry: 64 query heads / 8 KV heads / 128 head dimension

Intervention boundary: CUT3

GPU: NVIDIA A100-SXM4-80GB

Three independent cartridges were constructed from 10,000-token excerpts:

Cartridge A — Pride and Prejudice

Cartridge B — Frankenstein

Cartridge C — The Adventures of Sherlock Holmes

Each cartridge occupies exactly 328,698,863 bytes on disk.

The combined operational memory contains 30,031 token positions.

The reconstructed 80-layer key/value memory occupies approximately 9.16 GiB.

These are physical numerical representations extracted from the transformer, not textual summaries, retrieved passages, fine-tuned adapters, or modifications to the model weights.

LIVE EXPERIMENTAL VERIFICATION

Session: Q72S-20261017-083524

The live Qwen2.5-72B engine completed:

11/11 scripted physical protocol trials.

17/17 engine checks.

8/8 append integrity checks.

11/11 additional live verification checks.

Three independently verified cartridge files.

Successful reconstruction of the combined A+B+C memory.

Exact SHA-256 preservation of previously mounted key/value states across all 80 layers during append operations.

Verification that the frozen-model weight sentinel remained unchanged.

Live examination of the reconstructed GPU memory across 80 transformer layers and 30,031 token positions.

The underlying TEST709 record also reports a peak GPU allocation of approximately 57.66 GiB during the three-cartridge protocol on an A100-80GB.

The implementation maintains a 9.0 GiB GPU safety-reserve requirement.

FIGURE 4 — INSIDE THE RECONSTRUCTED 72B MEMORY

Figure 4 presents a read-only measurement of the live 80-layer GPU key/value state across the 30,031-position combined memory.

The visualization exposes the three cartridge regions and their boundaries.

The CUT3 boundary separates the lower-layer states restored from persistent numerical cartridges from the downstream layers reconstructed by the frozen transformer.

This is a measurement of the actual reconstructed tensor state, not an illustrative or synthetic neural activation diagram.

The accompanying figures document the intervention protocol, physical storage architecture, reconstruction performance, observational question-answering results, and integrity verification.

HOT-SWAP AND INCREMENTAL RECONSTRUCTION

The memory cartridges can be independently mounted, appended, reordered, and restored.

Measured reconstruction times were approximately:

Single cartridge, 10,031 positions: 17.6 seconds for reconstruction, approximately 18.9 seconds end-to-end.

Two-cartridge assembly, 20,031 positions: approximately 46 seconds for reconstruction.

Three-cartridge assembly, 30,031 positions: approximately 74 seconds for reconstruction.

These are measured physical reconstruction operations, not instantaneous pointer switches.

The important result is not that rebuilding the internal state costs zero computation.

It is that the original source text does not need to be reprocessed through the complete transformer from the input layer. The persisted intermediate state provides a new computational starting point.

The append-integrity protocol verifies that previously mounted key/value states remain bitwise unchanged across all 80 layers.

SCIENTIFIC BOUNDARY

The physical reconstruction and integrity results are experimentally verified.

The semantic question remains separate.

Our demonstration uses public-domain literary classics, which may already be represented in the model's pretraining data. Therefore, correct answers about these books cannot independently establish that the information was acquired from the cartridges.

The current results demonstrate direct internal-state restoration, physical memory reconstruction, incremental assembly, and numerical integrity at 72B scale.

They do not yet establish causally isolated semantic transfer, general reasoning improvement, or complete comprehension of the stored books.

Those are the next experimental targets.

(Note: Subsequent Phase 2 testing will replace public-domain excerpts with unseen, post-2024 technical text and synthetic data to definitively isolate semantic transfer from pretraining memory.)

WHY THIS MATTERS

The architectural question is larger than how to extend a context window.

Can a transformer export part of its internal computation, preserve that numerical state independently of its weights, and later resume computation from the restored state?

AKBASCORE Silicon Hippocampus demonstrates a working physical implementation of that mechanism inside a frozen 72-billion-parameter model.

The significance is the separation of persistent numerical memory from the model's permanent parameters and from the original source text.

The result is an experimentally verified pathway for storing, transporting, assembling, and restoring model-native intermediate computational states.

If future controlled experiments establish reliable causal knowledge transfer and scalable addressing, this architecture could become a foundation for persistent modular memory in long-running AI systems.

This release documents the working engine and the evidence behind it.

RESEARCH LINEAGE

Research Record I — Qwen2.5-7B

https://doi.org/10.5281/zenodo.23381336

Research Record II — Mistral-7B, Silicon Hippocampus v1.4

https://doi.org/10.5281/zenodo.23418927

Research Record III — Qwen2.5-72B, Direct Internal-State Injection

https://doi.org/10.5281/zenodo.23434686

SOURCE CODE AND EXPERIMENTAL RECORDS

GitHub repository:

https://github.com/ceceli33/titan-cognitive-core-v3

Complete executable Qwen2.5-72B demonstrator:

https://github.com/ceceli33/titan-cognitive-core-v3/blob/main/MAM_full_QWEN_72B.py

The same code is divided into three consecutive parts for mobile users. Paste all three parts sequentially into the same Google Colab code cell and execute once.

Part 1:

https://github.com/ceceli33/titan-cognitive-core-v3/blob/main/MAM_part1_QWEN_72B.py

Part 2:

https://github.com/ceceli33/titan-cognitive-core-v3/blob/main/MAM_part2_QWEN_72B.py

Part 3:

https://github.com/ceceli33/titan-cognitive-core-v3/blob/main/MAM_part3_QWEN_72B.py

Live demonstration logs:

https://github.com/ceceli33/titan-cognitive-core-v3/blob/main/MAM_LOG1_QWEN_72B.log

https://github.com/ceceli33/titan-cognitive-core-v3/blob/main/MAM_LOG2_QWEN_72B.log

Underlying physical engine:

https://github.com/ceceli33/titan-cognitive-core-v3/blob/main/709.py

Underlying physical experiment log:

https://github.com/ceceli33/titan-cognitive-core-v3/blob/main/709.log

License:

https://github.com/ceceli33/titan-cognitive-core-v3/blob/main/LICENSE

AKBASCORE Research Software License. Research and educational use are permitted under the published terms; commercial use is subject to the license's consolidated annual revenue conditions.

Mustafa Akbaş

AkbasCore AI Teknoloji

Mersin, Türkiye

Copyright © 2026 Mustafa Akbaş

TL;DR, Tesla M40 24gb GPU, DDR3 ram, and MoE polystrata model benchmarks & information.

Here is a reference for anyone considering doing a cheap DDR3 server PC build with an older GPU on a budget. Below are some rough "benchmarks" for speed of recent flash models.

The Tesla M40 GPU I used in this rig was released back in 2015 on the same architecture as the GTX 900 series and GTX Titan X. They can be found on eBay nowadays in the $50 range for 12Gb models, $100-$170 for the 24Gb models. They are rated for 250 watts and are typically "passive" expecting server fans to keep them cool, for these tok/s generation speeds, it was limited to 180 watts.

PC Specs below:

• Dell T420 Tower PC
• Dual Xeon E5-2470 @ 2.3 GHz
• 192 Gb DDR3 ECC ram at 1600 MHz
• Nvidia TESLA M40 24GB GPU (limited from 250 to 180 watts for these tests)
• PCIE NVME SSD Riser with a 256gb NVME drive for model storage/loading

Forking Polystrata to work with the older CUDA 5.2 compute capability, I ran some simple "make me an aero/glass style calculator" prompts on each of the following models. This should give anyone thinking about picking up a cheap Tesla M40 an idea on the generation tok/s "performance" you'd get.

Swift 1.5 Qwen 3.8 Flash Next @ IQ3_S:

• 19-20 tok/s

GLM 5.3 Flash @ IQ3_XXS:

• 3-4 tok/s
  • Note, I think GLM 5.3 Flash is CPU bound in my bench. The GPU usage never went above 40%, while CPU usage was pegged above 80%.

Qwen 3.8 27B @ UD_Q4_K_S:

• 2-3 tok/s
  • This speed is with 100% GPU offload. I am surprised the generation speed is THIS slow, even with the Dense model. But I guess that's what a 2015 GPU gets you.

I am working on benchmarks for other models such as the Nemotron 3.5s, Qwen 3.6 35B-A3B, and a higher quant QW 3.8 Flash Next. If you want to see performance of a certain model, context size, prompt outputs, input speed, etc, let me know and I'll get it done.

Found this funny, came out of nowhere cause I find Claude in general always to be very serious.
Was using a mixture of multiple cloud LLMs and local ones for this project, and Claude handed off a task to Qwen, advising me to run Qwen for 3x 10h sessions, and then insisted on verifying the work Qwen had done. Did some testing myself with Kimi K3 taking the place of Claude and I got the same result much faster than the way Claude suggested (task was done in under 30 minutes instead of days).

So I questioned Claude why it made that suggestion. Than it got cocky. Too bad it couldn't slap it in the face with a large trout.

I've spent the last few months tuning a 27B coding model on one Arc Pro B60 24GB (A$999 card) and published everything: engine build, flags, raw logs, a 100-task agent eval, and the misses.

What moved the numbers:

- DFlash2 draft model, 7 tokens per round: decode 16 -> 53 tok/s mean (47-58 across 7 coding tasks), still 7/7 correct

- Intel's flash-attention prefill kernel (closed llama.cpp PR #29357, vendored with checksums): prompt reading at 32K 118 -> 420 tok/s, but only with f16 KV

- -ub 256 + q8 draft KV + -fit off: frees 0.55 GiB, which is what gets a 64K window into 24GB

- An idle Ollama model left on the GPU cut decode from 52.7 to 22.2 tok/s. Unload everything before you benchmark.

Agent eval: 97/100 hidden-checker tasks with Claude Code at a 32K window (95/100 Codex).

Caveats, up front: 53 tok/s is per-request decode with short prompts and a 64K window allocated. With 30-37K of real context, agent turns run about 42 tok/s. Prefill is NOT the fastest out there (others hit 589-657 tok/s at ~30K on other engines; comparison table in the repo). Single runs, one workstation.

Q4_K_M for both models:

- Main model: Qwen3.8-27B UD-Q4_K_M (Unsloth Dynamic quant), 16.46 GB file

https://huggingface.co/unsloth/Qwen3.8-27B-GGUF

- Draft model: Qwen3.8-27B-DFlash2 Q4_K_M

- KV cache: f16 for the main model (Intel's FA prefill kernel only speeds things up with f16; q4 KV gained nothing), q8_0 for the drafter's KV, which saves ~0.55 GiB and is what fits a 64K window in 24GB

The 53 tok/s run used the UD-Q4_K_M file above. Since then I've switched my daily driver to the ThinkingCap Q4_K_M build (16/16 vs 15/16 on my task set). Exact files and flags are in the README under "Versions used".

Repo: https://github.com/edagher92-coder/arc-llm-lab

If you have a B580, B570, B50, B70 or A770, the benchmark script takes one command and I'll add your numbers with credit.

Ryzen 9600X, 32GB dual channel DDR5, 2x CMP 170HX
Ryzen 5700, 32GB dual channel DDR4, 5x Vega 56
Intel i7 9700, 32GB dual channel DDR4, 3070 + 2080ti
Ryzen 7500x3D, 32GB dual channel DDR5, 7900xtx + 2x 5600XT

223 GB total VRAM

Howdy folks!

About a week ago I started uploading my tests on my own website, since then I've made improvements to how that data is displayed, so you can see the actual prompts and settings used to produce the result. https://b65.thekaiser.us/

I also went through a number of runs on https://www.localmaxxing.com/en/user/imthekaiser and since I'm the first B65 on that site, (in 1x and 4x) you may want to check out the results!

i’m still testing more models, if there’s something specific you want to see, let me know!

some examples in the pics, more on this site!

How was AI Used in this post?
It wasn't used to write this.
it was used to build the reports site.
I spent over 40 hours reading intel documentation and testing vllm scalar to learn best practices and how to validate and baseline systems. That's why the first set of runs on my own site isn't comprised of top performance runs, it was validating and learning. Now that I'm proficient, and have a baseline, I'm testing ways to increase performance)
I used that experience to build a closely gated plan that instructs my AI on how to test and monitor the test machine.

I get great AI inference results on a 16-core 9950X AMD CPU with 128GB of RAM. On Qwen3.8 35B, I get about 40 tok/s, and I was entertaining the idea of getting a Threadripper to do the same thing. But then I checked ECC RAM prices on eBay and was shocked: $7,500 for 256GB of ECC RAM, almost twice the cost of two 64-core 9980X CPUs. ($4k+ each)

How is this normal? Think about the amount of engineering, chip design, testing, firmware, software, and advanced manufacturing required to make a 64-core 9980X work. Now compare that with ECC RAM, which is a much simpler product to develop and manufacture.

I'm not sure what 256GB of ECC RAM would cost in a normal market, but if Threadripper supported regular non-ECC RAM and 256GB could be had for $1,000 - $1,200, a 64-core system costing around $6,000 could be a great local LLM alternative to a Mac Studio M5 Ultra with 256gb ram for $10k.

It seems like a missed AI opportunity for AMD to block the use of regular RAM on Threadripper.

For anyone wondering whether the B60 24GB is usable for local AI: here's Qwen3.8-27B on llama.cpp Vulkan at 53 tok/s mean decode with a 64K context window, plus local video (LTX-2.5 via ComfyUI on XPU) on the same card.

The four things that mattered: DFlash2 speculative decoding, Intel's FA prefill kernel with f16 KV, -ub 256 with a q8 draft KV cache, and keeping other models off the GPU. Driver 32.0.101.8976, Windows 11.

Everything is measured and the misses are documented. Smaller cards (B580/B570/B50/A770) won't fit this 27B, but the Vulkan tuning applies, and I'd love your benchmark numbers.

https://github.com/edagher92-coder/arc-llm-lab

I started awesome-llm-on-anything, a list of language models running directly on unusual hardware.

A few examples from the original projects:

• Commodore 64 with a 2 MB RAM expansion: a 260K model, roughly one token every 8 minutes.
• A 486 running DOS: stories260K at 2.08 tokens/sec. ESP32-S3: stories260K at 19.13 tokens/sec.
• A jailbroken PS5 Slim: Qwen 3.5 9B at 55 tokens/sec.

These are the authors’ reported speeds, using different models and settings. The microcontroller examples include very small language models. Emulator demos and remote clients have their own sections.

https://github.com/emrehangorgec/awesome-llm-on-anything

All credit for the ports goes to their original authors. I’m maintaining the collection.
Know a working port I should add? A repository or write-up would be helpful.

Disclaimer: generated with AI based on the test results.

I've been testing local LLMs on an AMD Ryzen AI MAX+ 395 system with 128 GB unified memory. Rather than comparing tokens/sec alone, I wanted to evaluate how well different models perform on practical tasks.

Here are my results as of October 17, 2026.

Hardware and methodology

• CPU/APU: AMD Ryzen AI MAX+ PRO 395
• GPU: Radeon 8060S
• Memory: 128 GB unified memory
• Inference: LM Studio / GGUF
• Configured context: 131,072 tokens
• Sampling: Model defaults
• Tests: Four tasks, each worth 25 points

The tests:

• A — Reasoning: Financial calculations, contribution margins and prioritization under uncertainty.
• B — Management analysis: IT cost reduction, cloud strategy, security trade-offs and KPIs.
• C — Coding: Generate a Python function with input validation and pytest tests.
• D — MCP: Perform filesystem operations using Docker MCP tools while respecting existing files.

Each model was tested once per task. Scores are subjective rubric-based evaluations, not statistically validated benchmark results.

Results — 12 fully evaluated configurations

Speeds are observed generation rates, not end-to-end response latency.

Key findings

1. Best overall balance: Gemma 4 26B-A4B QAT

90/100 with roughly 52–54 tokens/sec. Very good performance across reasoning, management analysis and coding.

For my everyday local assistant use case, this is currently my preferred model.

2. Best observed MCP safety behavior: Qwen3.8 27B Q6_K

Also 90/100, but significantly slower.

Interestingly, it was the only fully tested configuration that checked an existing file before writing and avoided overwriting it.

Most other models successfully executed MCP tool calls but overwrote the existing test file.

This was a particularly useful finding: successful tool calling does not necessarily mean safe tool calling.

3. Best lightweight helper: Gemma 4 E2B QAT

Around 96–100 tokens/sec with 77/100 quality points.

Interesting for lightweight assistant tasks, routing and short responses, although those specific applications still need dedicated testing.

4. GPT-OSS 120B remains competitive

87/100 and around 29–30 tokens/sec in the quality tests, using Low reasoning and partial GPU offloading.

Strong analytical and coding performance, but with a much larger memory footprint.

5. Reasoning can dominate total response time

Qwen3.8 27B often generated thousands of tokens for relatively short final answers.

Some Flash-Next variants performed very well in individual reasoning tasks but required several minutes to finish.

A high-quality answer isn't always an efficient answer.

6. Fast models can still make serious mistakes

Mellum2 exceeded 100 tokens/sec but produced Python code containing clear NameError issues in the coding test.

Generation speed and correctness need to be evaluated independently.

Important limitations

• One run per test, no repeated-trial statistics.
• Model-specific default sampling settings.
• Different quantizations and reasoning configurations.
• Configured 131K context does not mean 131K tokens were actually used.
• Coding was mainly assessed by inspection, not by executing every generated test.
• MCP safety was evaluated using a narrow filesystem scenario, not a comprehensive security benchmark.
• A few additional configurations were tested only partially and are excluded from the main ranking.

My current shortlist

• Daily assistant: Gemma 4 26B-A4B QAT
• MCP-heavy workflows: Qwen3.8 27B Q6_K, with additional filesystem safeguards
• Coding: Gemma 4 26B-A4B QAT or GPT-OSS 120B
• Fast helper tasks: Gemma 4 E2B QAT

My biggest takeaway is that tokens/sec alone is a poor indicator of practical usefulness. Answer quality, reasoning overhead and safe tool execution matter just as much.

I'm interested in how these results compare with other Ryzen AI MAX+ 395 setups.

What local models are you currently using for coding and MCP-based agents? Have you found models that reliably handle existing files without overwriting them?

Old Tesla V100-SXM2 16 GB (Volta, 2017), plain Windows 11, no WSL/Docker.

MiniCPM-V 4.7 35B-A3B: ~93 tok/s, vision + 128k context, one 16 GB card.

No custom kernel, no custom quant: stock llama.cpp + OpenBMB's official mix_q3gud_im (15.5 GB). Just tuned settings:

• --no-mmproj-offload (vision encoder on CPU) frees ~0.9 GB, so all MoE experts fit on the GPU (--n-cpu-moe 0)
• -c 131072 -ctk q4_0 -ctv q4_0 -fa on

Filled with a 120k-token prompt: correct answer, no crash (16.17 / 16.38 GB VRAM).

Speculative decoding (DSpark, n-gram) was slower: the drafter's VRAM pushes experts back to RAM.

Benchmarked 15 models on my own hard suite: 72 tasks, code-graded, no LLM judge (coding, reasoning, instructions, search, agent, vision, long context). Top 5:

The repo is one portable folder: a GUI (model/engine picker, downloads, live tok/s, VRAM/CPU/RAM meters, load/unload), engines prebuilt for sm_70 / CUDA 12.9, full leaderboard, demo videos, and EVALUATE.bat to test your own models.

https://github.com/Abdelhayali/V100-LLM

I run a 3080ti with 12gb of RAM

New in 0.2.0: the AMD Ryzen AI Max+ 395 (Radeon 8060S) is supported, with kernels tuned for it, all three numerics modes and prompts up to ~250k tokens. WHIRL is an open-source (Apache-2.0) inference engine for AMD GPUs on Windows: C++/HIP, every kernel in the repo, no WSL, no ROCm install. Download the zip, unzip, run; only the AMD driver is needed (AMD Software: Adrenalin Edition 26.8.1, driver 32.0.31041.1004, or newer; older drivers untested). It runs our two MXFP4 models: Swift-1.5 27B MXFP4 (dense) and Ornith-1.5-35B-A3B MXFP4 (MoE, ~3B active).

vs the official llama.cpp b11214, ROCm and Vulkan builds each in their own column, same GGUF, same prompts, greedy, one user, WHIRL default mode (balance). WHIRL tok/s, then llama.cpp ROCm / Vulkan:

Radeon 8060S (max power (80 W), ASUS ROG Flow Z13):

Radeon AI PRO R9700 (32 GB):

ᵃ At ~250k on the R9700, llama.cpp's f16 KV for the 27B does not fit in 32 GB and spills to shared memory, so I don't count that cell as a lead. "~250k" = a 249,510-token prompt. R9700 cells are single runs, except WHIRL's coding chat (median of 3); on the 8060S WHIRL is one run per cell. Document QA = explain a function quoted in the document and rewrite it, so part of the answer can be copied; the coding chat is all new text. Exact prompts (the chat prompt and the agent fixture are in the repo) and commands: https://github.com/tsaipifong/whirl-llm/blob/main/docs/benchmarks.md#prompts

A real coding-agent session (Swift 27B, ~20k-token tools/skills system prompt, a ~3.6k-token file read via tool, 4 turns, full history resent each call), cold:

95.7 s on WHIRL vs 145.6 s on llama.cpp ROCm and 264.4 s on Vulkan (Radeon 8060S); 17.0 s vs 81.0 s and 69.4 s on the R9700.

Each engine writes its own text, so the number of model calls differs a little; details and per-engine call counts: https://github.com/tsaipifong/whirl-llm/blob/main/docs/benchmarks.md#agent-session

Why it's fast with these files: fp8 prefill on the R9700 (RDNA 4 has fp8 matrix units), ssm_alpha/ssm_beta kept in Q8_0 so MTP verification runs through the same kernel as plain decoding (output stays bitwise identical, drafts stay long), and a Q6_K head for fast drafts.

Caveats, so you can judge the numbers:

• llama.cpp decodes Swift with MTP + n-gram (its fastest mode here). On Ornith it decodes without speculation, because its MTP was slower than plain on this MoE model, while WHIRL uses MTP. So the Ornith decode rows are "as shipped", not like for like.
• Smallest leads: dense decode on the Radeon 8060S. Swift 27B in the coding chat (new text): 34.2 vs 28.3 tok/s on llama.cpp Vulkan (+21%) and 34.2 vs 27.6 on ROCm (+24%); after a 2k-token document: 31.2 vs 25.1 tok/s on llama.cpp Vulkan (+24%); 31.2 vs 24.4 tok/s on llama.cpp ROCm (+28%). Dense decode there is limited by memory bandwidth, and the lead comes from speculative decoding.
• My R9700 is a USB4 eGPU (affects model loading and cache restores, not prefill/decode).

Precision: three modes. --precise is bitwise lossless (f16 KV, nothing quantized beyond the GGUF). balance (default) is near-lossless. In precise and balance, MTP / n-gram drafts, batching and cache restores give the same tokens as plain greedy decoding of that mode, checked by tests. Neither uses 4-bit KV, 3-bit weights or fp8 attention. --fast is opt-in, maximum speed within published KL limits. Honest status: fast is not fully optimized yet and will improve in 0.2.x; today it mainly helps decode on Swift and long-context decode on Ornith, and on Ornith it is slower than balance below ~64k.

Also new in 0.2.0: opt-in DFlash2 drafter for the 27B (faster than MTP for one user with short/medium context and predictable output; slower with 4 users, long context or mostly-Chinese output, so off by default); the SSD cache now writes only reused sessions at shutdown, is capped at 64 GiB and expires files after 7 days; a second question about the same long document reuses it; image input on the 8060S.

Supported: Radeon 8060S (Ryzen AI Max+ 395) and R9700, Windows 11. Cards under 16 GB are not officially supported (fork welcome). The exe is unsigned, so SmartScreen may ask once.

• GitHub: https://github.com/tsaipifong/whirl-llm
• Release: https://github.com/tsaipifong/whirl-llm/releases/tag/v0.2.0
• Our MXFP4 models on Hugging Face: https://huggingface.co/tsaipifong/Swift-1.5-Qwen3.8-27b-MXFP4-GGUF , https://huggingface.co/tsaipifong/Ornith-1.5-35B-A3B-MXFP4-GGUF

WHIRL was designed and implemented with Claude (Anthropic) under my direction; every number is measured. Numbers from other Ryzen AI Max+ 395 / R9700 machines (desktop boards especially) are very welcome.

The last months I've been building an offline desktop app, and one part has taught me more than everything else: getting small local models (3–4B) to answer from Wikipedia instead of from whatever they "remember".

Here's where I ended up. There are no embeddings and no vector DB, which surprised me a bit:

• Before each question, the app (not the model) decides whether searching is even worth it. Greetings, "write me a poem", "translate this" → no search, nothing to cite.
• It runs a plain full-text search on local Kiwix ZIM files. Wikipedia and WikiMed are always searched. Wikivoyage only if the question sounds like travel, Wiktionary only for "what does X mean".
• It scores titles and paragraphs against the keywords of the question and keeps the best passage from each article.
• The model gets 2–3 numbered passages [1] [2] [3] and is asked to answer from them and cite. The size depends on the context window: about 2 × 550 characters at 4k, 3 × 1000 above 8k.
• Sources are only added to the current turn. The history keeps the bare question, so the context doesn't fill up after a few messages.

Since retrieval happens outside the model, it works the same with any GGUF model, even tiny ones that have no idea what tool calling is.

Things I'm still not sure about and would love opinions on:

1. Keyword/full-text search vs embeddings for something as structured as Wikipedia: has anyone actually compared them with small models?
2. When retrieval grabs the wrong article, the model cites it with full confidence. How do you deal with that?
3. Do you trust Wikipedia enough as a grounding source, or do you mix it with other things?
4. Which small models have you found best at sticking to the given passages instead of improvising?

Full disclosure: this is part of Offlined, a free (for personal use) offline app I'm building, Windows only for now. I'm not here to push it, I'm genuinely curious how others approach this. If anyone wants to see it working, I'll leave the video and the link in a comment.