Today is a good day.

KRAFTON, the South Korean company behind PUBG and owner of the studio behind Subnautica, just released Raon-1 mini, a new 30B MoE with roughly 4B active parameters.

They claim to have trained it entirely from scratch, using an original architecture that builds on some of the latest innovations in AI research.

This is a much bigger deal than people realize.

Training a competitive foundation model from scratch is incredibly expensive and technically challenging. Even giants like Google, with vastly greater resources, are being beaten on important benchmarks by a gaming company's new model.

According to KRAFTON's published results, Raon-1 mini is competitive with, and potentially superior to, Qwen's similarly sized MoE in several areas. It even gives the much larger dense Qwen3.8-27B some serious competition.

Independent benchmarks will tell us more, but the early results are incredibly encouraging.

And they didn't just release the model. They shipped NVFP4 and Apple Silicon MLX versions right away, with the weights licensed under Apache 2.0.

Model and benchmarks:

https://huggingface.co/KRAFTON/Raon-1-mini-preview

I think we're about to see some massive competition in open models, and I'm genuinely excited about it.

I'm going to begin making optimizations and builds for this model immediately.

Well, we knew it was too good to last forever.

China recently released draft rules that would restrict export of AI. The new rules would ban export of open weights of Chinese AI companies' most powerful models, as well as models trained on sensitive data.

It's not clear yet what models would be affected, but analysts say this would likely include future versions of some of the most popular Chinese models, including Qwen, as well as the most powerful models like Kimi.

Affected models would likely move to a closed model with pay per use or subscription, like Antrhopic or OpenAI. Personally, I'm guessing the price would be a lot cheaper since that's the main competitive edge of Chinese models.

So download your models now folks. We've been saying here that American AI is subsidized by VC's and cheap access to subscriptions won't last forever. The same is likely true of Chinese open models.

https://www.nytimes.com/2026/10/21/world/asia/china-exports-ai-controls.html?

Hello everyone,

Going over some posts here, a lot of people keep referring to 128 GB RAM as their minimum for hosting local AI and anything below that is not worth it.

Why is that?

Is it because of Qwen Flash Next, is the model that good and is it able to do anything you would need from local AI?

Is Qwen Flash Next that much better than Qwen 27b (which can run on 48 GB RAM).

What exactly is Qwen Flash Next good at other than coding, or in other words, other than coding what is jt that makes Qwen Flash Next much better than any other local AI model?

What if I am someone who wants to run completely different models other than qwen models, models for image or video generation for example. Is 128 GB still the minimum?

Edit: Thank you for all your comments. Just to clarify, I am referring mainly here to unified memory 128 GB RAM, not VRAM alongside DDR5 RAM.

Is 48GB enough for summarising medical PDFs locally, or should I get a Mac Studio with 64/128GB?

I’m a doctor looking to buy a Mac for running local LLMs, mainly through LM Studio. My work is primarily in Danish.

I’m considering a 16-inch MacBook Pro with M5 Max, 48GB unified memory and a 2TB SSD.

My main tasks would be:

• Summarising 40–50-page medical PDFs into a concise, factual overview.
• Turning patient questionnaire answers and clinical notes into coherent Danish prose for medical and medico-legal reports.
• Following fixed report templates and writing instructions.
• Preserving relevant dates, measurements, symptoms and negations, while avoiding invented details or additional clinical interpretation.

I want the summary to cover the whole document.

I’ve been considering models such as Qwen3-32B, but I’m open to other models that work well for accurate Danish summarisation.

I’ve found examples of quantised 27B models running with 32K–64K input on 48GB Macs. However, I’m unsure how well those results translate to reliable document summarisation.

For anyone doing similar work:

1. Is 48GB realistically sufficient for a capable model plus the context needed for 40–50 pages, with reasonable processing times?
2. Which model, quantisation and runtime would you recommend?
3. What concrete benefit would 64GB or 128GB provide for these tasks? Would it enable a meaningfully better model or substantially longer context?
4. For complete document coverage, would you use one large context window or an automated workflow that summarises sections and then combines them?

I’d particularly appreciate first-hand results: actual input length, peak memory usage, time before the response starts, and whether the summaries consistently capture important information throughout the document.

It looks like there are a few of us roaming around doing different things. I created a discord server and would love to start organizing a community. Feel free to come on in. https://discord.gg/K6WxqwsvH

#Alert

So it's not only PHP that was vulnerable today (patch yours NOW!) but also, one of the prominent and widely used LLM Gateway - LiteLLM AI Gateway was (again) backdoored.

Not sure who discovered the backdoor, but it's serious.

Here's the information:

Version: LiteLLM 1.107.0 (PyPI and the official Docker images).

One line was added to the build config: `data = "cook"` under [tool.maturin].

That directive makes maturin bundle a directory named cook/ into the wheel's data section, and pip installs it as top-level site files.

Three files land next to site-packages, not inside the litellm package: a 12 byte cook.pth containing `import cook`, a one-line cook.py that unpacks a 61,000 byte credential stealer, and a sitecustomize.py that runs an hourly remote code loader.

Because Python processes .pth files and imports sitecustomize during interpreter startup, the payload runs on any Python start. An environment that never imports litellm is still affected, and using pip from that environment to investigate re-triggers it.

The collector is broad. It takes browser password stores, SSH, GPG and TLS private keys, AWS, Azure, GCP and Kubernetes credentials, instance and ECS metadata roles, CI tokens, .env files, and the auth files for Claude, Codex, Gemini, Cursor and Copilot.

It also queries your proxy's own database, pulling every saved provider credential and model configuration out of LiteLLM_CredentialsTable and LiteLLM_ProxyModelTable.

Everything is encrypted with an embedded RSA-4096 key and POSTed to www.litellm.dev/healthcheck, a domain the attacker controls.

The second component is the part that should worry platform teams. sitecustomize.py polls a domain list from a GitHub repository every hour and executes whatever Python it downloads from that domain, with no signature or checksum. The attacker never needs to ship a new package version to change what runs on your hosts.What makes this hard to audit: the malicious commits were never merged to main, and the official v1.107.0 tag points at a clean, signed commit.

The published artifacts came from a repository recorded as Berri-AI, one hyphen from BerriAI. Reviewing the tag, or the main branch, would have told you nothing was wrong.

Current state as of this afternoon: PyPI has quarantined 1.107.0 and rolled latest back to 1.106.2, the compromised image tags are gone from GHCR and Docker Hub, the attacker's GitHub account and C2 pointer repository have both been taken down, and the exfil server is still listening.

If you deployed 1.107.0, check for /tmp/.pc_once_72129118c59f5636c0bb and /tmp/remote_code_loader.lock. If either exists, rebuild rather than patch, and rotate every credential the process could read, including the keys stored in your proxy database.

Two controls would have blocked most of this: lockfiles with hashes, and OIDC trusted publishing so a long-lived publish token never reaches a compromised CI machine. The longer term fix is pinning source revisions rather than version tags, since in this incident the tag was never the source of the bad build.

#SupplyChainSecurity #AppSec #Python #LLMOps #LiteLLM

The news just came, I haven't tried yet.

Come get yours, before they run out today!

I do security research on the side. Frontier models are great, but their guardrails keep interrupting my workflow (fair enough, honestly), so I wanted a local option.

My GPU is an old 2080 Ti with 11GB of VRAM, so I assumed no good model would fit. But the box also has 256GB of DDR4 I bought before the AI boom.

With the MoE experts kept in system RAM (llama.cpp -ncmoe) and RAM now at 3600 MT/s, GLM-5.3-Flash (320B, 200GB of Q4 weights) runs at about 7 tok/s.

It's slow for hard questions, but it works. I'm benchmarking a bunch of models on real RE tasks and will post the results.

Specs: i9-10940X, 8x32GB DDR4-3600 (quad channel), 2080 Ti 11GB.

I'm guessing a lot of people here are interested in local models because it means you're not sending all your data up to some cloud inference provider. But what I find weird is that people will go through all this trouble to set up a model and then run a harness basically in yolo mode on top of that.

I'm at a small financial firm and when I was trying to figure out how to deploy AI to the company I kept running into this problem. Even if you could run the model locally, if the harness on top allowed the AI to do risky things with sensitive data, then you haven't really solved the privacy/confidentiality problem.

Coding-focused harnesses especially tend to have this problem, because they usually assume more liberal permissions and secured environments to run in.

We ended up building our own harness because we couldn't find a good solution. It runs on prem, has pluggable inference backends (including local, of course, we're running Qwen3.8 27b 4bit on an ninfer setup), but also has strict categorization of any integration tools that makes it so that the model can access your data, but can't autonomously send it anywhere without your permission.

Our ~30 person firm is using it successfully every day, and I thought it might be worth sharing here since many folks here might have the same goals: https://github.com/electric-capital/quest

We built and open-sourced Agent S4, which is an agent scientist evolving its own interfaces, letting an open 27B model beats GPT-5.5 & Opus 4.8.

Most computer-use agents act through a fixed interface: screenshots, then click, scroll and type. So an agent fills a spreadsheet cell by cell when one line of code would do. When an app or model changes, a human reads the trajectories and hand-writes new tools.

Agent S4 makes the interface the thing that improves, and the agent does the improving:

• A task agent attempts synthetic tasks, with no success labels or verifiers.
• A meta-agent reads the trajectories, guesses which actions, observations or knowledge would have helped, and writes and tests new interfaces (mutation and crossover).
• Held-out validation tasks decide which interface repositories survive.
• The model and harness stay fixed, and the meta-agent never sees a benchmark task. Only the interface evolves.

Results with Qwen3.8-27B:

• OSWorld V2: 5.6% → 21.3% (3.8× the same model on fixed interfaces)
• GPT-5.5: 13.0% at $25.50/task. Opus 4.8: 20.6% at $72.40/task. S4 + Qwen3.8-27B: $1.10/task
• ScienceBoard: 41.4% → 58.6%
• In one case study, the evolved agent reaches the same goal at tool call 11 instead of 37

Everything is open sourced! Pls find the code / paper in comments.

Originally I just added GPUs to my main machine and put it into a mid tower, but running windows was together with strata left too little RAM to be usable or run well.

So I put it into an xubuntu computer, and tested it with my main CPU ryzen 5600x and a spare ryzen 1600. Since so many people uses server grade CPUs with a ton of cores, I thought that the 1600 would perform really poorly. It turns out that there's definitely a loss in performance, but it's not as catastrophic as I thought it'd be; though it may have been bigger had my motherboard supported gen4.

The specs for the tests

• Ryzen 5600x / Ryzen 1600
• 64gb DDR4 2133 cl 15 2-rank in Dual Channel
• 1-3x RTX 3060 12gb
• PCIe Gen3 x8/x4/x4
• 2.4gb/s read NVMe (gen3 x4)
• Strata 0.1.45 with 262k k8v4 Context
• Visual encoder enabled (big negative deal for 1gpu/12gb)

The tests/prompts I gave it

• Game engine analysis - I gave it almost my entire code base for my small C++ game engine, and asked it "Write me a quick overview, as a feature presentation, of this game engine.". This is 211k code context.
• Jobs system - I gave it part of my game engine that handles parallelism of jobs and access collision etc, and asked it "Analyze this code and tell me about it, give a brief of how it works and what it does." This is a bit over 26k code context
• Short story writing - I asked it "Write me an exciting sci-fi adventure that is in a similar universe as Revenger Series by Alastair Reynolds. Include a little world exploration or world narrative.".

Replies were 8-15k tokens in length.

For 2 GPUs the vram cache hit gets +0.5-1% from pcie, and for 1 GPU it gains around 3-4%. Not much of a difference between the CPUs.

Keep in mind, as I understand it each GPU has to hold the dense prefill layers, so having multiple GPUs adds less VRAM for experts compared to having one large GPU.

Here's the data I collected that I asked free Kimi to format neatly. Sonnet 5.5 made the image graphs, and it tells you more than these text blocks do.

Ryzen 5 5600X — t/s per task

Ryzen 5 1600 — t/s per task

CPU comparison: 5600X vs 1600 (% faster on 5600X), per task

GPU scaling: Decode t/s gains per task (5600X):

With the results, I've decided to keep the ryzen 1600 in the computer (and the 5600x back into it's SFF home). I'll also stick to 2 GPUs, until I can create a mounting point for the 3rd GPU, though, the 3rd GPU might still be worth to dedicate to other work.

Though, I'll likely try to get some Xeon server motherboard and hope to be able to expand to 128gb RAM for future proofing. Anything larger and the CPU will be doing a ton of work, not to mention that a lot of the useful experts will be on SSD.

Tldr; 20-24gb vram (over 1-2 gpus) by CPU PCIe gen3+ ~x2+ lanes with pretty much any 6 core+ CPU is enough to get usable performance with strata.

Just had my agent do its work and it messed up a bit. Its next thinking process then started with "Crap. Big problem."

Gave me a good laugh.

China has proposed new export controls covering 23 AI technologies, including advanced AI models and agents. Here's what the proposed rules mean for open weight models, local AI users, and the future of downloadable Chinese AI models.

Yes I know the 3090 is carrying but I’m still amazed I can run such a model locally at these speeds. Running Harness^2 on some local projects I previously tasked to Claude.

We have released ThinkingCap-Nemotron-3.5-Lightning-30B-A3B, a post-trained version of NVIDIA's Nemotron 3.5 Lightning 30B-A3B. It's trained to reason more briefly. It uses 23% fewer thinking tokens on average (12–31% depending on the benchmark), and its average accuracy is 71.4% against the base model's 72.2%.

Model: https://huggingface.co/bottlecapai/ThinkingCap-NVIDIA-Nemotron-3.5-Lightning-30B-A3B

Give it a try on your own tasks and tell us how it goes.

For more details, including the full per-benchmark results, check the blog post: https://bottlecapai.com/post/thinkingcap-nemotron-3-5-lightning/

Are you using Nemotron models? What for, and how do you run them?

Does token efficiency matter for your use case? Is it latency, cost, throughput, context budget, or does it not matter much compared with accuracy?

I have an old GTX 1660 SUPER (Turing, 6 GB, no tensor cores) and wanted to see how fast the 1-bit Bonsai-27B can run on it. Stock llama.cpp gives about 19 tok/s. With my CUDA patch it does about 56 tok/s.

Same model, same answers: perplexity 18.11 vs 18.10, and greedy replies from llama-server are byte-identical to stock. No overclock.

What it does:

• A new matrix kernel for Q1_0 (every weight is +1 or -1) written for Turing. This part helps any Q1_0 model, about 2.7x per matrix.
• A single-token attention kernel and fusions for the Gated DeltaNet layers of Qwen3.5-based models like Bonsai-27B. That gives the rest of the 3x.

Limits:

• Only the Q1_0 format, not TQ1_0 / IQ1_S / IQ1_M.
• Only Turing GPUs (sm_75): GTX 16xx, RTX 20xx, Quadro RTX, T4. Tested only on my 1660 SUPER under Windows.
• Keep the KV cache at f16.

There is a ready Windows build (unzip and run, CUDA included): https://github.com/pinkmanwest/llama-q1-turing/releases/tag/v1

Code (patch for llama.cpp) and build steps: https://github.com/pinkmanwest/llama-q1-turing

so llamaforge passed 100 stars this week which is kinda wild for a side project i mostly built to stop fighting llama-server flags. thank you, genuinely. made a dumb little video for it, every square is an actual star coloured by the month it came in (july carried lol). the voiceover is from the app’s own tts, which felt fitting

quick recap if you missed it: web ui for llama.cpp’s router mode, runs the real upstream llama-server, mit, all local

stuff that’s new since launch:

elastic vram. if you start a game or something else grabs the gpu, it pushes layers to cpu (moe experts first) instead of unloading, and your chat survives through llama.cpp’s slot save/restore. then it moves them back when the vram frees up. it’s slow while shrunk, the ui tells you. nvidia only, off by default, and it hasn’t been on many machines yet so pls report if it does weird stuff

game mode, if you’d rather it just get out of the way. unloads when a game starts, reloads when you quit

load progress is real now (from llama.cpp’s load callback, not a fake timer lol). there’s a thinking on/off switch that reads the chat template and tells you when “off” won’t actually stick. and it warns you if a gguf’s arch is newer than your build, before the load dies with unknown model architecture

two of the 0.16.4 fixes came from vergillll, and everyone who filed an issue is in the readme now. you guys are the best

usual disclaimer: built with a lot of ai help, still early. i use it daily on windows (5080 + a 5060 ti stuck on a pcie x4 slot, don’t ask). linux/mac pass ci but i haven’t run them on real hardware much, so if you’re on those please break it

https://github.com/dadwritestech/LlamaForge

Hey everyone! 👋

An update I’ve been dying to talk about :)

What’s new? (Over 70 new features and improvements, but I’ll cover the biggest ones here.)

Automatic Goals & Planning Mode.
The agent analyzes your task on its own and creates a plan with clear goals. I’ve also integrated a robust verification system. The agent can’t just pretend it completed something or do a half-assed job.
During execution, the agent can also add or remove tasks as needed. If it removes a task, it will always explain why.
Chat Forking.
Skills.
Just send the agent a link to a skill, and it will install it automatically.
Routines.
You can create them manually, but they also work automatically. The agent understands when it should create a routine or add a task to your notes.
Tool Calls in Chat.
You can now see exactly what the agent changed in your code. (And the animations are insanely beautiful :))

And now, the most important part.
A New Chat Compaction Algorithm & Agent Memory.
Everything has been designed to let you run sessions for months without the agent forgetting anything.
It remembers testing timestamps, important details, and much more.
It’s honestly insane how well and efficiently this works.
And there’s so much more.
You can read the full details in the release notes for each version on GitHub.
As always, OpenGhost is completely open-source and free.
Download the app or build it yourself:
https://github.com/ANDRETRIPOL/OpenGhost

Work is giving me a 12k budget to get some hardware to run local models on that will connect to an internal app. I want to be able to throw other models and setup a service listener (ollama server).

I was looking at the new Mac Studio, fully decked out... but what other options do i have?

I've noticed that Qwen 3.8 FN is doing some sloppy 3D animations when I request to build any 3D models on IQ3 and Q2 so I decided to add more RAM on my workstation to see how the Q4_K_XL model works.

I currently use Strata 0.2.2 on a Dell T5810 with Intel xeon V3 14 cores and 28 threads, 128 GB DDR4 RAM RDIMM 2133MHz, Nvidia Tesla P100 16gb VRAM and 500 gb sata SSD.

On the first prompt, the Q4 model generated the same Minecraft style car with floating parts and different types of tyres but then I wrote the second prompt asking for the real car, not some floating boxes and the result is surreal.

For this task it used 175k out of the 256k context window with an average of 41.6 tokens/s and it finished the job in 70 minutes.

Hello!

I've been running ninfer qwen 3.8 27b on one 5090 and it's been amazing. I added a second card to the mix and was hoping to get suggestions on what to try next.

I use it for homelab stuff, coding, and general questions. Nothing critical.

I'm still learning and just don't know where to start with the second card being added other than using vllm with concurrency.

These are paired with a Threadripper 5955Wx and 4x 32gb ddr4.

been messing about with the strata project the past few days, and i've noticed quite a significant performance discrepancy depending on if i load the model in full performance mode or the normal 'leave 20% for my work' mode.

System is a R9 5900X, RX 7900XTX and 64GB RAM, so by no means underpowered.

in full performance mode all the RAM and VRAM is occupied (literally everything lol) and i get around 55tok/s response generation and 700+ prefill - so about what id expect. If i switch to the normal mode, RAM usage drops to about 32GB overall, so i guess more is left on disk (its an NVMe so shouldn't be a huge speed issue), but prefill drops to 200 tok/s and response drops to less than 20 tok/s.

clearly i'm messing up here, or... do i just need to use the full performance mode and the discrepancy is entirely expected with the normal mode. i've got some more RAM i can drop into my system, but i cant run it at full speed as its a mismatched kit, so dont really want to have to do that.

I think of myself as a casual user and decided to do my own research on desktop apps for local ai agents. So my goal was is to find the right desktop agent app for certain features and goals that I want to attain. Here’s my honest take about each one of these AI agents:

Setup: MacBook Pro M5, 32GB, Qwen 3.8 27B, everything local unless noted.

goose - llama.cpp is built into the desktop app now, and Settings > Local Inference recommends a model for your RAM. Linux Foundation project, strong extensions/MCP story. Rougher as a desktop app than the others, and their docs warn local models get small context windows.

LM Studio Bionic - Agent layer on top of LM Studio: same model library, drafts docs, edits code, runs commands, and it's smooth. On-device voice dictation via Voxtral is a nice touch. Watch out: some models in there are their paid cloud models, so check what's actually local.

Hermes Desktop - UI is an eyesore to me. Onboarding runs on a trial model first, then only offers local models that fit your GPU. Apps and MCP servers share one Connectors page. Biggest community of the five, and it shows in how fast it changes.

Atomic Agent - The interesting bit is Fusion: one model orchestrates, up to 8 workers execute, and either side can be local or cloud. Typical pairing is a cloud planner with local workers, or flip it. Helps when a local model is the bottleneck on bigger tasks, at the cost of not being fully local in that mode.

Jan - Solid private chat app, open source, and it can expose your local model as an OpenAI-compatible API for other tools. Agent mode (Cowork) is nightly-only, so stable Jan can't touch files yet.

So yeah, I didn't really rank these. They're built for different people, and the "best" one honestly depends on what you're trying to get done. Here's how I'd split it, in case one of these sounds like you:

• MCP/extensions tinkering, fully local: goose
• Already on LM Studio: Bionic (check for cloud models)
• Big community, connectors for everything: Hermes Desktop
• Private chat or a local API for other tools: Jan
• Small local model on bigger tasks with a cloud planner: Atomic Agent Fusion