Details are developing…

so we moved our support bot onto our own self-hosted model earlier this year so customer data stays in house. It answers from our help docs with RAG and for months it was fine

and just ast week a customer sent a screenshot where the bot told him accidental damage was covered under warranty and we’d repair it for free. It isn’t and we don’t. The doc it pulled was about the extended plan, which he doesn’t have, and the bot blended the two we had to honor it cause arguing with a screenshot of your own bot is a losing game, air Canada literally lost a tribunal case over its chatbot giving misleading info about bereavement fares in 2024.haunting part is nothing in our logs looked obviously wrong. The answer was fluent and on topic, it was just built on the wrong chunk

and from reading around, catching this before the customer does seems to split into two jobs one is to logging the retrieved chunks next to every answer so you can trace where things went wrong, and tools like Langfuse or Arize Phoenix help with that. The other is a check on the live answer before it gets sent, blocking anything about coverage or refunds that the retrieved evidence doesn’t actually back up

then second part we’re looking at Orq AI, which runs evaluators as guardrails that block a response when a check fails, and has an on-prem install so nothing leaves our setup. The alternative is Nemo Guardrails and writing more of the checks ourselves

Still not sure whether a guardrail on the way out catches more than someone reading transcripts every morning, especially when the bot mixes up two policies that are both technically in the docs

I haven't tried most of the models I could run so I don't have much to go on but there's one Mistral 7b finetune I still load up and will continue to download to new systems, at this point the capabilities and knowledge of it has been eclipsed by smaller models with more features but this one finetune offers a nice structure and creativity that has earned its keep.

Are there some older models or finetunes of those that anyone still uses over newer models?

The Mistral 7b finetune is Hermes Trismegistos. Whatever occult finetune data made it creative or just different in many topics and then it was finetuned to be verbose. Creative and detailed is when I load it up.

Hey everyone,

Releasing v1 of XT-Luna 1B

a fine-tuned MiniCPM5 1B focused on agentic and coding workloads.

Tested against the official MiniCPM5 1B SFT and Qwen 3.5 4B reasoning benchmarks and comparison screenshots are in the model card.

Still actively working on it, so feedback and bug reports are very welcome. that's partly why I'm releasing early.

Recommend running with thinking mode enabled.

https://huggingface.co/XtrCom/XT-Luna-1B

I am building my first local LLM station and considering to start by picking a good graphic card. The V100 is significantly cheaper, but also apparently no longer supported for future CUDA releases.
How big of a problem is that considering the current trend of continuous optimizations and findings such as n-grams for MoE, or diffusion models and so on? Still learning here, and genuinely overwhelmed by the pace I have to keep to have even a grasp of what’s going on out there.

Thank you in advance!

I only found one sloppy-ass slop channel that published see pic attached. Obviously they celebrate its cost efficiency and their benchmark is some nonsense but jesus fucking christ that's a big indication it's a tokenslopper

Is there a better token efficiency benchmark available yet? I'm not sure AA will retest this one (I sure hope so though)

Mainly I care about whether it's better than previous ckpt, as I know already the previous one was so bad that GLM 5.3 Flash was a better option for me

EDIT:

For anyone that's interested, I found more numbers but just nothing quite useful. I went ahead and ran my classic set of tests, not all of them, but my preliminary conclusion is:
- on max thinking effort, there is no difference in CoT length
- on high effort, CoT length is significantly shorter, but not by much
- regardless of thinking effort, the CoT quality is much higher, uses less 'fluff', and the end result is that it takes less turns to solve the task

So it's worth trying!

Hello guys, hoping you're doing fine.

Today I want to share something that may affect your cards (?

I have released 3 tools:

1. A way to modify VBIOSes, so you can increase power limit, voltage, NVVDD offsets, MSVDD offsets (Blackwell 2.0), OCPs, fan channels, power per fan, core offsets more than 1000, VRAM offsets more than 3000, lower power limits (below 400W on a RTX 5090), lower clocks, etc. But as you know, these can't be flashed directly on the cards because falcon, right?
2. A custom VBIOS loader on Windows, that patches the registry file related to that GPU, it resets it and it's applied on every reboot. This does not change the real VBIOS on your card, but it just loads a copy via a patch and it let you use any modification!
3. A custom VBIOS loader on Fedora, that patches the GSP firmware start on memory, builds a separate boot entry and then rebuilds automatically with kernel updates. This does not change the real VBIOS either.

What are the requirements?

• Windows:
  • Windows 10 or 11, 64-bit.
  • NVIDIA driver 616.56 or newer (status shows the version).
  • A payload file for that exact GPU model (same PCI device ID).
• Linux:
  • A working NVIDIA 615.x driver or newer with the open kernel modules; nvidia-smi must list your GPUs.
  • The unpatched source of the open kernel modules for the exact installed version, for example: git clone --depth 1 --branch 615.78.08 https://github.com/NVIDIA/open-gpu-kernel-modules
  • Python 3.11 or newer, the headers for the running kernel (kernel-devel on Fedora), make, dracut, grub2-editenv and mokutil. With DNF5, also libdnf5-plugin-actions.
  • GRUB with GRUB_DEFAULT=saved and GRUB_ENABLE_BLSCFG=true (Fedora's defaults).
  • Secure Boot off, or your own module signing key: pass --signing-key and --signing-certificate to prepare.

You need to first get a copy of your VBIOS, you can do it with GPU-Z on Windows, or with nvflash on Linux.

Note that you can't modify EVERYTHING for the VBIOS, since it's still on research of what can be modified, what not and which rails are exposed. Also for example, VRAM timings can be modified (not on these tools) but they aren't applied, since the card reads it physically from the flash hardware.

So, some example pics!

First, some VBIOS editor example, for a RTX 5090 VBIOS, a RTX 4090 VBIOS, and a RTX 3090 VBIOS.

Then, the VBIOS loader on Windows looks like this:

Then you can choose a VBIOS, for example I will select a 671W VBIOS for the 4090.

And press OK/Accept, and a quick GPU restart after (no windows reboot), it works!

And it works just fine!

PS C:\Users\User> nvidia-smi -i 0 -pl 671

Power limit for GPU 00000000:05:00.0 was set to 671.00 W from 550.00 W.

All done.

On the other hand for Linux, the UI looks very similar, and steps are similar, but you can also guide yourself on the github repo!

You can edit the values you modified on MSI Afterburner, mvolt+ (recommended) on Windows. On Linux you can use LACT.

Tested GPUs that I own, and it works:

• RTX 6000 PRO MaxQ: Unlocked voltage (stock 0.9V) to 1.1V, power from 325W to 600W, core offset to be able to do +2000Mhz, VRAM to be able up to +6000Mhz (it depends of your luck), and some OCPs.
• RTX 5090: PL from 600W to 800W, core offset to be able to do +2000Mhz, VRAM to be able up to +6000Mhz.
• RTX 5080: PL from 360W to 500W, core offset to be able to do +2000Mhz, VRAM to be able up to +6000Mhz.
• RTX 5070Ti: PL up to 450W, core offset to be able to do +2000Mhz, VRAM to be able up to +6000Mhz.
• RTX 4000 PRO SFF: PL from 70W to 105W, core offset to be able to do +2000Mhz, VRAM to be able up to +7000Mhz, unlocked voltage from 0.85V to 1.1V.
• RTX 5070: PL from 250 to 350W, core offset to be able to do +2000Mhz, VRAM to be able up to +6000Mhz.
• RTX 5060Ti: PL from 185W to 300W, core offset to be able to do +2000Mhz, VRAM to be able up to +6000Mhz.
• RTX 4090: PL up to 700W, core offset to be able to do +2000Mhz, VRAM to be able up to +6000Mhz.
• RTX 2000 Ada: PL from 70W to 105W, unlocked voltage up to 1.1V.
• RTX A2000 (Ampere): PL from 70W to 105W, unlocked voltage up to 1.1V.

Tested GPUs that I don't own, by other users:

• RTX 6000 PRO Workstation Edition
• RTX 5090
• RTX 5080 Laptop
• RTX 4070
• RTX 3060

For now, I can't seem to be able to edit Ada laptop variants (RTX 40 laptop series). Laptop Blackwell series (RTX 50 laptop series) can edit some parts of the VBIOS, but not all of them. Laptop Ampere series (RTX 30 laptop series) can edit almost everything.

And finally, the links are:

VBIOS Editor (Linux and Windows): https://github.com/Panchovix/nvidia-vbios-vix-editor

Custom VBIOS loader (Windows): https://github.com/Panchovix/nvidia-windows-vbios-vix-loader

Custom VBIOS loader (Linux): https://github.com/Panchovix/nvidia-linux-vbios-vix-loader

Everything is open source, so you can check the code to see if there are applicable fixes and such.

So if you guys can try, please let me know! If you find crashes on the vbios editor, or a specific GPU has info you can't edit and may be feasible to do so, you can raise an issue on github and I will check it out.

Warning: You use these software entirely at your own risk. Any modification you make with it, anything derived from it or from its output, and any damage that results, is solely your responsibility.

big speedup, but only for specific quantizations like Q2_K and Q3_K

Total Parameters
4B

Architecture
Looped (latent-recurrent) normalized-GPT with interleaved sliding-window attention (SWA) and full-attention layers

Precision
BF16

Context Length
Up to 256K tokens

Supported Languages
English

Best For
General-purpose reasoning on a small footprint

License
OpenMDW License Agreement, version 1.1

Release Date
October 16, 2026

NVIDIA-Nemotron-Labs-Dory1.0-4B-BF16 is a looped (latent-recurrent) large language model (LLM) trained by NVIDIA. The model employs a normalized-GPT architecture with interleaved hybrid sliding window attention (SWA) and full attention layers. By repeatedly applying a shared block of layers, Dory1.0-4B can spend more compute per token allowing it to scale intelligence. This model is ready for commercial or non-commercial use.

Already posted here https://www.reddit.com/r/LocalLLaMA/comments/1x85hfj/strata_qwen_38_flash_next_fast_but_whats_it/phavui9/ , but I believe this topic deserves it's own post.

The PLE, aka the n-gram table that sits on the SSD, is quantized heavily in the common GGUFs and has a measurable impact on performance, demonstrated here by /u/MarkoMarjamaa : https://www.reddit.com/r/LocalLLaMA/comments/1x6kw7k/qwen38_flash_next_unsloth_quants_kldtop1_with/

Further down in the comments, /u/Eyelbee mentions running the full bf16 n-gram table. I asked Grok and apparently that only works in llama.cpp et.al and is called the bf16 sidecar, Strata needs FP8 but that should be close to full quality, over my head to be honest. So I built it with ple_fp8_pack.py and indeed it seems to positively impact results. I run iq3_s and before the change the model made many more stupid mistakes. No impact on speed! Only needs a bit more disk space.

If you want to try this, you need model-00005 up to model-00037.safetensors and model.safetensors.index.json from https://huggingface.co/Qwen/Qwen3.8-Flash-Next-FP8/tree/main (Edit: I accidentally linked the original repo and not the FP8 one!), then pack them into a GGUF with

tools/ple_fp8_pack.py --model /media/your_ssd/Strata-data/download_folder --out /media/your_ssd/Strata-data/ple/ple-fp8.gguf

and load it in your strata-xxx.json with

"--ple-gguf",
"/media/your_ssd/Strata-data/ple/ple-fp8.gguf"

Adjust your file paths obviously. Sorry I can't upload 47gigs. :-) Maybe someone has uploaded it already but I couldn't find it. Not sure how to properly test performance, but here's two scenes with the same silly prompt.

Build a beautiful 3D scene of a train driving through a futuristic cyberpunk city in HTML. dystopian vibes, detailed models, try not to use too many boxes in the models. Be creative and add some user interactivity like camera and weather controls.

IQ3_S with the included PLE (quantized to IQ4_NL) https://output.jsbin.com/fojexecanu/

IQ3_S with the full FP8 PLE from Qwen https://output.jsbin.com/hosusuponi/

Imo a clear improvement is visible. I noticed that with Hermes agent too and was able to add several features to a KDE app (haruna). 3D models seem to have less errors like missing faces.

Hi. I'm wondering. Why agentic harnesses don't send their users the prompt cache binary data on inactivity so they eventually upload it back in the next turn?

I have been working on implementing adaptive KV streaming and RDNA 4 / ROCm optimizations for llama.cpp to enable running Qwen3.8-27B (IQ4_XS weights, Q8_0/Q4_0 KV cache) with its full 262,144-token context window on a single 16 GB RX 9070 XT GPU at 80+ t/s decode.

Qwen 27B IQ4_XS is ~14.3 GB, leaving insufficient VRAM headroom on a 16 GB card. The ASCII-condensed version trims non-ASCII vocabulary down to ~13.6 GB (12.6 GiB), leaving ~3.4 GiB of usable VRAM. Even with a Q8_0/Q4_0 KV cache, 262k context needs ~7.4 GiB for KV tensors alone, triggering OOM past ~65k tokens on llama.cpp.

I came across RaymondHuang210129/llama.cpp-adaptive-kv-streaming, which introduced adaptive KV streaming for NVIDIA/CUDA but did not support AMD/ROCm. I ported this to ROCm/HIP, retaining the full 262k cache in host RAM (~7.4 GB) while asynchronously streaming bounded working sets (~450 MiB prefill, ~2,100 MiB decode) over PCIe SDMA alongside compute.

I also incorporated kernel optimizations from stew675/llama-cpp-rdna-boosts (4-row multi-column mmvq, 16-byte vectorized loads for Q4_K/Q5_K, SwiGLU fusions, and single-queue async HIP graphs).

Further improvements:

• Sibling activation reuse: Reuses quantized activations across parallel projections (gate/up and Q/K/V) to avoid redundant memory passes.
• In-register fused residual add + RMSNorm: Retains activations in VGPRs between passes and directly writes quantized q8_1 copies for mmvq.
• Multi-column SwiGLU & GDN fusions: Extends fused Gate/Up + SwiGLU to 4-column verify batches in mmvq and fuses recurrent linear-attention gates for hybrid layers.
• Online Log-Sum-Exp prefill & pipelined prefetch: Computes chunked prefill across streamed pages via online LSE accumulation and overlaps layer L+1 PCIe transfers with layer L compute.
• Persistent graph arenas: Reuses arenas across prefill/decode phase switches to eliminate teardown overhead.
• Stream-optimized speculative decoding: Merged single-head MTP verify catch-up into the first draft step (cutting MTP forward passes in half and preventing draft KV pollution), dedicated arenas for verify graphs to eliminate rebuild bubbles, extended paged decode attention to verify batches, and stacked opportunistic n-grams with probabilistic rejection sampling.

Benchmark performance:

• Decode: 36.5 t/s (1k) -> 32.1 t/s (65k) -> 27.0 t/s (131k) -> 9.5 t/s (262k).
• Decode (with MTP): 71.8-86.5 t/s (1k-4k) -> 72.9 t/s (65k) -> 78.6 t/s (131k) -> 31.1 t/s (262k).
• Prefill: 1368.4 t/s (1k) -> 1120.3 t/s (65k) -> 863.0 t/s (131k) -> 585.8 t/s (262k).

Operating the 9070 XT at a 231W limit (-24% relative to 304W default TDP) incurs only a 3.6% prefill latency penalty and delivers virtually identical decode throughput, saving power and keeping the card quiet during long runs.

Take a look at the repository for the full details: https://github.com/achyudh/llama.cpp/tree/feature/adaptive-kv-stream-rocm

I swear I managed to get mine for like 1500 euro about 2 months ago but they're suddenly up by more than 1000 euro hovering about 2500-2800, what the hell is happening? Bro i shoulve bought two

During a post-event interview at the RTX Spark Launch Party: Taipei on October 16, 2026, Nvidia's Senior Vice President of Personal Computing, Jeff Fisher, explicitly debunked the recent discontinuation rumors.

Following a recent interesting post that challenges the performance of higher vs lower quants and generally advised sampling parameters (namely temp), I decided that I had too much free time on my hands and ran 12 test tasks.

I ran three models that work on my hardware:

- q3_xxs

- q3_s

- q4_xs

For each model I tested tasks with two temperatures :

- advised sampling parameters: temp 1,top_k 20, top_p 0.95, min_p 0, presence_penalty 0.0, repetition_penalty 1.0

- temp set to 0 both in strata json and in pi models json; other sampling params removed.

For each of the above model/temperature combination, I did 3 runs.

Task: in-browser Tetris game with a defined set of requirements for gameplay, settings and visual effects.

Quality assessment: Features delivered vs Requirements.

Results in the picture. Pardon my artisanal Excel dataviz skills.

Conclusions:

• All quants did OK-ish; not a single perfect result but all acceptable.
• Lower quant models don't trust themselves and generate way more tokens than higher quants. This is both from thinking and extra tool calls, tests, "actually"-s.
• Ktok total decode per task drops consistently with every quant step up.
• Higher TPS decode on lower quants does not compensate for bigger total decode and task time (including repeated tests, playwright, etc).
• Temp 0 task runs seem to have lower variance in task size/time; the sample is low but I would say that it's consistent with what it's supposed to do.
• Temp 0 may offer marginal task time decrease, better consistency and (supposedly) no impact on quality but the sample is way too low to draw conclusions. I expect that temp 0 will start underperforming in more complex tasks.
• Anectodally, no quality gain with higher quant; no drop either, even though it's done less testing then lower quants.

So, the conclusion on performance seems to be : Run the highest quant that you can afford with your hardware, disregard the decode speed, focus on task time.

I will stick with Q3s myself for now, as it's the highest quant that runs without ssd caching on my hardware, on Windows. Maybe will not be the case if I run it as a headless Linux inference box.

I'm still drawn on temperature and did not validate or invalidate the findings in the post linked at the beginning. I think I'll do some more deterministic/technical tests focusing on temperature alone.

European 78B A3.4B model is now supported by llama.cpp, I am waiting for some GGUFs

https://huggingface.co/Aleph-Alpha/Kolibri-1

Kolibri is Aleph Alpha's mixture-of-experts (MoE) reasoning model, with a focus on German and English. The model supports an explicit reasoning mode and tool calling. It is optimized for long-context and inference efficiency.

Model overview

I've got some nice boost by increasing --adapt-every value from calibrated 1 to 6.
I tested on 4 consecutive prompts(Slovak language) in chat. Prompt 1-3 are simple questions, 4. is describing source code(20K tokens)

Results:
[prompt number / ~avg. output length]

UD-Q4_K_XL:
[1 / 996] 30.3 -> 36.1 t/s (+19%)
[2 / 473] 31.6 -> 35.4 t/s (+12%)
[3 / 1245] 32.6 -> 40.1 t/s (+23%)
[4 / 4148] 27.8 -> 43.1 t/s (+55%)

UD-IQ4_XS:
[1 / 947] 57.9 -> 63.1 t/s (+9%)
[2 / 565] 51.9 -> 64.3 t/s (+24%)
[3 / 1162] 57.6 -> 71.6 t/s (+24%)
[4 / 4881] 56.0 -> 65.3 t/s (+16%)

PC: Intel 11700F@4.4GHz, RTX 4070 ti super, 64GB DDR4, NVMe, Arch linux TTY

I am not complaining, I just want to understand.

So I am running llama-server on a headless Linux with Qwen3.6 35B A3B at a Q5_K_XL quant (26GB safetensor) and 132k context and KV cache at q8_0.

I have a 4060TI 16GB with an old i7 gen 7 processor and 48GB DDR4 sysRAM.

When I run a dense model of like 26B (Gemma or Qwen) at a Q4_K_M quant I push it with -ngl 25 or so to the limit of my available VRAM and then creep at around 2 t/s through molasses.

But with the MOE version I get relatively good speed while only occupying 4.5GB VRAM:

> $ ./build/bin/llama-server -m /mnt/nvmepci/llms/unsloth/Qwen3.6-35B-A3B-UD-Q5_K_XL.gguf \
-c 131072 \
-np 1 \
-ngl 99 \
--cpu-moe \
-fa on \
-ctk q8_0 \
-ctv q8_0 \
--host 0.0.0.0 --port 8000

22.18.899.555 I slot print_timing: id 0 | task 343 | n_gen = 218, tg = 19.54 t/s, tg_3s = 19.57 t/s

22.21.923.659 I slot print_timing: id 0 | task 343 | n_gen = 277, tg = 19.53 t/s, tg_3s = 19.51 t/s

22.23.102.888 I slot print_timing: id 0 | task 343 | prompt eval time = 13539.35 ms / 1795 tokens ( 7.54 ms per token, 132.58 tokens per second)

22.23.102.892 I slot print_timing: id 0 | task 343 | eval time = 15308.76 ms / 300 tokens ( 51.20 ms per token, 19.53 tokens per second)

22.23.102.892 I slot print_timing: id 0 | task 343 | total time = 28848.11 ms / 2095 tokens

22.23.102.893 I slot print_timing: id 0 | task 343 | graphs reused = 621

22.23.103.045 I slot release: id 0 | task 343 | stop processing: n_tokens = 2094, truncated = 0

22.45.681.142 I slot get_availabl: id 0 | task -1 | selected slot by LCP similarity, f_sim_best = 1.000 (> 0.100 thold), f_keep = 0.857

That's almost 10 times as fast as with the dense model and it leaves me puzzled. What's the catch here? I know it caches a lot in sysRAM however shouldn't that be an absolute bottleneck to push expert layers for each token through the PCIe lanes with that relatively old CPU and slow DDR4?

Okay so there are strongish open jevlikes. But jev's official benchmarks seem to indicate they think open models have bad calibration, and their ranking of which they think performs best doesn't match the open benchmarks. Which could mean the open benchmarks suck, or could mean jev's official benchmarks suck. But either way, it would be cool if there was an open benchmark that at least correlated strongly with the official one, because that would be a hint that maybe it's measuring something more like what the official one does.

The official one:

the official one (calibration)

official one (intelligence)

What I'd like to see is a local model which matches them on their benchmarks; but failing that, I want to see what sort of data one has to put in a benchmark in order to get these results. because they're not the results we see on the jev decision index. snapshot of the index results as I post this:

jev decision index results

so... that's odd. now obviously the official benchmarks don't particularly test that many others, but kev coming in second on the official one but not even on the first page on the unofficial one seems like a hint something is wrong!

Are there other good jev benchmarks? the only other I currently know of is jevbench, which is kind of a ghost town in terms of results count (and also has a website with a very bad ratio of content to information, and only tests on game playing strength.)

Ideally I'd like to see a benchmark which has breakdowns for individual skills tested, eg image parsing, video parsing, math, code next step, game playing, etc, and ideally that benchmark would cover different ways to present the same information to each model in case the models disagree on favored input format for different types of information.

If I have to I might make my own, but if I do I probably won't post it, because I just can't be bothered. Someone who can be bothered would get my appreciation!

I understand this kind of product may not be exactly right for this subreddit, but I still think it deserves some attention, as it's an LLM frontend for local engines in the first place, and can be used for general-purpose chats with LLMs once lore extraction is disabled in the settings.

Otaku is built with a focus on three key ideas:

• maintain writing quality as the story grows;
• keep the context small, even for long stories, without losing the details that matter;
• offer high-quality interfaces, in the browser and in the terminal.

The two interfaces:

• a web UI (customizable) — see the interactive demo;
• a terminal UI — see its own interactive demo.

Both the web UI and the terminal share the same functions with one exception for now — the characters library is in the web interface only and will be added to the terminal later. The difference between the interfaces is that in the terminal you execute the functions with slash commands (the reference is available with /help), while in the web UI the operations are available from the menu.

A while ago I posted about it here and got some positive feedback. Since then, it has gained new features:

• characters library that works both ways:
  • you can add cards to the library and start stories with them, or
  • you can start a story from scratch, and after characters are extracted automatically from your story, you'll be able to create a card in the library from a character, or link it to an existing card;
• story mode similar to NovelAI: you write not in the prompt box but directly in the transcript, and the model continues when you press Enter;
• attaching pictures, and the pictures stay in the context, so you can refer to them later in the story;
• web search tool so that the model can search for facts on the internet;
• questions tool so that the model can ask you questions on how to continue the story;
• injections like ST's Author's Note;
• custom cap on context size: when the story exceeds the cap, its middle is replaced with summaries.

The web interface now:

• is optimized for mobile;
• supports HTTPS and password protection;
• has several new layouts.

Install

Otaku is free and open source; it works on macOS, Linux and Windows. Install it with uv — uv tool install otaku — or see the GitHub README for other options.

Repo: https://github.com/enclavum/otaku

Get started

To start, you launch either otaku for the terminal or otaku web for the web UI. The web UI's default URL is http://localhost:9600. Two sample chats ("stories") along with sample character cards are imported on first start to give you an idea of the features and what play looks like, and you land right in the middle of one of them.

On first start, you choose a provider and a model: Otaku automatically detects local installations of llama.cpp, KoboldCpp, oMLX and LM Studio, and lets you pick from their models. Cloud providers (OpenRouter, NanoGPT) are also there — enter an API key and their catalogs appear. After exploring the provided stories, you can start your own chat with the /new command.

Roleplay

Otaku is primarily developed for creative writing and roleplay, an alternative to SillyTavern and the like.

The context-building mechanism itself is described in an engineering doc that may also interest some people.

Asking for feedback

Otaku is a personal side project, and I'd like to get feedback from the community on the product and on what to add.

Lots of hype around Strata lately and the speeds look great. But everything I see is tokens per second and needle tests.

Has anyone checked if quality holds up? Same GGUF on llama.cpp should answer the quant question, but Strata also prunes experts. Does that cost anything in actual use?

Is self-hosting even worth it on your electricity rate? The answer is 3 numbers you can plug in yourself :)

I made a calculator for it (one of 153 free tools, runs in your browser, nothing is uploaded and there is no sign-up):

https://appdoesit.com/apps/llm-cost-calculator

It does 2 jobs. 1) local side: watts x hours x your kWh rate, with 3 GPU presets (RTX 4070 Ti Super 500 W at ~1k/50 tok/s, RTX 5090 900 W at ~2.5k/120 tok/s, the same 5090 in eco-mode 650 W at ~2k/100). 2) cloud side: input, output and cached tokens at reference API rates ($0.25 / $1.20 / $0.02 per 1M).

The line that moves my answer the most is the cached-input rate. Most calculators guess at it, and that is where the crossover hides: a cached prefix re-read costs almost nothing on some providers and real money on others, so your local-vs-cloud answer swings with your provider's cache bill, not just the sticker price.

From my experience the crossover sits further out than most people expect once the prompt is long and repeated: on the fixed scenario (60k input / 50k cached / 2k output, 90% uptime), the electricity side is almost flat while the API side keeps climbing.

I've had the thread running 8 days with 40 comments, and the most useful replies were the people with their own rig numbers in it. Run the scenario that matches your workload and tell me where your line landed :)