Hey guys, I need some suggestions.

I want to build an AI automation that can take call recordings, convert them into text, and then summarize the conversation.

The calls are in English, and the main thing is that everything needs to run locally. The company deals with sensitive client data, so uploading recordings or transcripts to cloud-based AI services is not an option. Data privacy is a big concern.

I'm looking for something that's free and open source, where the entire process can run on the company's own computer or server.

I've heard about Whisper for transcription and Ollama for running LLMs locally, but I'm not sure which models would be best for this or how to connect everything together.

Has anyone built something similar?

- Which model would you recommend for transcribing English calls?

- Which local LLM is good for summarizing conversations?

- Can I automate the whole thing using tools like n8n?

- What kind of hardware would I need?

I'm open to any suggestions, existing projects, or custom solutions. Just want to make sure no company data goes to the cloud.

Would appreciate any help. Thanks!

i cant get a proper answer from anywhere. some say its qwen 3.8 27b, some say qwen 3.5 122b a10b? and then i hear also gpt 120 oss but from what i heard it seems to be a speed model? thing is i just want the smartest 96gb model and know how good it actually is since im thinking if i should get 96gb of vram (4x intel arc pro b60s). im pretty new to local ai btw. i dont prioritize speed, i care most about inteligence and in my opinion the smartest models (cloud ai not local) are models like claude sonnet 4.6 but in benchmarks it scores very low even when models that are higher scores feel worse? im not even sure what to do man :/

Hi all!

I just put out the first release of Dear, an open-source Android app that works as a second brain.

You share links, photos, videos, voice notes or text into it from any app, and a local LLM catalogs them for you. No cloud involved.

What it does

- System share target: share from any app and it's saved

- Gemma 4 runs on the phone (llama.cpp + libmtmd over JNI) to generate a title, summary and tags

- Photos: describes them and transcribes any visible text

- Videos: looks at a few frames and transcribes the audio

- Voice notes: transcribed in 30s chunks, then summarized

-Links: title/description; for public Instagram posts it also pulls in the carousel photos and catalogs each one

- Search over titles, summaries, tags and transcribed text, with type filters

- Ask mode: chat with your collection ("what was that link about slow living?"), with citations that open the source entries

- Export/import the whole collection as a .zip

Privacy

No account, no telemetry, no cloud, Android backup disabled. It goes online only to download the model once from Hugging Face and to fetch metadata for the links you save.

Try it

Install the APK from the release, then download the model from Settings (~3.4 GB for E2B, once)

The release page has the SHA-256 of the APK if you want to verify it.

Status

It's v0.1, so expect rough edges. Next up: semantic search with an embedding model and Instagram videos/reels.

Release https://github.com/deivdev/dear/releases/tag/v0.1.0

Source (MIT): https://github.com/deivdev/dear

I'd really appreciate feedback, especially on cataloging quality and speed on different devices.

Happy to answer questions!

Hi

I bought an old xeon server and can't figure out the reason for the abysmal performance compared to my desktop PC.

Server hardware:

Motherboard: Supermicro X10DRH-CLN4
CPU: 2 x Intel Xeon E5-2630 v3 (8 core)
Memory: 8 x 16GB + 8 x 8 GB DDR4 1866
Storage: Samsung 970 Pro 1TB
VGA: 7900XT 20GB
OS: KDE Neon latest

When I run ./llama-cli -m ~/llm/models/hf_models/Qwen/Qwen3.8-27B_Q4_K_M.gguf -ngl 99 -c 2048 I get around 7-9 t/s speed. Sometimes even lower.

My desktop:

Motherboard: Gigabyte B550 AORUS PRO
CPU: Ryzen 9 5900X (12 core)
Memory: 2 x 16GB G.SKILL Aegis DDR4 3200MHz
Storage: WD BLACK SN850P
VGA: 7900XT 20GB
OS: KDE Neon latest

In this case I get 30+ t/s.

llama.cpp is built with VULKAN support in both cases. GPU has high load in both cases when inference is running, and CPU has minimal load.

What could be the issue with the server?

• I know the OS is a weird choice, I also tested it with ubuntu Ubuntu 26.10 Desktop, same results.
• prompt processing is also similarly slow on the server

TL;DR: Compared to NInfer: 6× closer to BF16, up to 3.7× more context, about 25% slower decode.

https://github.com/The8Darkness/voidinfer

I have seen a lot of engines popping up that are all focusing on speed and sacrificing quality for it. I have been working on getting the highest possible quality Qwen3.8-27b inference engine optimized for the 5090 without sacrificing other areas too much (speed / context size).

The engine was developed and focused on windows since my machine is a gaming pc and I just wanted an hassle free experience that I can directly run after a gaming session without having to resort to dual boot or WSL. I know that Linux provides better memory handling and speeds.

In my experience, since it is realistic to run a main agent at full 262k context and a subagent at 135k context, some tasks can even perform faster with higher quality output than NInfer running a single agent at 209k context.

Keep in mind that is all with the 5090 driving the display and running full unmodified Windows 11.

Same RTX 5090, token-identical prompts, greedy decoding, 4K prompt / 1,024 tokens for decode. VoidInfer: EXL3 6 bpw + OSCAR INT4 KV + VeriCache. NInfer: NVFP4 + FP8 KV.

EXL3 6 bpw weights provide the maximum quality possible for 32gb VRAM. OSCAR compresses the KV cache on the GPU to INT4, which is what fits roughly 400k context under Windows on one card. Drafting runs on this compressed cache. VeriCache checks every drafted token against the exact FP16 KV history kept in host RAM, so the compression costs no accuracy (only speed).

I'm seeing all these personal AI assistants (Instinct, Muse, Dots, etc.) that seem mostly geared towards scheduling, booking, and buying stuff.

I feel like they sort of followed what happened when the tech-savvy first started running OpenClaw on Mac Minis with Telegram portals. Back then I used mine to monitor concert ticket drops.

As local models get better, it seems a matter of time before we can recreate those assistants in a privacy-first manner. Maybe this actually is the next computer, a box with a local model and a personal AI assistant harness loaded in.

WDYT?

Hi and sorry for same question as every one else ask.

I have Titan V 32GB ( Volta GV100 ) and 128 GB of DDR4 ECC LRDIMM 2133Mhz quad channel with E5-2690V4 cpu.

Using llama.cpp on freebsd so vulcan no cuda.

Looking for a model , more interested towards python, postgresql side.

Im kinda slow on understanding all the context , kv cache etc.

For example and simple question i ran bartowski/Seed-OSS-36B-Instruct-Guff Q5 K_M and got 14t/s without any kind of modifications.

What context size i should use and or other settings ? What would be the best model to suit my needs ? in the past i told myself i can live with 10t/s but i think after checking 14t/s i think i can live even with 5 t/s .

Edit: bartowski, not unsloth

Thanks !

Generative UI is honestly so OP for elevating the local LLM experience. I've been using local models for both coding and everyday chat for a while now, and generative UI basically forces the model to stop outputting a huge wall of text and render something genuinely easy to read instead.

I'm moving soon and browsing places has been so much easier with it on. Normally I ask a question and get a wall of markdown back when I actually want photos and quick specs I can scan.

It's definitely not ideal for every single prompt since the component library adds ~7.5k tokens to the system prompt, so you need the context budget for it. But if you have the headroom, it makes everyday chat so much nicer.

I know I might be a little late to the party on this, but I just added openui support to my chat client and tested it with the new (not really new) Mimo V2.6 Flash model and it's amazing.

I mean, with the right setup, you can build a great coding agent using a quantized open-weight model, but it still won’t be as good as Opus/Sol/Astra right now.

My main issue with cloud models is that sometimes an agent ends up reading sensitive data, even with guardrails and hooks in place. Sometimes you need to debug an API or query a database. Other times, the agent might access something you forgot to secure, and you end up having to revoke an API key.

So I built a privacy proxy to help with this. It obfuscates sensitive data and comes with a tutorial and a dashboard.

You can customize it with your own rules and terms, and optionally add a small LLM to review everything (I have an RTX 3080 with 10 GB of VRAM, and it works great for this).

You can also act as a human reviewer and have the agent wait for your approval.

https://github.com/Latet-AI/latet

I hope you find this helpful, please let me know if you try it! Thanks :)

Hey everyone,

Yesterday, KRAFTON released Raon-1-mini-preview (a 28.8B hybrid MoE with only 4.3B active parameters, scoring 70.8% on SWE-Verified) along with official GGUF weights:

• Base Model: https://huggingface.co/KRAFTON/Raon-1-mini-preview
• Official GGUF: https://huggingface.co/KRAFTON/Raon-1-mini-preview-gguf

However, upstream llama.cpp support is currently pending, and the model README instructs users to clone and build KRAFTON's private fork (krafton-ai/llama.cpp) to run it.

I've just opened PR #2733 on ik_llama.cpp providing full native runtime support and an HF-to-GGUF converter:
PR Link: https://github.com/ikawrakow/ik_llama.cpp/pull/2733

What's implemented in this PR:

• Full Architecture Runtime (LLM_ARCH_RAON1):
  • 18x Raon Delta Attention (RDA) linear layers with softplus erase/write decoupling (algebraically mapped directly onto the recurrent scan engine).
  • 7x NoPE gated MLA layers (192-dim non-rotational key channels).
  • 4-stream Helmert Manifold-Constrained Hyper-Connections (mHC) batched down to ~5,500 graph nodes.
  • 128 routed experts (top-6 + 1 shared expert) with SiTU-GLU bounded activation (beta=4.0 / 25.0).
• DFlash2 Speculative Decoding: Full multi-stream support, compatible out-of-the-box with KRAFTON/Raon-1-mini-preview-dflash2.
• HF to GGUF Converter: Adds Raon1Model to convert_hf_to_gguf.py, including on-the-fly MXFP8 dequantization for the routed experts.
• Pre-tokenizer: C++ regex state machine with canonical Unicode NFC normalization (tokenizer.ggml.normalizer.nfc).

Verification:
Tested and validated against the official GGUF weights (KRAFTON/Raon-1-mini-preview-gguf) on a memory-constrained Intel Core i5 CPU box:

• 731 tensors mapped and loaded cleanly.
• Graph optimization & numerical parity: Batched the mHC streams down to ~5,500 graph nodes (-41% reduction vs naive loops). Strict numerical parity confirmed (identical logits and outputs before/after batching).
• Greedy generation produces coherent output.

Quick test from the PR branch:

git clone https://github.com/ikawrakow/ik_llama.cpp
cd ik_llama.cpp
git fetch origin pull/2733/head:raon1 && git checkout raon1
cmake -B build -DGGML_NATIVE=ON
cmake --build build --config Release -j$(nproc)

./build/bin/llama-cli -m Raon-1-mini-preview.gguf -p "Hello!" -n 64

Feel free to test it out and report benchmarks on different hardware setups!

I’m using the official Quinn 3.827B q4_k_m through LM Studio and hermes with
9800x3d
3060 12 gb
4080 super 16gb
32 gb DDR5
I’m currently getting about seven tokens on 121k context trying to revamp my website with qwen is lacking in Speed is there any thing I can do to speed up the token count or is there any other quants you guys would recommend for my set up that would be more efficient? If not, any other coding models with quant would be greatly appreciated.

I was using OpenCode with MiniMax and DeepSeek (Free) to make a few mods for games. It was working fine, but today they added a Subscription for daily messages, so now I can't use it that often anymore.

So now I'm looking for a local AI coding agent that can handle complex, multi-step development tasks without subscription fees, API costs, or daily usage limits.

My PC:

• NVIDIA RTX 4070 Ti SUPER — 16 GB VRAM
• 32 GB DDR5 RAM
• AMD Ryzen 5 7600X
• Windows 11

I need an agent that can:

• Read and modify files across an existing project
• Execute PowerShell commands and scripts
• Diagnose errors, fix issues, and verify its own changes
• Handle long, multi-step tasks without constantly asking me to do everything manually
• Work with tools reliably, not just generate code or provide instructions

I've tried OpenCode & Open Code Visual Studio with different Models through Ollama, but the results have been disappointing. Sometimes I get Errors, sometimes the Model can't even do basic commands.

What local model + coding agent combination would you genuinely recommend for this hardware and use case?

I'm specifically looking for something that is free to use, with no API costs or provider-imposed daily limits. I understand that local models have hardware limitations, but I want the strongest practical setup for autonomous coding that will fit my system.

Please recommend combinations you've actually tested, including the model's quantization, approximate VRAM/RAM usage, and any limitations you've encountered.

Hello everyone 😀
I just started using chat gtp 2 months ago and I've used it for a programming projekt for my work any it has done a great job, but I am getting a little tired of the 5h limit while I'm in the zone for building and I'm worried the code gribbirish if someone was to review it.
2nd I'm a photographer in my free time, doing boudoir, nude art and sometimes more sexual pictures for clients and I wanna get my ideas into some visuals but get stuck by limitations of chat gtp.

What would be a good local AI to accomplish both ?

Thanks in advance 😀

Let's see which place we are building the llms locally.

The recent developments around Strata, Infernix, Project Maya, DFlash 2, etc. have made me rethink what the future of local LLM hardware might look like.

Until recently, running a 500B+ MoE model on a consumer GPU seemed more like a cool experiment than something genuinely useful. But now we're seeing projects running models like GLM-5.3 Flash and DeepSeek V4.1 Flash 1017 on gaming PCs with 24–32GB VRAM, using system RAM and sometimes NVMe storage for the rest.

Performance is still all over the place, but progress seems incredibly fast.

I'm currently running an RTX 5090 with 128GB RAM, and I'm wondering whether upgrading to a PCIe 5.0 platform with faster RAM makes more sense than investing in additional VRAM. Or whether I should just keep my current setup and see how far the software improves.

I can see two possible directions:

1. The 24–32GB VRAM crowd might benefit the most from these developments.

If quantization keeps improving, and engines get better at caching experts in VRAM and fetching or computing the rest efficiently, maybe gaming GPUs will remain surprisingly capable of running huge MoE models.

After all, only a fraction of the total parameters are active per token. If the software gets good enough at handling the memory hierarchy, maybe the gap between a 32GB GPU and a 96GB GPU could become much smaller than we'd expect.

Of course, RAM bandwidth, PCIe transfers, cache misses and SSD latency are still potential bottlenecks.

2. Or maybe there's a limit to how far you can push low-bit quantization.

Perhaps Q1–Q2 quants will always lose too much quality for serious, complex tasks compared to Q3–Q4 or higher.

And even if only a fraction of the model is active at once, you still need access to a potentially huge number of different experts over time.

In that case, having 64, 96 or even 128GB of fast VRAM could make an enormous difference. Multi-GPU setups, workstation GPUs or unified-memory systems might become increasingly attractive.

What I'm really curious about is the practical experience so far.

For those of you experimenting with Strata, Infernix, Maya, aggressive MoE quantization or multi-GPU setups:

• How much quality do you actually lose with Q1–Q2 compared to higher quants? Are these models becoming genuinely useful for complex coding and agentic tasks, or are we still mostly looking at impressive tech demos?
• What's currently the biggest bottleneck: VRAM capacity, PCIe bandwidth, system RAM bandwidth, or the inference software itself?
• Do you think software improvements could eventually make a 32GB GPU competitive with much larger VRAM setups for single-user inference, or will there always be a substantial gap?
• If you already own a 3090/4090/5090, would you invest in a faster CPU/motherboard/RAM, add another GPU, or just wait?

I'm especially interested in where people think we'll be 6–12 months from now, given how quickly these projects are progressing.

NVIDIA seems to be prioritizing datacenter hardware, high-VRAM consumer GPUs remain expensive, and the next GeForce generation is still uncertain.

So I'm wondering whether the smartest upgrade right now might actually be no upgrade at all.

Would love to hear from people who've tested these setups, especially if you've compared different quantizations or measured the actual impact of PCIe 4.0 vs 5.0 and additional VRAM.

I've spent the last week or so building something a little different from the usual local LLM setup.

Instead of taking a pretrained model and quantizing it, I wanted to build an assistant using integer rules, cellular automata, symbolic reasoning and smaller specialized systems.

I called it CAEL.

The first experiment was only 77 KB and could follow simple stories and keep track of things like who owns an object after several events.

That eventually turned into a modular assistant with separate components for conversation, maths, logic, basic coding and memory.

Everything runs locally on CPU. No transformer inference and no cloud API required for the core system.

Although I have to admit the project isn't particularly tiny anymore.

The current complete bundle is around 1.4 GB because I kept adding more capabilities and supporting data.

I recently tested it against Qwen2.5-0.5B on 174 questions. Qwen won pretty comfortably, but CAEL handled some basic logic and code tasks better than I expected.

The bigger issue right now is reliability. Sometimes the correct solver exists, but the system doesn't recognize that it should use it.

I've also connected CAEL to my Vynly project as a local chatbot and prompt helper, so I'm starting to see how it behaves outside benchmarks.

I'm curious if anyone else is exploring non-neural approaches for local AI.

Would you rather have one small general-purpose model or a collection of specialized components that can be upgraded independently?