I am wanting to generate images and video locally. I realize that I may have to get a 24Gb card to do this well. I'm fine with that.

But... can I really do it "well" locally? Or is the truth something like this -

"Locally is more work, just pay a subscription somewhere and save yourself a ton of time".

Can local ai somehow be competitive with paid services? I really don't want to spend a lot of extra time retrying and redoing. I've heard some people end up remaking their initial image 10 or 20 times before the Ai finally gets close to what they are wanting.

So hopefully someone here can give me the overall picture. Thank you

Curious what model you guys use. Started with lfm25. It’s not bad but wondering if you guys pushed it to qwen3.6. I think Qwen3.8 will choke. Lol

Hey guys, I need some suggestions.

I want to build an AI automation that can take call recordings, convert them into text, and then summarize the conversation.

The calls are in English, and the main thing is that everything needs to run locally. The company deals with sensitive client data, so uploading recordings or transcripts to cloud-based AI services is not an option. Data privacy is a big concern.

I'm looking for something that's free and open source, where the entire process can run on the company's own computer or server.

I've heard about Whisper for transcription and Ollama for running LLMs locally, but I'm not sure which models would be best for this or how to connect everything together.

Has anyone built something similar?

- Which model would you recommend for transcribing English calls?

- Which local LLM is good for summarizing conversations?

- Can I automate the whole thing using tools like n8n?

- What kind of hardware would I need?

I'm open to any suggestions, existing projects, or custom solutions. Just want to make sure no company data goes to the cloud.

Would appreciate any help. Thanks!

i cant get a proper answer from anywhere. some say its qwen 3.8 27b, some say qwen 3.5 122b a10b? and then i hear also gpt 120 oss but from what i heard it seems to be a speed model? thing is i just want the smartest 96gb model and know how good it actually is since im thinking if i should get 96gb of vram (4x intel arc pro b60s). im pretty new to local ai btw. i dont prioritize speed, i care most about inteligence and in my opinion the smartest models (cloud ai not local) are models like claude sonnet 4.6 but in benchmarks it scores very low even when models that are higher scores feel worse? im not even sure what to do man :/

Hi all!

I just put out the first release of Dear, an open-source Android app that works as a second brain.

You share links, photos, videos, voice notes or text into it from any app, and a local LLM catalogs them for you. No cloud involved.

What it does

- System share target: share from any app and it's saved

- Gemma 4 runs on the phone (llama.cpp + libmtmd over JNI) to generate a title, summary and tags

- Photos: describes them and transcribes any visible text

- Videos: looks at a few frames and transcribes the audio

- Voice notes: transcribed in 30s chunks, then summarized

-Links: title/description; for public Instagram posts it also pulls in the carousel photos and catalogs each one

- Search over titles, summaries, tags and transcribed text, with type filters

- Ask mode: chat with your collection ("what was that link about slow living?"), with citations that open the source entries

- Export/import the whole collection as a .zip

Privacy

No account, no telemetry, no cloud, Android backup disabled. It goes online only to download the model once from Hugging Face and to fetch metadata for the links you save.

Try it

Install the APK from the release, then download the model from Settings (~3.4 GB for E2B, once)

The release page has the SHA-256 of the APK if you want to verify it.

Status

It's v0.1, so expect rough edges. Next up: semantic search with an embedding model and Instagram videos/reels.

Release https://github.com/deivdev/dear/releases/tag/v0.1.0

Source (MIT): https://github.com/deivdev/dear

I'd really appreciate feedback, especially on cataloging quality and speed on different devices.

Happy to answer questions!

Hi

I bought an old xeon server and can't figure out the reason for the abysmal performance compared to my desktop PC.

Server hardware:

Motherboard: Supermicro X10DRH-CLN4
CPU: 2 x Intel Xeon E5-2630 v3 (8 core)
Memory: 8 x 16GB + 8 x 8 GB DDR4 1866
Storage: Samsung 970 Pro 1TB
VGA: 7900XT 20GB
OS: KDE Neon latest

When I run ./llama-cli -m ~/llm/models/hf_models/Qwen/Qwen3.8-27B_Q4_K_M.gguf -ngl 99 -c 2048 I get around 7-9 t/s speed. Sometimes even lower.

My desktop:

Motherboard: Gigabyte B550 AORUS PRO
CPU: Ryzen 9 5900X (12 core)
Memory: 2 x 16GB G.SKILL Aegis DDR4 3200MHz
Storage: WD BLACK SN850P
VGA: 7900XT 20GB
OS: KDE Neon latest

In this case I get 30+ t/s.

llama.cpp is built with VULKAN support in both cases. GPU has high load in both cases when inference is running, and CPU has minimal load.

What could be the issue with the server?

• I know the OS is a weird choice, I also tested it with ubuntu Ubuntu 26.10 Desktop, same results.
• prompt processing is also similarly slow on the server

TL;DR: Compared to NInfer: 6× closer to BF16, up to 3.7× more context, about 25% slower decode.

https://github.com/The8Darkness/voidinfer

I have seen a lot of engines popping up that are all focusing on speed and sacrificing quality for it. I have been working on getting the highest possible quality Qwen3.8-27b inference engine optimized for the 5090 without sacrificing other areas too much (speed / context size).

The engine was developed and focused on windows since my machine is a gaming pc and I just wanted an hassle free experience that I can directly run after a gaming session without having to resort to dual boot or WSL. I know that Linux provides better memory handling and speeds.

In my experience, since it is realistic to run a main agent at full 262k context and a subagent at 135k context, some tasks can even perform faster with higher quality output than NInfer running a single agent at 209k context.

Keep in mind that is all with the 5090 driving the display and running full unmodified Windows 11.

Same RTX 5090, token-identical prompts, greedy decoding, 4K prompt / 1,024 tokens for decode. VoidInfer: EXL3 6 bpw + OSCAR INT4 KV + VeriCache. NInfer: NVFP4 + FP8 KV.

EXL3 6 bpw weights provide the maximum quality possible for 32gb VRAM. OSCAR compresses the KV cache on the GPU to INT4, which is what fits roughly 400k context under Windows on one card. Drafting runs on this compressed cache. VeriCache checks every drafted token against the exact FP16 KV history kept in host RAM, so the compression costs no accuracy (only speed).

I'm seeing all these personal AI assistants (Instinct, Muse, Dots, etc.) that seem mostly geared towards scheduling, booking, and buying stuff.

I feel like they sort of followed what happened when the tech-savvy first started running OpenClaw on Mac Minis with Telegram portals. Back then I used mine to monitor concert ticket drops.

As local models get better, it seems a matter of time before we can recreate those assistants in a privacy-first manner. Maybe this actually is the next computer, a box with a local model and a personal AI assistant harness loaded in.

WDYT?

Hi and sorry for same question as every one else ask.

I have Titan V 32GB ( Volta GV100 ) and 128 GB of DDR4 ECC LRDIMM 2133Mhz quad channel with E5-2690V4 cpu.

Using llama.cpp on freebsd so vulcan no cuda.

Looking for a model , more interested towards python, postgresql side.

Im kinda slow on understanding all the context , kv cache etc.

For example and simple question i ran bartowski/Seed-OSS-36B-Instruct-Guff Q5 K_M and got 14t/s without any kind of modifications.

What context size i should use and or other settings ? What would be the best model to suit my needs ? in the past i told myself i can live with 10t/s but i think after checking 14t/s i think i can live even with 5 t/s .

Edit: bartowski, not unsloth

Thanks !

Generative UI is honestly so OP for elevating the local LLM experience. I've been using local models for both coding and everyday chat for a while now, and generative UI basically forces the model to stop outputting a huge wall of text and render something genuinely easy to read instead.

I'm moving soon and browsing places has been so much easier with it on. Normally I ask a question and get a wall of markdown back when I actually want photos and quick specs I can scan.

It's definitely not ideal for every single prompt since the component library adds ~7.5k tokens to the system prompt, so you need the context budget for it. But if you have the headroom, it makes everyday chat so much nicer.

I know I might be a little late to the party on this, but I just added openui support to my chat client and tested it with the new (not really new) Mimo V2.6 Flash model and it's amazing.

I mean, with the right setup, you can build a great coding agent using a quantized open-weight model, but it still won’t be as good as Opus/Sol/Astra right now.

My main issue with cloud models is that sometimes an agent ends up reading sensitive data, even with guardrails and hooks in place. Sometimes you need to debug an API or query a database. Other times, the agent might access something you forgot to secure, and you end up having to revoke an API key.

So I built a privacy proxy to help with this. It obfuscates sensitive data and comes with a tutorial and a dashboard.

You can customize it with your own rules and terms, and optionally add a small LLM to review everything (I have an RTX 3080 with 10 GB of VRAM, and it works great for this).

You can also act as a human reviewer and have the agent wait for your approval.

https://github.com/Latet-AI/latet

I hope you find this helpful, please let me know if you try it! Thanks :)

Hey everyone,

Yesterday, KRAFTON released Raon-1-mini-preview (a 28.8B hybrid MoE with only 4.3B active parameters, scoring 70.8% on SWE-Verified) along with official GGUF weights:

• Base Model: https://huggingface.co/KRAFTON/Raon-1-mini-preview
• Official GGUF: https://huggingface.co/KRAFTON/Raon-1-mini-preview-gguf

However, upstream llama.cpp support is currently pending, and the model README instructs users to clone and build KRAFTON's private fork (krafton-ai/llama.cpp) to run it.

I've just opened PR #2733 on ik_llama.cpp providing full native runtime support and an HF-to-GGUF converter:
PR Link: https://github.com/ikawrakow/ik_llama.cpp/pull/2733

What's implemented in this PR:

• Full Architecture Runtime (LLM_ARCH_RAON1):
  • 18x Raon Delta Attention (RDA) linear layers with softplus erase/write decoupling (algebraically mapped directly onto the recurrent scan engine).
  • 7x NoPE gated MLA layers (192-dim non-rotational key channels).
  • 4-stream Helmert Manifold-Constrained Hyper-Connections (mHC) batched down to ~5,500 graph nodes.
  • 128 routed experts (top-6 + 1 shared expert) with SiTU-GLU bounded activation (beta=4.0 / 25.0).
• DFlash2 Speculative Decoding: Full multi-stream support, compatible out-of-the-box with KRAFTON/Raon-1-mini-preview-dflash2.
• HF to GGUF Converter: Adds Raon1Model to convert_hf_to_gguf.py, including on-the-fly MXFP8 dequantization for the routed experts.
• Pre-tokenizer: C++ regex state machine with canonical Unicode NFC normalization (tokenizer.ggml.normalizer.nfc).

Verification:
Tested and validated against the official GGUF weights (KRAFTON/Raon-1-mini-preview-gguf) on a memory-constrained Intel Core i5 CPU box:

• 731 tensors mapped and loaded cleanly.
• Graph optimization & numerical parity: Batched the mHC streams down to ~5,500 graph nodes (-41% reduction vs naive loops). Strict numerical parity confirmed (identical logits and outputs before/after batching).
• Greedy generation produces coherent output.

Quick test from the PR branch:

git clone https://github.com/ikawrakow/ik_llama.cpp
cd ik_llama.cpp
git fetch origin pull/2733/head:raon1 && git checkout raon1
cmake -B build -DGGML_NATIVE=ON
cmake --build build --config Release -j$(nproc)

./build/bin/llama-cli -m Raon-1-mini-preview.gguf -p "Hello!" -n 64

Feel free to test it out and report benchmarks on different hardware setups!

I’m using the official Quinn 3.827B q4_k_m through LM Studio and hermes with
9800x3d
3060 12 gb
4080 super 16gb
32 gb DDR5
I’m currently getting about seven tokens on 121k context trying to revamp my website with qwen is lacking in Speed is there any thing I can do to speed up the token count or is there any other quants you guys would recommend for my set up that would be more efficient? If not, any other coding models with quant would be greatly appreciated.

I was using OpenCode with MiniMax and DeepSeek (Free) to make a few mods for games. It was working fine, but today they added a Subscription for daily messages, so now I can't use it that often anymore.

So now I'm looking for a local AI coding agent that can handle complex, multi-step development tasks without subscription fees, API costs, or daily usage limits.

My PC:

• NVIDIA RTX 4070 Ti SUPER — 16 GB VRAM
• 32 GB DDR5 RAM
• AMD Ryzen 5 7600X
• Windows 11

I need an agent that can:

• Read and modify files across an existing project
• Execute PowerShell commands and scripts
• Diagnose errors, fix issues, and verify its own changes
• Handle long, multi-step tasks without constantly asking me to do everything manually
• Work with tools reliably, not just generate code or provide instructions

I've tried OpenCode & Open Code Visual Studio with different Models through Ollama, but the results have been disappointing. Sometimes I get Errors, sometimes the Model can't even do basic commands.

What local model + coding agent combination would you genuinely recommend for this hardware and use case?

I'm specifically looking for something that is free to use, with no API costs or provider-imposed daily limits. I understand that local models have hardware limitations, but I want the strongest practical setup for autonomous coding that will fit my system.

Please recommend combinations you've actually tested, including the model's quantization, approximate VRAM/RAM usage, and any limitations you've encountered.

Hello everyone 😀
I just started using chat gtp 2 months ago and I've used it for a programming projekt for my work any it has done a great job, but I am getting a little tired of the 5h limit while I'm in the zone for building and I'm worried the code gribbirish if someone was to review it.
2nd I'm a photographer in my free time, doing boudoir, nude art and sometimes more sexual pictures for clients and I wanna get my ideas into some visuals but get stuck by limitations of chat gtp.

What would be a good local AI to accomplish both ?

Thanks in advance 😀

Let's see which place we are building the llms locally.