Qwen3.5-4B running CPU-ONLY at up to ~11+ tps on a Ryzen 5 laptop. No GPU. Running models in RAM and CPU is something the Qwen4 team is already thinking about. 💡 I built this standalone .exe because businesses are already deploying local models to save money and ensure privacy. (thumb drive friendly - just drag drop and doubleclick) My llama.cpp recipe 👉 🧠 Qwen3.5-4B Q4_K_M GGUF ⚙️ Ryzen 5 7540U — 6C/12T 🧵 --threads 9 🧵 --threads-batch 12 ⚡ --prio 2 🔄 --poll 50 📦 --batch-size 2048 📦 --ubatch-size 512 🚀 --flash-attn on 🧠 KV cache: q4_0 / q4_0 🔧 --repack 💾 --mmap 👤 --parallel 1 🚫 --device none 🚫 --gpu-layers 0 🚫 KV/op GPU offload 🚫 MTP OFF Interesting result 👉 MTP=3 was slower (~10 tps in benchmarks). Plain decode + 9 threads + Q4 KV hit ~11.4 tok/s. That's about 10% faster just from tuning llama.cpp — on a basic laptop CPU.

A new open LLM can remember an old conversation without putting that conversation back into its context. And at 128K history, its authors claim the persistent memory is ~66× smaller than keeping the full KV cache. MemT-1.0 Nano dropped today. Stats 🧠 36B total / 3B active ⚙️ built from Qwen3.6-35B-A3B 💾 10M dynamic memory state 🧠 451M trainable memory parameters 🔓 MIT-licensed weights But it does things a little differently. Normally, a long-running LLM needs old conversation + new message. So as history grows, you keep dragging more context/KV around, or retrieving old text and stuffing it back into the prompt. MemT puts memory inside the model itself. Its memory blocks run alongside the normal LLM attention. As you interact with it, the model learns to: 📝 write information 🗑️ forget information 🔄 update information 🧠 read it back later So you can: 💬 have a conversation 💾 save its latent memory state 🗑️ discard the plaintext conversation 🔄 start a new session with zero previous messages 🧠 restore the memory ➡️ and ask about what happened before Their demo does this. At 128K history, the authors report for MemT-Nano 📦 persistent memory → ~66× smaller than full-context KV cache ⚡ end-to-end inference → ~1.86× faster And the memory state stays a fixed size as history grows. MemT isn't a little memory model sitting next to Qwen. It takes a normal generative LLM and makes persistent memory part of the LLM's computation itself. They also released MemT-Flash: 🧠 288B total / 17B active 💾 336M dynamic memory state 🧠 3.61B trainable memory parameters 🎯 This could be especially interesting for long-running local agents. ⚠️ The 66× / 1.86× results are author-reported, and strong text-retrieval systems still beat MemT on some tasks where exact historical details matter. I can't find consumer GGUF benchmarks yet. Link in ALT.

This new-ish 35B multimodal model went from 3.38 → 20.51 tps on an iPhone. MiniCPM-V 4.7 35B-A3B was updated to decode faster on the iPhone by improving how it moves the model between storage and memory. Constraints 🧠 35B total / ~3B active 📱 iPhone 18 Pro Max — 12GB RAM 👁️ image + multi-image + video understanding Most of those 35B weights don't sit in RAM. The experts can stay in the phone's flash storage and get loaded when they're actually needed. Normally that's slow 🐌 SSD expert loading → 3.38 t/s So OpenBMB built Lookahead Expert Scheduling (LES) It predicts which experts the next layer will probably need and starts loading them while the current layer is still computing. Then they stacked the optimizations: 💾 baseline SSD loading → 3.38 tps 👀 expert lookahead → 6.57 ⚙️ optimized scheduling/loading → 11.34 🗜️ mixed-precision quantization → 17.16 🚀 + DSpark speculative decoding → 20.51 tps That's a 6.07× speedup over the same-phone unoptimized baseline. ⚠️ And these are still OpenBMB's own measurements. Link in ALT

A new 122B model dropped from @ornith_, and Localmaxxers already squeezed it into a 47.6GB GGUF for 64GB Macs. And Local AI users already have it running through llama.cpp. Ornith released Ornith-1.5-122B today. It's built on the Qwen3.5-122B-A10B architecture/lineage 🧠 ~122B total / ~10B active 📚 262K native context 🔓 MIT license 📦 GGUF + MLX already available ⚙️ llama.cpp + Ollama support And Ornith reports strong results against Qwen3.8-Flash-Next in the same harness 🧑‍💻 SWE-bench Verified Ornith → 84.5 Qwen3.8-Flash-Next → 82.8 💻 SWE-bench Pro Ornith → 76.9 Qwen → 43.9 🌐 BrowseComp Ornith → 78.5 Qwen → 75.3 AtomicChat has already released a GGUF ladder for it 💾 Q8 → 132.6GB 💾 71.2GB → 93.5% top-1 agreement with BF16 💾 62.2GB → 91.0% 💾 54.3GB → 88.8% 🍎 47.6GB → 86.1% And they specifically recommend that 47.6GB build for 64GB Macs. Support by llama.cpp already exists. ⚠️ Ornith also says the model can serve on a single 80GB GPU, but the exact checkpoint/precision behind that claim needs clarification. Link in ALT.

DeepSeek open-sourced the inference engine behind its own production AI service. DeepInfer is the framework DeepSeek uses to serve V4.1 Flash and they're giving away production tricks such as ⚡ DSpark speculative decoding 🧠 confidence-scheduled verification 📚 SWA bounded replay ⚙️ asynchronous scheduling 🧩 MoE expert parallelism + load balancing 🎯 structured JSON / regex / grammar output 🔬 deterministic + batch-invariant inference Plus DeepSeek's own stack of FlashMLA + DeepGEMM + DeepEP + DeepSelect + custom TileLang kernels DeepSeek says that future model inference code + new optimizations will land on this stack too. This is basically a look inside the code DeepSeek uses to make its enormous models practical to host. So now other inference harnesses can study the production implementation. ⚠️ Right now DeepInfer supports only ❌ no consumer NVIDIA GPUs ✅ NVIDIA SM100 ✅ Huawei Ascend 950 So this isn't Local AI yet. GH link in ALT

This is a little disappointing for MiniMax M3.1 Flash. The model behind Space Bunny landed at #26 on DesignArena's Overall Frontend leaderboard. MiniMax M3.1 Flash → 1290 Compared with some of the other big Chinese models: 🌙 Kimi K3 → 1360 🔴 DeepSeek V4.1 Flash → 1319 🟠 MiMo-V2.6-Pro → 1315 That's especially interesting because MiniMax's launch benchmarks look quite strong on coding and agentic work. But #26 is not the debut I expected after the Space Bunny hype. 👀

16GB of VRAM seems to be the last affordable consumer GPU tier. As such, people are getting the most out of them. Case in point, a 120B multimodal model is running on two RTX 5060 Ti 16GB cards. Someone got Nvidia's new Nemotron 3.5 Super VL 120B-A12B running locally with these stats 🧠 120B total / 12B active 👁️ text + image + video 📚 up to 1M context 🎮 2× RTX 5060 Ti 16GB → 32GB VRAM total 💾 96GB DDR5-6000 system RAM 👈 🎯 ⚙️ Vulkan + layer splitting ❌ no MTP Using Unsloth's NVFP4 build, they reported ⚡ generation → ~8–10 tps 🚀 prompt processing → ~150–170 tps So no, the whole model isn't sitting inside 32GB of VRAM. The machine is combining consumer GPU VRAM + ordinary system RAM to serve a 120B multimodal MoE. Link in ALT

So we have language models and decision models. Don't forget search-models exist too. One way to make your Local AI agent faster might be simple, just read less of the web. Artificial Analysis tested Exa's new search setup with Dynamic Highlights. But instead of huge chunking of webpages into the LLM, Exa decides how much useful text to take from each page across one shared context budget. The results 📚 tokens/task → 299K → 157K 🔎 searches/task → 9.4 → 7.6 💰 cost/task → $0.083 → $0.066 And quality went UP. Artificial Analysis Search Index 🥇 Exa Auto + Dynamic Highlights → 80 🥇 Perplexity Search Medium → 80 On DeepSearchQA Exa → 85.6 F1 Perplexity → 80.8 When a local agent searches the web, every unnecessary token it gets back still has to go through your local model. So maybe we should also optimize what the model has to read → smaller 🎯 Exa itself is a hosted search API, not a local/open-weight search engine. Link in ALT

So a few days ago I said DeepSeek V4.1 Flash 1017 had been squeezed from a ~510GB checkpoint → ~114GB, specifically targeting a single 128GB DGX Spark… …but nobody had actually benchmarked it on the Spark yet. Now they have. 👀 One DGX Spark: 🧠 DeepSeek V4.1 Flash 1017 💾 ~114GB VQ build 🖥️ 111–113GB memory used 💿 203GB Engram tables streamed from SSD ⚡ 39.4 tps weighted decode 🚀 ~1,145 tps prefill @ ~32K 📚 tested with a 110K-token prompt So yeah, a single 128GB Spark is actually serving it. Meanwhile, the independent scores are in. Artificial Analysis: Previous V4.1 Flash → 39 New 1017 → 45 🚀 That's +6 points and puts it equal #2 among open-weight models on their Intelligence Index. AA also measured: 💰 cost/task → $0.49 → $0.26 ⚠️ The Spark build is an aggressive ~1.6-bit VQ, so this is not full-checkpoint quality.

👀 A brand-new Local AI runtime is already beating llama.cpp on an 8GB RTX 3050 at least in some tests. SonicBoom v0.0.3 is a new compiler-first LLM inference engine from HardenedLinux. No Python at build time & no Python at runtime. And on a RTX 3050 8GB, its developers compared it against llama.cpp using the same GGUFs and same session setup. With Gemma 4 E4B Q3_K_M: ⚡ decode → 1.18× llama.cpp 📥 prefill → about the same With Qwen2.5-0.5B Q4_K_M: ⚡ decode → 1.12× llama.cpp 🚀 prefill → 1.09× It also does prefix reuse + execution-plan caching, which the developers say cuts TTFT by several times in their multi-turn agent workload. 🎯 Someone built a new inference engine and is already getting competitive performance on an 8GB consumer GPU. ⚠️ These numbers currently come from the developers themselves, on one RTX 3050. Now they're asking Localmaxxers with other GPUs to reproduce the benchmarks. Links in ALT

🍎 SGLang's main inference stack now runs on Apple Silicon. Previously, running SGLang on a Mac meant using its separate MLX backend. With v0.5.22, Apple Silicon now gets a standard PyTorch/MPS backend too. That means Macs can now run through SGLang's main pipeline ⚙️ scheduler 🧠 model runner 💾 memory-pool system 🌐 serving stack while MLX remains available as another backend. And we already have a first benchmark. On an M4 Pro with 48GB unified memory, Qwen3-0.6B in an offline batch-4 test reached: ⚡ 65.9 tps median decode 🚀 2,531 tps prefill On an M4 Pro with 48GB unified memory, SGLang reports Qwen3-0.6B at: ⚡ 65.9 tps median decode 🚀 2,531 tps prefill The Mac support also understands Apple's unified-memory limits properly. SGLang now plans memory around the amount Metal will actually allow. Link in ALT.

👀 A diffusion LLM just got more than 40× larger and apparently didn't get slower. 🤯 Inception officially launched Mercury 3. And it looks like it was quietly live on OpenRouter for a few days before the announcement. The numbers ⚡ 1,068 tps reported by Inception 📚 1M context 🧠 >40× larger than Mercury 1 💰 $0.10/M input + $0.35/M output at launch Mercury 2.5 was already insanely fast at ~1,100 tps. This is insane! model size → >40× Mercury 1 intelligence → way up context → 260K → 1M speed → still ~1,000 tps Inception says it beats GPT-6 Luna and Haiku 5.5 on several benchmarks while running 4–8× faster than those models. ⚠️ OpenRouter is currently seeing only about ~300 tps not 1,068. 🎯 If open diffusion LLMs can eventually scale intelligence without giving up this kind of throughput, then local diffusion text models is the future. Link in ALT

Ant open-sourced a 0.5B model that can clone a voice, design a new one from text, and make it laugh, sigh, cough, or whisper. #1 open-weight model on Artificial Analysis Controlled Voice at 1069 Elo The smallest in the family has these stats 🎙️ Ming-3.0-TTS-0.5B → 0.5B dense → zero-shot voice cloning → design a voice with natural-language instructions → control emotion, pace, pitch + emphasis → laughter, sighs, breaths, coughs, hesitation → 7 languages → 44.1 kHz speech They released more than TTS. 🗣️ Ming-3.0-TTS-Tiny → 7.9B total / 1.3B active → multi-speaker dialogue + long-form speech 👂 Ming-3.0-Audio-Tiny → 9.3B / 2.7B active → understands speech, music + environmental sounds → spoken Q&A + dialogue 📝 Ming-3.0-ASR-Tiny → 8.6B / ~2B active → dialects, accents + contextual hotwords And there's a much larger: 🧠 Ming-3.0-Audio-Flash → 126B / 6.5B active 🎯 So the entire stack can 🎤 hear locally ↓ 🧠 understand locally ↓ 🤖 agent decides ↓ 🗣️ speak back in a cloned/designed voice No GGUF/llama.cpp yet, and no consumer-GPU benchmarks. Links in ALT

🐰 Remember Space Bunny? This anonymous model became #1 on OpenRouter for 10 straight days and processed 56.7T tokens. MiniMax just revealed that Space Bunny was an early MiniMax M3.1 Flash. Stats 🧠 428B total parameters ⚡ ~23B active/token 📚 1M context 🧩 MoE + sparse attention 👁️ native image/video understanding If the final checkpoint keeps roughly that architecture, raw weight math looks like (guesses) 💾 4-bit → ~214GB 💾 3-bit → ~161GB 💾 2-bit → ~107GB So possibly for these machines 🍎 256GB M3 Ultra 🔴 192GB Strix/Gorgon Halo 🟢 128GB DGX Spark 🖥️ multi-GPU + RAM-offload rigs And MiniMax M3 already has llama.cpp support, so the ecosystem isn't starting from zero. Today's API model is also super cheap 💰 $0.15 / 1M input 💰 $0.60 / 1M output 💰 $0.015 / 1M cached input Link to blog in ALT

A tiny 4B local model made RAG a lot less willing to BS you. LocalFirstAI had a common problem, their local RAG model was supposed to answer only from retrieved documents. If the answer wasn't in the material, it should refuse, but instead, on questions the documents couldn't answer, it still tried to answer 55% of the time. 👀 So they stopped asking the generative model to decide whether it knew. They put a separate local decision model in front of it. Retrieved docs + question ↓ 🧠 Kev-4B ↓ “Does the evidence actually contain the answer?” ↓ only then let the LLM respond The result (drum roll) 🎯 caught 92.1% of the questions that should have been refused ✅ only 1 wrong refusal out of 101 answerable questions 📈 overall correctness → 82.0% They tested a 27B decision model too. 🧠 Clef-27B → ~12 sec/question 🧠 Kev-4B → ~2 sec/question The little 4B model got almost the same gating result. So they chose the 4B model for the actual local deployment on a Mac Mini. 🎯 Don't waste your big generative model deciding whether it should answer. Use a tiny local model to decide first. Even cooler, they're doing the decision through llama.cpp's /v1/systemone style fixed-choice probability output. Link in ALT

A 16GB Radeon is running Qwen3.8-27B at 35 tps with 200K context. And of course, the whole thing does not fit inside 16GB of VRAM. Someone got Qwen3.8-27B IQ4_XS running on a single RX 9060 XT 16GB + ordinary system RAM: At ~20K context ⚡ 42.7 tps generation 🚀 658 tps prefill At ~200K context ⚡ 35 tps generation And the setup is targeting 262K context. The trick is a llama.cpp fork with KV-cache streaming between VRAM and system RAM, combined with: 💾 Q4 KV values / Q8 keys ⚡ MTP + n-gram prediction 🖥️ 24GB DDR5 system RAM about 16GB used here ✅ A 16GB consumer Radeon can drive a 27B model at ~35 tps at 200K context by using system RAM as another tier of memory. That is a lot of Local AI from one 16GB GPU. 👈 🎯 16GB GPUs are still serving up Qwen3.8-27B!! Links to Reddit and Fork in ALT

A new dense 32B model ran a 239K-token conversation inside a 16GB GPU-memory budget. Agens Volundr 32B v1 compares well to Qwen3.8-27B. And the trick isn't just the ~2.8-bit quant. 54 of its 72 layers don't build a growing KV cache at all. 👀 Blockway released Agens Volundr 32B v1 today. Stats 🧠 32B parameters 📚 262K configured context 💾 ~2.82 bits/weight on the 16GB build 🔓 Apache 2.0 weights ⚙️ MIT-licensed Agens Engine 📦 Official GGUFs already available Special Source 👉 instead of every layer accumulating KV as context grows 72 layers ↓ 54 keep fixed-size recurrent state ↓ only 18 layers maintain growing KV cache That helps with long-context memory Using Agens Engine's hard 16 GiB GPU-memory cap, Blockway fed it a 219K-token document and reports at T=0.6 ⚡ Chat → 57 tps ⚡ Code → 79 tps ⚡ Thinking → 71 tps ⚡ Tool calls → 67 tps They pushed the limits of the 239K-token envelope while GPU memory peaked around 15.5 GiB. At short context, the same profile reaches roughly 86–123 tps depending on workload. And there's already a separate 24GB profile using the 4-bit weights for cards in the RTX 4090 class. 🎯 Volundr attacks the other monster in the room, : the memory cost of long context. ⚠️ Those published 16GB and 24GB performance numbers were measured on a 48GB GPU running under hard memory caps, not yet on physical 16GB/24GB consumer cards. The official GGUFs work today with Blockway's llama.cpp fork. Stock llama.cpp support is still being upstreamed. Link in ALT

A new speech-to-text model of only 1.06MB can transcribe audio ~436× faster than real time on ONE CPU thread! No GPU. AuroraASR-1M is a super super small local speech-recognition model with these stats 🧠 949K parameters 💾 FP32 → 3.81MB 💾 FP16 → 1.91MB 💾 INT8 → 1.06MB 🔓 Apache 2.0 🖥️ CPU or GPU The developer measured the full end-to-end pipeline — feature extraction + speech recognition + CTC decoding at ... 🖥️ 1 CPU thread → ~436× real time 🔥 RTX PRO 6000 Blackwell → ~1,362× real time 🎯So 10 minutes of audio could be processed in about 1.4 seconds at the reported single-thread CPU rate. So to underscore, this isn't using an external language model! It's a tiny Conformer-CTC model trained on LibriSpeech and can run through Transformers locally. There's a tradeoff, accuracy AuroraASR-1M test-clean WER → 11.43% Whisper Tiny English → 5.6% So Whisper Tiny is substantially more accurate. Aurora is also 🇺🇸 English only 🔡 lowercase output ❌ no punctuation A 1MB model can now turn speech into text locally at hundreds of times real time on a single CPU core. That makes completely offline transcription, voice commands and embedded voice features almost absurdly cheap. 🔗 Hugging Face model in ALT.

Someone finally spent ~$8,000 on the new 192GB Ryzen AI Max+ PRO 495 Minisforum and tested the crap out of it. He spent the weekend running DeepSeek, MiniMax, GPT-OSS and Qwen on it. Now he's boxing it back up and returning it. 👀 Interestingly, the machine works as intended. With llama.cpp/Vulkan 🧠 GPT-OSS 120B → ~58 tps 🧠 Qwen3.8 Flash-Next → ~28 tps 🧠 MiniMax M2.7 230B-A10B → ~31 tps 🧠 DeepSeek V4 Flash 284B-A13B → ~18 tps That 192GB unified memory does exactly what you want 💾 Qwen3.8-27B @ 262K → 33.3GiB 💾 GPT-OSS 120B @ 131K → 61.3GiB 💾 DeepSeek V4 Flash @ 131K → 108.9GiB 💾 MiniMax M2.7 @ 196K → 127.4GiB He also found some huge optimization wins. Qwen3.8-27B: 🐌 no MTP → 9 tps 🚀 MTP → 24 tps That's 2.6× faster from speculative decoding. And cached agent turns changed the experience completely: GPT-OSS → 35 sec → 1.9 sec Qwen → 106.6 sec → 1.2 sec MiniMax → 131 sec → 4.4 sec DeepSeek → 164 sec → 1.6 sec So why return it? Ryzen AI Max+ 395 MINISFORUM 💰 $3,879 💾 128GB 🚀 256GB/s Ryzen AI Max+ PRO 495 💰 $7,399 💾 192GB 🚀 273GB/s That's roughly 91% more money for 50% more memory and only ~7% more memory bandwidth. The extra memory unlocks models the 128GB box can't comfortably hold. It doesn't make inference twice as fast. His final verdict was still “I am boxing this back up and sending it back. Not worth $8k!” 🔗 Original owner review in ALT.