#moe
489 items
So... has anyone actually figured out whose model Elephant Alpha is yet? (www.reddit.com) Re. what ever happened to Cohere’s Command-A series of models? (www.reddit.com) Hey everyone, Nick Frosst here from Cohere. A few months ago Aidan (my cofounder) left a comment in here about our Command series and how we were working on some more powerful, open-weights models behind the scenes.
DeepSeek Updated their repo DeepGEMM testing Mega MoE (www.reddit.com) https://github.com/deepseek-ai/DeepGEMM/pull/304 https://preview.redd.it/vcmqwmvzijvg1.png?width=1014&format=png&auto=webp&s=76b1739925f0699b0763aa7814614dd40329c41e https://github.com/deepseek-ai/DeepGEMM/commit/a050d09461e86eb6bba35a8c74…
Qwen3.5-35B running well on RTX4060 Ti 16GB at 60 tok/s (www.reddit.com) Spent a bunch of time tuning llama.cpp on a Windows 11 box (i7-13700F 64GB) with an RTX 4060 Ti 16GB, trying to get unsloth Qwen3.5-35B-A3B-UD-Q4_K_L running well at 64k context. I finally got it into a pretty solid place, so I wanted to s…
Qwen3.6 35B-A3B is quite useful on 780m iGPU (llama.cpp,vulkan) (www.reddit.com) I have ThinkPad T14 Gen 5 (8840U, Radeon 780M, 64GB DDR5 5600 MT/s ). Tried out the recent Qwen MoE release, and pp/tg speed is good (on vulkan) (250+pp, 20 tg): ~/dev/llama.cpp master* ❯ ./build-vulkan/bin/llama-bench \ -hf AesSedai/Qwen3…
VLLM PR : New MoE model from Cohere soon (github.com via reddit) Easy, fast, and cheap LLM serving for everyone | Documentation | Blog | Paper | Twitter/X | User Forum | Developer Slack | 🔥 We have built a vLLM website to help you get started with vLLM. Please visit vllm.ai to learn more.
AMD Strix Halo refresh with 192gb! (videocardz.com via reddit) Looks like the next strix halo, the Gorgon halo 495 max will have more then 128gb! I already bought a strix halo mini forms couple months ago since the 2026 refesh rumors was not interesting.
Show HN: Run an 80B Qwen in 4.3 GB of RAM on a Mac, and a 35B on an iPhone (github.com via hn) Swiftlet Run 35B and 80B Qwen models on ordinary Apple devices, including iPhones. Swiftlet is a Swift + Metal runtime for the Qwen3-Next and Qwen3.5/3.6 MoE hybrid model family.
LiquidAI/LFM2.5-8B-A1B · Hugging Face (huggingface.co via reddit) looks like you can run it on any potato (A1B)! https://huggingface.co/LiquidAI/LFM2.5-8B-A1B-GGUF from LiquidAI: LFM2.5 is a new family of hybrid models designed for on-device deployment.
Comparison Qwen 3.6 35B MoE vs Qwen 3.5 35B MoE on Research Paper to WebApp (www.reddit.com) Note: First is Qwen3.5 35B MoE (Left) and Second is Qwen3.6 (Right) Hi Guys Just did quick comparison of Qwen3.6 35B MoE against Qwen 3.5 35B MoE. with reasoning off using llama.cpp and same quant unsloth 4 K_XL GGUF First is Qwen3.5 outco…
Qwen 35B-A3B is very usable with 12GB of VRAM (www.reddit.com) Hardware: RTX 3060 12GB 32GB DDR4-3200 Windows CUDA 13.x Model: Qwen3.6-35B-A3B-MTP-IQ4_XS.gguf The model is a 35B MoE, so -ncmoe matters a lot. Lower -ncmoe means more MoE blocks stay on GPU.
Tinygrad Driver testing! (www.reddit.com) Boutta Thrash some MoE speeds on a blackwell + m3 Ultra RDMA cluster. Theres a bit less than 2tb of ram here.
RTX 5080 16GB: Qwen3.6 35B MoE at 128k context — 56 tok/s, and why MTP doesn't help (www.reddit.com) MTP (Multi-Token Prediction) just merged into mainline llama.cpp at b9190. I promised u/WarthogConfident4039 a Qwen3.6 benchmarking round.
Local model on coding has reached a certain threshold to be feasible for real work (www.reddit.com) We ran open-weight 27B–32B models on Terminal-Bench 2.0 (89 tasks, terminal-bench-2.git @ 69671fb) through our agent harness. Best result was Qwen 3.6-27B at 38.2% (34/89) under the default per-task timeout — the same constraint the public…
FINAL-Bench/Darwin-36B-Opus · Hugging Face (huggingface.co via reddit) https://huggingface.co/bartowski/FINAL-Bench_Darwin-36B-Opus-GGUF Darwin-36B-Opus is a 36-billion-parameter mixture-of-experts (MoE) language model produced by the Darwin V7 evolutionary breeding engine from two publicly available parents:…
My thought on Qwen and Gemma (www.reddit.com) This spring is really hot since the localLLM giant, both Qwen and Gemma released major models. I'm really excited with those release and happy with their capability.
AIDC-AI/Ovis2.6-80B-A3B · Hugging Face (huggingface.co via reddit) We introduce Ovis2.6-80B-A3B, the latest advancement in the Ovis series of Multimodal Large Language Models (MLLMs). Building on the strong foundation of Ovis2.5, Ovis2.6 upgrades the LLM backbone to a Mixture-of-Experts (MoE) architecture…
Granite 4.1: IBM's 8B Model Matching 32B MoE (firethering.com via hn) IBM just released Granite 4.1, a family of open source language models built specifically for enterprise use. Three sizes, Apache 2.0 licensed and trained on 15 trillion tokens with a level of pipeline obsession that's worth understanding.
DeepSeek-V4 Technical Report [pdf] (huggingface.co via hn) DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence Technical Report👁️ Introduction We present a preview version of DeepSeek-V4 series, including two strong Mixture-of-Experts (MoE) language models — DeepSeek-V4-Pro wi…
Qwen3.6 35B MoE on 8GB VRAM — working llama-server config + a max_tokens / thinking trap I ran into (www.reddit.com) Built an open-source one-prompt-to-cinematic-reel pipeline on a single GPU — FLUX.2 [klein] for character keyframes, Wan2.2-I2V for animation, vision critic with auto-retry, music + 9-language narration in the same pipeline (www.reddit.com) Shipped this for the AMD x lablab hackathon. Attached video is one of the actual reels the pipeline produced - one English sentence in, finished mp4 with characters, story, music, and voice-over out (fast demo video, not the best quality).
Dense vs. MoE gap is shrinking fast with the 3.6-27B release (www.reddit.com) 27B Dense vs. 35B-A3B MoE): - Dense still holds the crown: It still wins out on most tasks overall.
For chat and Q&A: Which MoE model is better: Qwen 3.6 35B or Gemma 4 26B (no coding or agents) (www.reddit.com) [P] Built GPT-2, Llama 3, and DeepSeek from scratch in PyTorch - open source code + book (www.reddit.com) I wrote a book that implements modern LLM architectures from scratch. The part most relevant to this sub: Chapter 3 takes GPT-2 and swaps exactly 4 things to get Llama 3.2-3B: LayerNorm → RMSNorm Learned positional encodings → RoPE GELU →…
Qwen 3.8 Max Live Now (www.qwencloud.com via hn) Qwen3.8-Max Copied! Try AIAdd to Compare ReasoningVisual UnderstandingText Generation Overview ReasoningVisual UnderstandingText Generation 2.4-trillion-parameter MoE flagship delivering a comprehensive leap in coding and professional work.
Got DFlash speculative decoding working on Qwen3.5-35B-A3B with an RTX 2080 SUPER 8GB (www.reddit.com) ## Got DFlash speculative decoding working on Qwen3.5-35B-A3B with an RTX 2080 SUPER 8GB I managed to get **DFlash speculative decoding** working in llama.cpp on a pretty VRAM-limited setup. This was tested with the DFlash PR: https://gith…
MiMo-V2.5-GGUF (preview available) (huggingface.co via reddit) Hi, AesSedai here - I've put up a PR to support the text-to-text inference of MiMo V2.5 with llama.cpp (and should also support Pro, will work on those quants after finishing V2.5): https://github.com/ggml-org/llama.cpp/pull/22493 I've als…
Gemma 4 MTP vs DFlash on 1x H100: dense vs MoE results (www.reddit.com) Benchmarked Gemma 4 MTP and z-lab's DFlash on a single H100 80GB using vLLM and NVIDIA's SPEED-Bench qualitative dataset. Setup: Hardware: 1x H100 80GB Runtime: vLLM Dataset: SPEED-Bench qualitative Prompts: 880 total, 80 prompts across ea…
Larger Gemma-4/Qwen3.6 (www.reddit.com) Qwen3.5-122B-A10B at Q6_K is really good. Do you think we will see a larger MoE Gemma-4 or Qwen3.6 at some point?
Poolside Laguna XS.2 (www.reddit.com) 33B A3B MoE, Apache 2 licensed. Reported agentic results put it about level with Qwen 3.5 35B A3B, behind the 3.6 version.
Abliterated version of the new Qwen3.6-35B-A3B up on HF (www.reddit.com) Pushed an abliterated Qwen3.6-35B-A3B to HF. Worth noting because MoE abliteration is genuinely different from dense — the refusal signal lives in the expert path, not attention, so standard Q/K/V LoRA doesn’t cut it.
Tencent Hy 30B/7B/1.8B (www.reddit.com) from tencent: Hy-MT2 is a family of “fast-thinking” multilingual translation models designed for complex real-world scenarios. It includes three model sizes: 1.8B, 7B, and 30B-A3B (MoE), all of which support translation among 33 languages…
llama.cpp MTP support landed - Qwen3.6 27B at 2.44× on a Strix Halo, 2.17× on a RTX 3090 rig (www.reddit.com) PR #22673 (commit 4f13cb7) landed MTP speculative decoding in mainline llama.cpp on May 16. I tested it on two separate rigs.
First direct side by side MoE vs Dense comparison. (www.reddit.com) https://arxiv.org/pdf/2507.17702
Qwen3.6 One Shot Tetris Game (www.reddit.com) I am blown away by what this model can generate locally. I asked for a flashy Tetris game with particle effect and boy did it deliver!
REAP-pruned Nemotron-3-Super (512 -> 256 experts) + GRPO fine-tune + FP8/AWQ. AIME 2026 90%+. Benchmark inside. (www.reddit.com) Hey r/LocalLLaMA, Dropping a release I've been working on during AIMO3 (Kaggle competition). Took NVIDIA's Nemotron-3-Super-120B-A12B (latent MoE + Mamba2 hybrid), REAP-pruned from 512->256 experts (removed MTP layer too), LoRA-RL fine-tun…
New Release of ROCm based MLX LLM Engine - lemon-mlx-engine (www.reddit.com) Hey everyone lemon-mlx-engine just got done integrating TheRock / ROCm 7.13 into the lemon-mlx-engine which means you get to try the latest ROCm on your local hardware with the MLX engine! This also includes various bug fixes and kernel fi…
↯ Qwen 3↯ Qwen 3↯ Qwen 3↯ Qwen 3↯ Qwen 3↯ Qwen 3↯ Qwen 3↯ Qwen 3↯ Qwen 3moe
Qwen 3.6-35B-A3B on dual 5060 Ti with --cpu-moe: 21.7 tok/s at 90K context, with benchmarks vs dense 3.5 and Coder variant (www.reddit.com) Qwen 3.6 dropped yesterday and I wanted to see if hybrid offloading actually earns its keep on this hardware. My box is two RTX 5060 Ti (32GB VRAM total) with 64GB system RAM.
Gemma 4 31B passed 7/8 real-world production tests — including ones I designed to make it fail. Full prompts + outputs. (www.reddit.com) I've been waiting for a capable free local LLM for a while. I think we're close — the quality is getting there fast, and Gemma 4 is the first open-weight model where I genuinely considered using it in production for simple-to-medium tasks.
DeepSeek-v4.1-Exp (huggingface.co via hn) DeepSeek-V4.1-Flash: Pushing the Limits of KV Cache Compression Technical Report 👁️ Introduction We introduce DeepSeek-V4.1-Flash, a multimodal Mixture-of-Experts (MoE) model with 552B backbone parameters and support for contexts of up to…
Is Qwen3.6 current king for local agentic use? (www.reddit.com) I've been testing other models but it seems like nothing even come close to Qwen3.6 35B A3B for agentic use. The worse I'd get is a loop sometimes, while Gemma4 produced broken tool calls occasionally and I couldn't even get GLM 4.7 Flash…
hipEngine: Fast Native Qwen 3.6 Inference for RDNA3 (Strix Halo, 7900 XTX) (www.reddit.com) A few weeks ago, after finishing FastDMS, I started toying around writing some RDNA3 kernels again to see how fast I could get Qwen 3.6 MoE running. It turned out well enough, so over the past couple weeks, I turned those experiments into…
Vulkan backend outperforms ROCm on Strix Halo (gfx1151) — llama.cpp benchmark (www.reddit.com) Just ran some llama-bench comparisons between ROCm and Vulkan backends on my Strix Halo system. Vulkan came out ahead, which surprised me.
Ran the same models across Strix Halo, RTX 3090, and RTX 5070 because I wanted my own numbers (www.reddit.com) I kept seeing inference-speed claims for these models and wanting an apples-to-apples comparison on the hardware I actually have. So I built a harness and a public page that dumps every run as YAML.
Show HN: Maple-Preview – ternary 20B MoE running at 120 tok/s on a iPhone (deepgrove.ai via hn) could not extract summary
MTP experiences on 7900xtx? (www.reddit.com) Hi! I have been using Qwen3.6 35B A3B happily the past few weeks, and I wanted to try out Qwn3.6 27B with the new fancy MTP speculative draft!
MiroThinker-1.7, an open-weight deep research agent (Qwen3 MoE base) — mini is 30B/3B active, curious what tok/s people get on consumer hardware (www.reddit.com) As usual, disclosure first: I'm on the team that built this. Our MiroThinker-1.7-deepresearch and 1.7-mini-deepresearch API went live, mini is a deep research agent built on Qwen3 MoE (30B total, 3B active for mini).
GPU advice for Qwen 3.5 27B / Gemma 4 31B (dense) — aiming for 64K ctx, 30+ t/s (www.reddit.com) Hey all, Looking for some real-world advice on GPU choices for running the new dense models — mainly Qwen 3.5 27B and Gemma 4 31B. What I’m targeting Context: 64K+ (ideally higher later) Speed: 30+ tok/s @ tg128 minimum Power: not critical…
Qwen3.6-35B-A3B Q4 262k context on 8GB 3070 Ti = +30tps (www.reddit.com) ..and on 8GB VRAM I can even push the context to 320K, 400K, 512K, and yes.. 1M.
Strip Qwen3.6 dense of its multimodal capabilities (www.reddit.com) This may be naive but if we stripped a model of its image processing/voice processing capabilities, can it make it smaller or faster? Is that even possible?
Alibaba open-sources Qwen3.6-35B-A3B, a 35B MoE model with 3B active parameters (huggingface.co via hn) Qwen3.6-35B-A3B [!Note] This repository contains model weights and configuration files for the post-trained model in the Hugging Face Transformers format. These artifacts are compatible with Hugging Face Transformers, vLLM, SGLang, KTransf…
Running Qwen3.6 35b a3b on 8gb vram and 32gb ram ~190k context (www.reddit.com) If anyone is looking for a good high-speed setup with ~190k context, this config has been working insanely well for me. I’m using my laptop as a server over Tailscale.
Qwen Models are such good models? (www.reddit.com) https://preview.redd.it/o1uxb57u47yg1.png?width=862&format=png&auto=webp&s=d38204fe6ccd0d8326dcd98a534e9a226d213f99 How trustworthy are Artificial Analysis intelligence index? so according to them Qwen 3.6 27B is better than bigger MoE mod…
qwen3.6-35b-a3b-mtp running on GTX 1060 6GB (www.reddit.com) I have this old 10-year old Dell T5810 workstation with 32GB ddr3(?) memory and a E5-2698v3 (16 cores 32 threads), a GTX 1060 6GB that's used for mining back in the old days (paid itself back many times over). I managed to get the model ru…
Experts first llama.cpp (www.reddit.com) This is for all with 12GB VRAM. Hi, I created a fork of llama.cpp with an experimental implementation of experts instead of layers.
Gemini 3.5 Flash is twice as expensive as ChatGPT 5.5 on GitHub Copilot. Also, Gemini reasoning models are MoE (www.reddit.com) Also FYI, Gemini reasoning models (2.5 Pro, 3.0 Pro and 3.1 Pro) were MoE. I don't know why this isn't more broadly discussed.
If you use continue.dev and Qwen 3.6 (dense / MoE) - I could use your help (www.reddit.com) Someone suggested I give Continue (Vscode extension) a try. I've been using Roo / Zoo now and liking it but it is pretty tough on context and I was told continue has more control over it.
Pushing the limit: minimax m2.7 q8_0 128k on 2x3090, 256GB DDR4 (www.reddit.com) CPU is just a secondhand 10900x. Using 128k context, unquantized kv cache.
Drastically improve prompt processing speed for --n-cpu-moe partially offloaded models (www.reddit.com) Bigger ubatch made gpt-oss-120b prompt processing much faster on my RTX 3090 I was tuning gpt-oss-120b-F16.gguf with llama.cpp on a 24 GB RTX 3090 and found that increasing the physical micro-batch size (-ub) can massively improve prompt p…
Has anyone tried Zyphra 1 - 8B MoE? (www.reddit.com) https://x.com/ZyphraAI/status/2052103618145501459?s=20 Today we're releasing ZAYA1-8B, a reasoning MoE trained on u/AMD and optimized for intelligence density. With <1B active params, it outperforms open-weight models many times its size o…
[7900XT] Qwen3.6 27B for OpenCode (www.reddit.com) I'm just looking for some advice on optimally setting up Qwen3.6 27B for OpenCode. The VRAM is a little bit scarce, but I ended up with this so far: llama-server --model models/Qwen3.6-27B-IQ4_XS.gguf \ --port 8080 \ --host 127.0.0.1 \ --t…
Llamacpp server : How do the -np and -c flags interact? (www.reddit.com) I've been using lm studio for a few months. I want to try hermes agents with Qwen 3.6 MoE, so I'm switching to llama.cpp and I don't understand well how the server slots -np and the context size -c interact.
Command A+ (218B MoE) running on Apple Silicon — MLX port, PR open (www.reddit.com) Cohere dropped Command A+ on the 20th (218B total / 25B active, 128 experts top-8, Apache 2.0). Wrote a cohere2_moe implementation for mlx-lm to get it running on Apple Silicon.
Any reason to run dense over MOE for RAGs? (www.reddit.com) I tend to use Claude for a lot of research and I also increasingly worry about things like misinformation or things in the model I can't audit. So, I'm building my own all in one RAG with big datasets like all of Wiki, research papers, all…
How small can the orchestration model in an agent be? (separating it from code-gen — that obviously wants a big model) (www.reddit.com) I'm building a local-first agent — a plain ReAct loop (think, pick a tool, observe, repeat) on a llama.cpp backend — and I want to be precise about a question that usually just gets answered with "it depends." It does depend. So let me spl…
Is there a limit on the number of active parameters in an MoE model? (www.reddit.com) Hi. We recently had MoE models as big as 1T and 1.6T total parameters.
24+ tok/s from ~30B MoE models on an old GTX 1080 (8 GB VRAM, 128k context) (www.reddit.com) I got Qwen 3.6 35B-A3B and Gemma 4 26B-A4B running on a $200 secondhand machine (i7-6700 / GTX 1080 / 32 GB RAM) using llama.cpp (the TurboQuant/RotorQuant KV cache quantisation allows 128k context within the 8 GB VRAM). Results (Q4_K_M mo…
Testing MiMo-V2.5-IQ3_S with 1'048'576 context (www.reddit.com) llama-server.exe --model "H:\gptmodel\AesSedai\MiMo-V2.5-GGUF\MiMo-V2.5-IQ3_S-00001-of-00004.gguf" --ctx-size 1048576 --threads 16 --host 127.0.0.1 --no-mmap --jinja --fit on --flash-attn on -sm layer --n-cpu-moe 0 --threads 16 --parallel…
Sorry if it's not the best place to ask this, of the models in the image, which is the best for (problem solving)/Coding and the best one for studying (ask LLM concepts) ? My PC build is RX 9060 XT 16GB + I3 12100F + 16 GB DDR4 + llama.cpp with Vulkan backend + Linux Mint. (www.reddit.com) I gave some math problems to Qwen 3.5 27B and Qwen 3.6 27B and they got all of them right, pretty smart models I would say, but very slow and electricity consuming, they took like 5 mins with my GPU at 120 W to solve a problem. The MoE mod…
What I got by 5060Ti 16GB + Qwen3.6-35B-A3B-UD-Q5_K_M (www.reddit.com) I tried local model couple weeks ago. At the beginning, I tried Ollama, but reddit says better to switch to llama.ccp.
A note of warning about DFlash. (www.reddit.com) It started saying 4/5x speed advantage against usual bf16 models (test are less optimistic but let think this is true). Then MoE gain is not that good, value was for dense models.
Mixture-of-Kittens: our open-source MoE megakernel for NVL72s (cursor.com via hn) Mixture-of-Kittens: our open-source MoE megakernel for NVL72s We're open-sourcing Mixture-of-Kittens, a deterministic MoE training megakernel for NVL72s that fuses communication and computation into a single kernel. Today, we're open-sourc…
I patched llama.cpp to gain 20% prompt processing TPS. Help me make a PR (news.ycombinator.com) I've been running Qwen3.6-35B-A3B locally on llama.cpp and noticed that prompt processing throughput gets too low with MTP. I got nerd-sniped.
Expert Selections in MoE Transformer Models Reveal Almost as Much as Text (arxiv.org via hn) We present a text-reconstruction attack on mixture-of-experts (MoE) language models that recovers tokens from expert selections alone. In MoE models, each token is routed to a subset of expert subnetworks; we show these routing decisions l…
Launch HN: General Instinct (YC P26) – Frontier models on edge devices (news.ycombinator.com) Hey HN, Guanming and Bill here from General Instinct (https://general-instinct.com/). After years of working in robotics, we kept running into the same problem: the best models never fit the hardware we actually had available.
Gemma4 26b MoE running in MLX with turboquant (and custom kernel) (www.reddit.com) TL;DR I spent a few crazy evenings this past week seeing if I could get Gemma4 running with proper turbo quant and rotating KV cache support. The answer was yes, and I'm now able to run Gemma4 26b on my MacBook Air M5 at 128k context with…
Qwen3.6:27b single-shot fixed a CSS UI bug that had Gemma4:26B doom looping uselessly for 15 minutes (www.reddit.com) Warning: long post ahead. On the bright side, it's 100 percent human-written, typos and all.
OBLITERATUS by elder_plinius anyone actually used it on a real model? Worth running over Heretic for MoE targets? (www.reddit.com) https://github.com/elder-plinius/OBLITERATUS
I ran an experiment on the 30b class of gemma4 and qwen3.5 models to try to learn about energy cost and performance tradeoffs. In other words, which models use more energy to give the same answer quality? (www.reddit.com) llama.cpp / ik_llama MoE Expert Offloading - Main Memory Bandwidth vs. PCIe Bandwidth (www.reddit.com) Qwen3.5 50% expert reduction success (news.ycombinator.com) We surgically removed half the experts from Qwen3.5-35B-A3B to create 8 memory efficient domain specialists (coding, web, math, physics, biology, engineering, vocational, humanities). A cross-domain test shows a 96-point pass@5 gap between…
How does MOE training ensure different experts are chosen? (www.reddit.com) I’m training a coding model that is basically a large model and a mini model built into one. Think of it like a person with two heads.
Run 290B+ frontier MoE models locally on your gaming PC (github.com via hn) | Download | Paper | Developer Slack | Community Discord | Community WeChat | Unlock datacenter-class intelligence on the hardware you already own — Run 290B+ frontier MoE models locally on your gaming PC at blistering interactive speeds.…
Ornith 1.5 35B-A3B: SOTA 35B MoE model (huggingface.co via hn) Ornith-1.5-35B-A3B Chirp Chirp! 🐦 We are introducing Ornith-1.5, a major step toward building foundation models through end-to-end self-improvement.
AntGroup releases Ling 3.0 Flash MoE model (twitter.com via hn) Today, we’re releasing Ling-3.0-flash—a hybrid-reasoning MoE model built for production-scale agents. 124B parameters.
PithTrain – a compact, agent-native MoE training system (blog.mlc.ai via hn) TL;DR. PithTrain is a compact, agent-native Mixture-of-Experts (MoE) training framework, in about 11K lines of Python.
I ran GLM-5.1 on a 16GB RAM machine (github.com via hn) 🧠 MoE-on-a-Potato Running a 754-Billion Parameter LLM on a 16GB RAM Consumer PC "Saying it's impossible is not engineering. Saying we don't know how yet is science." MoE-on-a-Potato is an experimental project dedicated to testing the extre…
Server build for local inference. 128 gb 3200 or 256 gb 2133mhz RAM? (www.reddit.com) Hi, I am building a server so that my dual rtx 3090 setup runs at full speed. - asrock romed8 t2 revision 1.3 - epyc 7642 - ddr4 128 gb 3200 or 256 gb 2133 (256 gb is a bit cheaper) 8 channel - dual rtx 3090 - gigabyte psu 1600 w What do y…
Why not dynamic active parameters (and other questions for the knowledgeable) (www.reddit.com) Why do we have to choose between MoE or Dense models? Wouldn't it be possible to have a model where the user can select the number of active parameters?
Cohere Open-Sources Command A+, a 218B Moe Model That Runs on Two H100s (firethering.com via hn) Cohere spent the past year deploying North, its enterprise AI workspace, with actual customers doing actual work. Agentic question answering over company file systems.
Running DeepSeek-V4 locally with 4x legacy RTX 2080 Ti ($2k budget setup). Custom Turing kernels, W8A8 quantization, and 255 prefill tok/s! (www.reddit.com) Hey r/DeepSeek, Who says we need an H100 cluster or the latest expensive GPUs to run frontier MoE models? I wanted to see how far we could push a single node of consumer legacy hardware, so we spent less than $2,500 total to build a budget…
What is the point of MoE models, beyond being faster? (www.reddit.com) Hi. Besides the fact that an xByA MoE models runs as fast as a yA models but produces better results, what are other benefits of pursuing an MoE architecture and not a dense one with e.g.
LLM's on Android (Snapdragon 8 Elite) MOE Experience (www.reddit.com) So I bought a phone with Snapdragon 8 elite (gen 4) and 24GB ram (Honor magic 7 pro). My experience has been mixed but with solid potential.
Developers who use local AI - Q4_0 vs Q8_0 KV quant? (www.reddit.com) I'd love to hear from developers who use big context windows if they notice a difference? Obviously I would love to cut the KV cache VRAM requirement in half, but I'm worried about quality especially when we enter into 50k+ context territo…
Who is your favourite quant publisher and why? (www.reddit.com) Hey everyone, I’ve been a big fan of Unsloth for several reasons: They publish models ASAP after release. They usually offer the lowest PPL.
how i can improve inference speed (www.reddit.com) specs : core i5 14400F 32gb ram d4 3200mhz rtx 4060 current speeds 30tps in output 500 tps in prefill command i currently use .\llama-server.exe ` >> -m "H:\model\unsloth\Qwen3.6-35B-A3B-GGUF\Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf" ` >> --host 0.…
Need advice: Qwen3.6 27B MTP or 35B-A3B MoE MTP on 16GB VRAM RTX 5080)? (www.reddit.com) Hey folks, looking for advice before I delete or keep a huge model file. I’m testing local coding/agentic workflows on an RTX 5080 16GB + 96GB RAM.
SubQ - claims to be a different architecture - anyone tried? (www.reddit.com) Has anyone tried SubQ, LLM using a so called " fully sub-quadratic sparse-attention architecture (SSA)" as opposed to flash attention - https://x.com/alex_whedon/status/2051663268704636937 Without flash attention - is it just a hybrid MoE…
Smaller gguf getting way less tokens per second?? So confused! (www.reddit.com) Noob here, Running Qwen3.6 35B A3B in LM Studio on a 3080 10GB + Ryzen 5 3600 on Windows 10. Tried some unsloth quants with identical settings (GPU offload 40, MoE layers to CPU 40, context 8192, flash attention on).
Five labs, one suite, do model families have personalities? (benchmark) (www.reddit.com) Bench 3 from my 18GB M3 Pro. Bench 2 was the 4B-class post where the comments were mostly right: I gave thinking models a fixed 1024-token cap, Qwen got kneecapped, Gemma E4B needed clearer active-param labeling, and the headline was partl…
XiaomiMiMo MiMo-V2.5 (not pro) - Architecture: Sparse MoE (Mixture of Experts), 310B total / 15B activated parameters (www.reddit.com) https://huggingface.co/XiaomiMiMo/MiMo-V2.5 Interesting because unlike its bigger brother it can be run on "more human" configurations
Show HN: I ported OmniAID image detection model to Apple's Neural Engine (apps.apple.com via hn) OmniAID is a hybrid MoE detector, so the PyTorch model dynamically routes each image through top-k semantic experts plus a fixed artifact expert. For the CoreML/ANE port, I rewrote that into a static graph.
[Qwen3.6 35b a3b] Used the top config for my setup 8gb vram and 32gb ram, and found that somehow the Q4_K_XL model from Unsloth runs just slightly faster and used less tokens for output compared to Q4_K_M despite more memory usage (www.reddit.com) Config CtxSize: 131,072 GpuLayers: 99 CpuMoeLayers: 38 Threads: 16 BatchSize/UBatchSize: 4096/4096 CacheType K/V: q8_0 Tool Context: file mode (tools.kilocode.official.md) Metric M Model XL Model Difference Avg Tokens/sec 28.92 29.78 +0.86…
Better MoE model inference with warp decode (cursor.com via hn) Better MoE model inference with warp decode By flipping the parallelism axis we achieve 1.8x faster and more accurate MoE model inference. Most MoE inference systems organize the token generation path around experts.
SSD-Llama: SSD-Native Inference for Trillion-Parameter Moe on a Consumer PC (arxiv.org via hn) Frontier open-weight language models increasingly use Mixture-of-Experts (MoE) architectures to expand model capacity while activating only a small subset of experts per token. Local inference must nevertheless keep the complete expert poo…
I joined the 'anti-chatbox' interface movement, and build Moe because of it (moebot.app via hn) It has read your mail and your calendar, it remembers what you told it last month, and then it does the work. Free on your Mac.
Show HN: I made a tiny MoE/Engram viz tool (sw-ml-study.github.io via hn) As part of a ML study group I've been building tools to experiment with ML concepts. I use AI Coding agents to develop software, mostly cloud-based but I keep trying local LLMs on 12G, 16G, and 24G VRAM GPUs.
Ask HN: What do AI improvements at these labs look like? (news.ycombinator.com) I'm struggling to understand what improvements at labs look like. Some part of it is more training data, more weights, more post training.
Inference-only expert boost saves 8.5% reasoning tokens in Qwen 35B MoE (zenodo.org via hn) This paper introduces layer-scoped expert-budget expansion, a training-free, runtime-only routing modification for sparse Mixture-of-Experts (MoE) language models. By expanding the expert selection budget ($N \ge K$) exclusively within lat…
Moe expert offloading on a 2-core Celeron with 2.7GB RAM (github.com via hn) moe-offload-bench Real, on-hardware measurements of MoE expert-weight offloading on a resource-constrained box: does prefetching mixture-of-experts weights off disk, ahead of the matmuls that need them, actually help — and if so, which mec…
Imprint – Fine-tune MoE LLMs bigger than your RAM (github.com via hn) 🐣 imprint Fine-tune frontier MoE models on hardware you already own. colibri taught your machine to run them.
Mixture-of-Kittens: An MoE training megakernel for NVL72 (twitter.com via hn) We're open-sourcing Mixture-of-Kittens (MoK), our MoE training megakernel for NVL72s. It fuses all Mixture-of-Experts communication and computation into a single, fully deterministic kernel, and runs up to 2.37x faster than the strongest p…
Show HN: RunNburn – Run a 295B Moe from a 98GB GGUF on a 64GB RAM Desktop (github.com via hn) runNburn is an Apache-2.0 Rust inference engine for quantized GGUF models that are too big for your fast memory. The core idea: weights stay file-backed (mmap), host residency stays under an explicit byte budget (--ram-budget), and GPU cac…
Ask HN: HotPin – lossless 120B MoE inference on 24GB RAM (CPU, 50 loc) (news.ycombinator.com) I'm a mechatronics designer with a background in control systems, robotics, PCB design, and embedded hardware. I design physical systems: motors, sensors, microcontrollers, and real-time control loops.
Instella-Moe: An Open Mixture-of-Experts Language Model (rocm.blogs.amd.com via hn) Introducing Instella-MoE: A State-of-the-Art Fully Open Mixture-of-Experts Language Model# AMD is excited to introduce Instella-MoE, a state-of-the-art fully open Mixture-of-Experts (MoE) language model with 16 billion total parameters and…
Benchmarking Qwen 3.6 35B MoE (3B active) on an RTX 3090 (www.gilesthomas.com via hn) Benchmarking Qwen 3.6 35B MoE (3B active) on an RTX 3090 I mentioned I'd got a second RTX 3090 on a group chat, and a friend said: I know this is not really your thing... but let me know how quickly it runs Qwen 3.6 35bn MoE.
Show HN: SkewAdam – A tiered optimizer that cuts MoE state memory by 97% (github.com via hn) SkewAdam Tiered optimizer state allocation for memory-efficient Mixture-of-Experts training. Training a 6.78B-parameter Mixture-of-Experts model with AdamW allocates 50.6 GB of optimizer state to update 12.6 GB of bfloat16 weights.
Show HN: Qwen3.6-35B-A3B on a 16 GB M1 Pro with SSD-streamed MoE (github.com via hn) DwarfStar Specialized local inference for models that do not fit in memory. A transparent research and co-development fork of antirez/ds4, focused on Metal, adaptive SSD streaming, common 16–64 GB Apple Silicon systems, and measured experi…
Inkling: A New Open-Weight 975B Moe with a Few Surprises (sebastianraschka.com via hn) Inkling: A New Open-Weight 975B MoE with a Few Surprises Interesting surprise open-weight LLM drop from Thinking Machines Lab yesterday. Their nearly 1T-parameter Inkling model looks pretty solid on the reported benchmarks.
Moe P. Wellington (objkt.com via hn) could not extract summary
LongCat-2.0 (news.ycombinator.com) https://huggingface.co/meituan-longcat/LongCat-2.0/tree/main. DeepSeek V3.2/DSA + their own LSA/N-gram/LongCat MoE The domestic scene [in China] over the past few years has truly been a battle royale + a massive leap forward.
LongCat-2.0, a large-scale MoE model with 1.6T total and 48B Active (longcat.chat via hn) 2026-06-30 GitHubHuggingFaceTry itAPI Access We are introducing and open sourcing LongCat-2.0, a large-scale MoE language model with 1.6 trillion total parameters and ~48 billion activated per token — a substantial step up from previous Lo…
Was GLM-5.2 trained on Opus 4.5 outputs? (1chat.com via hn) Recently there is a lot of excitement about GLM-5.2 which is an open-weight MoE LLM performing on Claude Opus 4.5 level in chat arena and overperforming all models except Claude Fable in WebDev arena [1]. Even though it is very good that t…
I was curious why MTP affects PP TPS in llama.cpp. My PoC recovers it? (news.ycombinator.com) I've been running Qwen3.6-35B-A3B locally on llama.cpp and noticed that prompt processing throughput gets too low with MTP. I got nerd-sniped.
Running a 35B MoE model on a 2017 AMD RX 580 8GB via Vulkan (no ROCm/CUDA) (github.com via hn) ██████╗ ██╗ ██╗ ███████╗ █████╗ ██████╗ ██╔══██╗╚██╗██╔╝ ██╔════╝██╔══██╗██╔═████╗ ██████╔╝ ╚███╔╝ ███████╗╚█████╔╝██║██╔██║ ██╔══██╗ ██╔██╗ ╚════██║██╔══██╗████╔╝██║ ██║ ██║██╔╝ ██╗ ███████║╚█████╔╝╚██████╔╝ ╚═╝ ╚═╝╚═╝ ╚═╝ ╚══════╝ ╚════╝…
Cohere's open agentic North Mini Code – accelerated with NVFP4 on spark-arena (forums.developer.nvidia.com via hn) Hey all, I just put up two Spark Arena runs of North Mini Code 1.0 — an FP8 reference and an NVFP4 quant we made — to see what the GB10’s native FP4 support buys us. It’s Cohere’s first open agentic coding model: a 30B MoE (3B active), Ap…
Unlocking Extreme AMD Instinct Inference with Software-Hardware Co-Optimization (rocm.blogs.amd.com via hn) ATOM: Unlocking Extreme AMD Instinct Inference with Software-Hardware Co-Optimization# As LLM serving enters a phase defined by high concurrency, long-context workloads, sparse MoE activation, and multi-GPU deployment, the challenge is no…
Mixture-of-Experts (Moe), Explained: Why "Active Parameters" Decide What Runs (vettedconsumer.com via hn) Here is a puzzle that trips up almost everyone new to local AI: a 671-billion-parameter model can run at usable speeds on the right desktop, while a "smaller" 70B model feels sluggish on the same hardware. How?
A 35B MoE on a 16 GB GPU, without the offload tax (www.lucebox.com via hn) Step 3.7 Flash – 198B-A11B MoE vision-language model (huggingface.co via hn) [ModelPage]: https://static.stepfun.com/blog/step-3.7-flash/ Introduction Step 3.7 Flash is a 198B-parameter sparse Mixture-of-Experts (MoE) vision-language model that combines a 196B-parameter language backbone with a 1.8B-parameter visio…
Rotary GPU: Exploring Local Execution for Large MoE Models Under Limited VRAM (arxiv.org via hn) Large language models have achieved remarkable capabilities through scaling, and this paper does not challenge that. It instead investigates a different question: once large models already exist, can they become more accessible to environm…
DeepSeek-OCR Visualized (medium.com via hn) 6 min read Dec 11, 2025 Understand SAM, Token compression, DeepSeek-MoE, Multi-Head-Latent-Attention. DeepSeek-OCR is essentially a combination of known architectures, namely SAM, CLIP and CNNs for the vision encoder and MoE decoder langua…
How Qwen3.6-35B-A3B fails differently as a sub agent compared to solo (www.reddit.com) Been running Qwen3.6-35B-A3B as a sub agent on a single 4090 for a few weeks. The failure modes are different from solo use and I haven't seen this written up anywhere.
Could someone please help explain these results? (www.reddit.com) I'm running Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf on 12 GB VRAM and 32 GB RAM via the TurboQuant variant of llama.cpp. I increased the --n-cpu-moe value from 8 to 30, and my inference rate doubled!
What nobody's measuring about dense MoE in production tool calling agents (www.reddit.com) Most of the model selection conversation I've seen focus on benchmark scores and cost (no surprise there). The question I can't find good production data on is whether dense vs MoE actually affects reliability for tool heavy agentic flows,…
Interesting paper advocates for quantized prefilling and precise decoding (arxiv.org via reddit) From other people's tests, NVFP4 decoding speed hasn't really allowed people to hit higher peaks (let's say: 85-90% memory bandwidth utilization) versus other approaches. The development leans toward a different class of optimization like…
Show HN: Modernizing my old PhD work in an evening with little Qwen3.6 MoE (github.com via hn) pge-jax JAX implementation of the Prioritized Grammar Enumeration (PGE) algorithm for symbolic regression. Overview pge-jax is a complete symbolic regression system that automatically discovers mathematical formulas from data.
LLM detector with science behind it (huggingface.co via hn) Arepo - MoE AI Text Workbench Arepo scores text with a 4D Benford/mantissa engine and a mixture of local Gaussian experts. It reports confidence levels and evidence classes instead of forcing every text into a binary Human/AI verdict.
Any good MOE ~60B models? I have 64GB vram (www.reddit.com) I have a build with 2 x MI50 32GBs and 64 gigs of DDR4 (bought before rampocolypse for ~630 USD total, I’m not rich) and I’m not gonna upgrade it for a long while. Are there any good MOE models that are around 60B in parameters so I can ma…
Strix Halo ROCm + MTP Notes (May 2026) (www.reddit.com) With the MTP merge into mainline llama.cpp I wanted to try out some other optimizations i could think of. Ended up tested backends, mtp, and bumping to ROCm nightlies.
Efficient use of Large system RAM (www.reddit.com) For example, if I have 128 GB of system RAM but only 16 GB of VRAM, am I still limited to models that fit within GPU memory (aside from CPU offloading techniques like MoE)? Are there ways to increase context size using system ram with usab…
EMO: Pretraining mixture of experts for emergent modularity (allenai.org via hn) Today we're releasing EMO, a new mixture-of-experts (MoE) model pretrained end-to-end so that modular structure emerges directly from the data without relying on human-defined priors. EMO lets you use a small subset of its experts – just 1…
How does llama-server pick which MoE experts go on the GPU and which stay on the CPU? (www.reddit.com) If you are using a MoE model that does not fully fit in your GPU, some of the experts must stay on the CPU. Putting the experts that you will actually need on the GPU will give you GPU inference speeds.
Show HN: Transformer Math Explorer (simonramstedt.com via hn) Interactive reference for transformer models, presented via dataflow graphs, drillable down to elementary mathematical operations. Covers models from GPT-2 to Qwen 3.6, with MLA, MoE, RoPE, MTP, hybrid attention, and other variants togglea…
Running Qwen3.5 / Qwen3.6 with NextN MTP (Multi-Token Prediction) speculative decode in llama.cpp — single RTX 3090 Ti GPU guide (www.reddit.com) I was asked for this guide, so here it is. Some overlap with someone else’s post from yesterday.
Fine-tuned Qwen3.6-35B-A3B DeltaNet experiment (www.reddit.com) I fine-tuned Qwen3.6-35B-A3B on its own outputs for $7 on Apple Silicon + Modal. DeltaNet LoRA targeting was the hard part.
tested four newest open source Kimi K2.6 is the fastest, GLM 5.1 the fanciest, DeepSeek V4 is the most comprehensive, and Xiaomi MiMo is the slowest (www.reddit.com) Architecture explains the gap: MiMo's MoE runs more active params per token than Kimi K2.6's optimized routing hence slowest. DeepSeek V4's 'comprehensive' edge is partly MLA: ~75% KV-cache compression makes it far better for long agentic…
APEX MoE quants update: 25+ new models since the Qwen 3.5 post + new I-Nano tier (www.reddit.com) Quick follow-up on APEX, the MoE-aware mixed-precision quant strategy. The original post was just about Qwen 3.5 35B-A3B ( https://www.reddit.com/r/LocalLLaMA/comments/1s9vzry/apex_moe_quantized_models_boost_with_33_faster/ ); since then t…
PI agent integrated with Cline-Kanban repo: All using PI and Qwen 3.6 35B MOE UD 4K_XL (www.reddit.com) Repo: statisticalplumber/kanban at pi-agent-integration Hi Guys, To test Qwen 3.6’s potential, I also wanted the Cline Kanban project to have an open-source agent to work with. The last time I tested Cline Kanban, it didn’t support agents…
Were Qwen3.6 models scrubbed from openrouter? (www.reddit.com) I made a simple app using openrouter, hoping to use the new small qwen models (the a3b moe and the 27b dense one), but they aren’t listed. Also, I swear some qwen3.6 models that were listed before are missing now.
Gemma 4 is not your standard transformer (idlemachines.co.uk via hn) Gemma 4 makes five quiet departures from the standard transformer recipe. QK-norm instead of 1/√d, partial RoPE on global layers, per-layer input gating, KV sharing across layers, and an MoE that sits alongside the MLP rather than replacin…
PSA re Qwen 3.6 35B A3B q4 + agents (www.reddit.com) Recommended parameters for Qwen 3.6 35B A3B on a 8GB VRAM card and 24GB RAM? (www.reddit.com) TPU v7x Ironwood vs Nvidia B200 (www.reddit.com) Google published Ironwood inference benchmarks in their AI-Hypercomputer/tpu-recipes repo. Nvidia has InferenceMAX numbers for B200.
Intel Lunar Lake 258V (32GB) vs Qwen 3.6 35B-A3B: Pushing the limits of MoP architecture. (www.reddit.com) Hardware: Intel Core Ultra 7 258V, 32GB Unified Memory. Model: Qwen 3.6 35B A3B (Quant: Q3_K_S) via LM Studio.
"LORAs"? (www.reddit.com) Hi. I'm curious about something.
Dense vs. Moe Models: Active Parameters, Throughput, and When to Choose Each (developer.nvidia.com via hn) How can a 30B-parameter model activate only 3B parameters per token, and still use the capacity of the larger model? Nemotron 3.5 Lightning illustrates the answer: It uses a Mixture-of-Experts (MoE) architecture that selects only a subset…
Gemma 4 26B on a 24 GB Mac Mini M4: from 66% CPU to 100% GPU (mac-mini-m4-doc.masterfabric.co via hn) TL;DR Three facts and one command. Bench — Mac mini M4 · 24 GB unified memory · macOS 26.5 · Ollama 0.34.0 · gemma4:26b Q4_K_M (MoE, 25.2B total / ~4B active) · measured 2026-09-13 ollama ps )sudo sysctl iogpu.wired_limit_mb=20480 # raise…
The Newsroom EP01 – OpenAI Ships GPT-5 to Azure – DeepSeek Open-Sources 236B Moe (www.youtube.com via hn) About Press Copyright Contact us Creators Advertise Developers Terms Privacy Policy & Safety How YouTube works Test new features NFL Sunday Ticket © 2026 Google LLC
Empirical Research on Moe Safety Mechanisms (dealign.ai via hn) Independent research into the safety architecture of large-scale Mixture of Experts reasoning models. 200+ controlled experiments.
Extending Raschka's GPT-2: an MoE trained from scratch on an RTX 3090 (www.gilesthomas.com via hn) Extending Raschka's GPT-2: an MoE trained from scratch on an RTX 3090 Mixture-of-experts models are really nifty. You get inference speed close to a small model's, with a lot of the smarts and knowledge of a large one.
Fractal-BLT – Zero-allocation .NET 10 MoE runtime streaming NVMe to GPU (github.com via hn) FRACTAL-BLT https://h4zey86.github.io/Fractal-BLT/ Disk-Native, Zero-Allocation Mixture-of-Experts (MoE) Inference Runtime in .NET 10 NativeAOT 🧭 Navigation & Guides Architectural Manifest - Deep dive into Zero-Allocation routing, unmanage…
Show HN: PicoLM v1.0-rc1 (github.com via hn) PicoLM is an LLM inference engine written in C99. It currently supports llama-2, GPT-2, Qwen 3.6/3.8(+MoE) and Gemma-3n models.
Show HN: PulsarForge – run a 744B MoE model on 32GB RAM with zero GPU (pure C) (github.com via hn) PulsarForge A from-scratch, CPU-only LLM inference engine in C11 that pushes a 744-billion-parameter model through a 2018 laptop with 32GB of RAM and a USB SSD — five architectures validated bit-exact against an independent oracle, cross-p…
LongCat-2.0 is now free to try in cline (twitter.com via hn) LongCat-2.0 is free in Cline right now. It's a 1.6T open weights MoE model with 1M context from @Meituan_LongCat scoring similar to Claude Opus 4.7 and Gemini 3.1 Pro.
Slotstream: Swift, SSD Expert Streaming, and Breaking the LLM Memory Wall (singularitymoments.com via hn) The Catalyst & The Problem: Bypassing the VRAM Monopoly When Alibaba released Qwen3.8-Flash-Next—a 125-billion-parameter Mixture-of-Experts (MoE) preview of the Qwen4 architecture—it highlighted a persistent dilemma in open-source AI: memo…
FreeToken to Speed Up Moe on low VRAM (www.youtube.com via hn) About Press Copyright Contact us Creators Advertise Developers Terms Privacy Policy & Safety How YouTube works Test new features NFL Sunday Ticket © 2026 Google LLC
Show HN: Moe-Direct – MoE Models far larger than your RAM, on a consumer desktop (github.com via hn) I wanted to try using the larger models on my computer (32GB RAM, RTX 5080, Gen5 NVMe), but the best I could do was around 30B. So I started with the idea that it might be possible by taking advantage of the fact that MoE models use only s…
Pushing the Limits of Serving DeepSeek-V4-Pro (www.lmsys.org via hn) DeepSeek-V4-Pro is a 1.6-trillion-parameter Mixture-of-Experts (MoE) model released with both FP8 and FP4 weights. Models at this scale naturally benefit from accelerators such as NVIDIA Blackwell GPU...
Soofi S 31.6B. German Open Moe Explained (kie.ai via hn) What Is Soofi S 31.6B? German Open MoE Explained Marcus Bell Frontier Models Correspondent TLDRSoofi S 31.6B is a hybrid Mamba-MoE with 3.2B active params, trained on ~27T tokens by a German consortium on Deutsche Telekom's Munich cloud.
FreeToken: Efficient Edge-Native Moe Serving with Bandwidth-Adaptive Execution (arxiv.org via hn) Frontier open-weight models are increasingly available, but serving them still largely assumes datacenter infrastructure. We present FreeToken, an edge-native MoE serving system that treats a personal machine not as a small GPU, but as a u…
Show HN: Ullis – Local Ternary Moe-Kan Training and Inference Engine in Rust (github.com via hn) Ullis Standalone ternary Mixture-of-Bumps Kolmogorov–Arnold engine. Zero Python at runtime.
Train-infer mismatch for Open-weight MoE RL in Open-source code (kiddyboots216.github.io via hn) Where does mismatch come from, and where does it go? In RL we sample rollouts with an inference engine and update our model by backpropagating gradients through log probabilities computed by prefilling those sampled rollouts through a trai…
Solving Moe Load Imbalance in LLM Training via Optimal Transport (arxiv.org via hn) Mixture-of-Experts (MoE) has become a key architecture for scaling large language models (LLMs), yet its dynamic routing causes severe load imbalance in expert-parallel training. Existing dynamic-replica methods copy hot experts onto idle…
Mixture of Experts (Moe): How Transformers Scale Without Activating Everything (chizkidd.github.io via hn) Mixture of Experts (MoE): How Transformers Scale Without Activating Everything. - Mixture of Experts (MoE) is one of the main techniques used to scale modern language models without making every token pay the full computational cost of the…
MoE routing is just branch prediction (ssenthilnathan3.github.io via hn) if you’ve spent time in both systems programming and ML infrastructure, you get this weird deja vu reading about transformer inference. the words are new.
Show HN: Offline 180.9M-parameter LLM and Agent inference on ESP32-P4 (github.com via hn) PFor 中文文档 PLE-MoE-W1.58A8 Architecture PFor is an LLM running on ESP32-P4, although technically it should be called an SLM. It has Instruct(ChatML) and Agent capabilities, despite both being extremely early and highly unstable.
Show HN: Hardware-software co-design for MoE models to bypass NCCL bottlenecks (github.com via hn) fluidic-expert-fabric (PoC) This repository contains an exploratory Proof of Concept (PoC) investigating a hardware-software co-design approach for distributed Mixture-of-Experts (MoE) architectures, such as DeepSeek-V3 and Mixtral-8x7B. T…
The Inference Engine Guide for K3 Deployment (twitter.com via hn) Kimi K3 is here: a 2.8T-parameter MoE with a 1M-token context and native vision, running on vLLM from day 0. Here is our canonical deployment guide: architecture, kernels, recipes, and the flags to run it in production.
The Voyage 4 model family: shared embedding space with MoE architecture (blog.voyageai.com via hn) TL;DR – We’re excited to introduce the Voyage 4 series, a new generation of text embedding models featuring industry-first shared embedding spaces. The series includes voyage-4-large , voyage-4 , voyage-4-lite , and the open-weighted voyag…
Show HN: Run a 120B-parameter MoE on Android mid-range phone CPU-only llama.cpp (github.com via hn) BigMoeOnEdge Run Mixture-of-Experts models far bigger than your edge device's RAM. The result: a ~60 GB model on a 12 GB phone: 1.3 tok/s lossless, byte-identical to running from RAM, 2.2 tok/s with one speed knob.
LingBot-Video: An open-source MoE video foundation model for embodied AI (github.com via hn) LingBot-Video 🌐 Project Page | 🤗 Hugging Face | 🤖 ModelScope | 📄 Paper | ⚖️ License| 💬 WeChat 微信 Group 📘 English Usage: English Documentation \ 📕 中文使用文档: 中文文档 We are excited to introduce LingBot-Video, the first open-source large-scale MoE…
Hy3 – 295B MoE model (21B active, 192 experts) (twitter.com via hn) 🚀Hy3 is here. 295B MoE.
Moe Estimator – Simulate decode speed with layer-major prefetch hiding (www.agrillo.it via hn) MoE SSD-Streaming Speed Estimator Includes Layer-Major Double Buffering (Prefetch Hiding) simulation and Model Presets Model Preset Hardware Architecture PIPELINED Estimated Decode Speed Bottleneck: Calculating... Model Allocation & Native…
Sharded Inference of a 229B-Parameter Moe over the Internet at Interactive Speed (twitter.com via hn) 1/ We published our first technical report today. We ran a 229B model split across five consumer GPUs in five countries over the public internet and measured 12.6 tok/s interactive, 194 tok/s batched.
Modular 26.4: SOTA Moe Serving, Model Bringup via Agent Skills, Mojo 1.0 Beta 2 (www.modular.com via hn) Modular 26.4 brings state-of-the-art mixture-of-experts (MoE) serving to Modular Cloud, expands MAX support for the newest open-weight models, and takes another step toward Mojo 1.0. Modular Cloud is expanding and now supports the latest f…
Efficient and Lossless Moe Diffusion LLM Inference with I/O-Aware Expert Offload (tide-paper.vercel.app via hn) TIDE: Efficient and Lossless MoE Diffusion LLM Inference with I/O-aware Expert Offload.
Show HN: Ported Cerebras REAP to MLX – Prune MoE Experts on a MacBook (github.com via hn) REAP MLX Apple Silicon REAP expert pruning for MLX-LM MoE models. Quick Start | Workflow | Supported Models | CLI Reference | Metrics | References | Development | License REAP MLX applies Router-weighted Expert Activation Pruning (REAP) to…
Show HN: Ministry of Everything – CLI agent harness for a single operator (github.com via hn) ▓▒░ MINISTRY OF EVERYTHING ░▒▓ Ministry of Everything (MoE) is a CLI-first harness for one operator directing AI agents through durable markdown work. MoE runs Claude Code or Codex against living markdown documents.
GH200 NVL2 or 8x RTX 6000 Blackwell for running Kimi K2.6 / DeepSeek V4 locally? (5 devs, agentic coding) (www.reddit.com) Trying to figure out the right box for my team and wanted to see if anyone had any clue which would be a better fit or if it is not worth our time in our budget. Situation: 5 of us doing agentic coding (lots of long context getting re-sent…
Is a 128 GB MacBook Pro M5 Max actually too slow for large-context local LLM coding workflows? (www.reddit.com) People are warning me about the prompt-processing speed of a MacBook Pro M5 Max with 128 GB RAM. My main concern is prompt ingestion / prefill latency and large-context handling — not raw token generation speed (which I think is OK).
Fused MoE dispatch kernel in pure Triton: 89-131% of Megablocks, runs on AMD with zero code changes (www.reddit.com) I've been working on MoE inference and wrote a fused dispatch kernel entirely in Triton, no CUDA. At inference batch sizes (up to 512 tokens) it reaches 89-131% of Megablocks(Stanford's CUDA-optimized MoE lib), and the same kernel runs on…
Dense vs. Moe Model (engineersmeetai.substack.com via hn) Yesterday, I ran out of tokens in OpenAI Codex while oxidizing parts of my Python codebase into Rust. It was around 11:30 PM, and I had to wait another two hours for the limits to reset.
Micro-Expert-Router: Running Mixtral-Class Moe Models on NVMe SSDs Without a GPU (github.com via hn) Micro-Expert-Router, SSD-Streamed MoE Execution Engine A Rust execution engine for Mixture-of-Experts models that keeps the router resident in RAM and hot-swaps individual experts on demand from a PCIe-attached NVMe drive into a pool of pr…
I'm running an agentic system with kobold.cpp as my backend. Am I losing performance? (www.reddit.com) Currently, I'm running a Hermes agent with an OpenAI v1 compatible endpoint provided by Kobold. My setup is a a 24GB 3090Ti + 512GB DDR4 running Qwen3.6-35B-A3B.
Moe inference optimizations: 15% lower expert load by request reordering (blog.doubleword.ai via hn) MoE expert co-activations: Reordering inputs yields easy throughput gains. Doubleword's batch inference offering keeps costs down by keeping throughput high, something which isn't easily done given the architecture of popular Mixture-of-Ex…
Volatile prefill speed after each reboot - llama.cpp (www.reddit.com) After every machine restart I get a different prefill speed, it can be only 300t/s or 1500t/s. It's like a lottery at each restart.
Command A+: Making sovereign agentic capabilities available to all (cohere.com via hn) Today, we’re releasing Command A+ open-source. A mixture-of-experts (MoE) model, Command A+ is an efficient, versatile, and privately deployable LLM built for high-performance agentic tasks with minimal compute overhead.
Qwen3.6 35B MTP, t/s varies on different scenario (www.reddit.com) Tried Qwen3.6 35B Q5_K_M MTP, HW: 9700x, 64GB 5600 RAM, 5060 TI 16GB. --n-cpu-moe 30 ^ -ngl 99 ^ -c 131072 ^ --no-mmap ^ --flash-attn on ^ --cache-type-v q8_0 ^ --cache-type-k q8_0 ^ --threads 8 ^ --parallel 1 ^ -rea off ^ --reasoning-budg…
Running Mimo 2.5 q4_k_m on single rtx5090 need recommendations (www.reddit.com) Getting 10.3 tps using this prompt: CUDA_VISIBLE_DEVICES=0 OMP_NUM_THREADS=8 GOMP_CPU_AFFINITY="0 2 4 6 8 10 12 14" ./build-mimo-5090-3090/bin/llama-server -m "$MIMO" -ngl 999 --n-cpu-moe 43 --no-mmap -c 100000 -ctk q8_0 -ctv q8_0 -fa on -…
Are the rich RAM /poor GPU people wrong here? (www.reddit.com) Hello Guys, I know everyone has his definition of local models, but for me i see 2 "reasonable" type of frontier local models. a dense one that barely fit in a 32GB ou 24GB of gpu for the most "reasonable" GPU wealthy guys and a MOE in the…
Stratum: System-Hardware Co-Design with 3D-Stackable DRAM for Efficient Moe (dl.acm.org via hn) Abstract Abstract As Large Language Models (LLMs) continue to evolve, Mixture of Experts (MoE) architecture has emerged as a prevailing design for achieving state-of-the-art performance across a wide range of tasks. MoE models use sparse g…
Anyone else experiencing heavy hallucinations with MiMo-V2.5 (310B) quantized version? (www.reddit.com) Has anyone else run into major issues with MiMo-V2.5 (the 310B total / 15B active MoE model from Xiaomi)? I tried the UD-Q4_K_XL quant from Unsloth.
Local-first LLM context dedup: 22-71% chunk overlap measured across 22M passages (2 arXiv papers). MCP server, MIT, 250KB binary, zero telemetry. (www.reddit.com) I'm the author of this thing, disclosure up front. Been hanging around this sub lately on cache invalidation, MoE memory tradeoffs, long-session token bloat.
The Trillion-Parameter Dilemma: MiMo-V2.5-Pro went open-source (1.02T params). Is self-hosting worth it when the API costs $70 for 387M tokens? (www.reddit.com) Xiaomi open-sourced MiMo-V2.5-Pro. 1.02 trillion parameters, 42B active (MoE), 1M context, MIT license.
Nemotron-Cascade 2: Post-Training LLMs with Cascade RL (research.nvidia.com via hn) We introduce Nemotron-Cascade 2, an open 30B MoE model with 3B activated parameters that delivers best-in-class reasoning and strong agentic capabilities. It is the second open-weight LLM, after DeepSeek-V3.2-Speciale-671B-A37B, to achieve…
Local LLM autocomplete + agentic coding on a single 16GB GPU + 64GB RAM (www.reddit.com) Today I set up a full coding toolbox on a single RTX 5080 (with RAM offloading) that's actually viable. Autocomplete: bartowski/Qwen2.5-Coder-7B-Instruct-GGUF:Q6_K_L Agentic: unsloth/Qwen3.6-35B-A3B-GGUF:UD-Q8_K_XL Why these models: Qwen2.…
I let four MoE LLMs from different model families argue stocks to try and pick the best ones. (www.reddit.com) I built an AI trading experiment in which four local LLMs argue bull and bear cases on stocks, and a host model grades the debate and decides BUY, SELL, or HOLD. Most days it holds.
Local Context Compression: Big or Small? (www.reddit.com) What are your thoughts/what is the consensus on local context compression model size? Are you guys using small MoE models to do this quickly and move along hoping you get all the important bits, or large dense models that take forever (giv…
MoE-Hub Taming Software Complexity for Seamless MoE Overlap on Multi-GPU Systems (arxiv.org via hn) The Mixture-of-Experts (MoE) architecture is crucial for scaling large language models, but its scalability is severely limited by inter-GPU communication bottlenecks in multi-GPU systems. Although overlapping communication with computatio…
Dual gpu question (www.reddit.com) Hı, i have rx 9060XT and rx 6600. 16gb and 8gb.
new MoE from ai2, EMO (www.reddit.com) new MoE release from ai2 - EMO, 1b-active/14b-total trained on 1t tokens interesting thing is document-level routing. experts cluster around domains like health, news, etc.
Swapped from a lighter agent runtime to Hermes Agent on a local 35B MoE — what changed (capability up, latency up, context budget down) (www.reddit.com) Two weeks of running Hermes Agent as the daily driver on a local stack. Sharing the trade-offs because anyone evaluating agent runtimes for local models is going to hit these.
ZAYA1-8B: An 8B Moe Model with 760M Active Params Matching DeepSeek-R1 on Math (firethering.com via hn) Who should care If you work with math, science problems, or complex coding tasks and you're looking for something small enough to run locally or cheaply via API, this is worth serious evaluation. The benchmark numbers at 760M active parame…
What models for coding are you running for a mid level PC? (www.reddit.com) I have a 4060 (8GB Vram) and 16GB of ram wondering which models could fit in my setup for coding, the new Qwen 3.6 and Gemma 4 MoE models look good but might not fit, wondering about your experiences
Zyphra releases the ZAYA1-8B MoE model optimized for intelligence density (huggingface.co via hn) ZAYA1-8B ZAYA1-8B is a small mixture of experts language model with 760M active parameters and 8.4B total parameters trained end-to-end by Zyphra. ZAYA1-8B sets a new standard of intelligence efficiency for its parameter count through a co…
Best Llama Config for Turboquant_Plus? (Stats below) (www.reddit.com) So I'm running the below and I've seen guys run this setup with TurboQuant_plus and get 35 tokens/second. I find the speeds I'm getting acceptable but if I could hit 30-35 I'd be soooooo happy.
Advice needed on eGPU and Mini PC (www.reddit.com) Hi all, I come across to relatively niche problem and could not find much useful posts or guides about it. I have a mini pc (Beelink Ser 8, 8745HS and 32GB 5600 DDR5 SODIMM) headless server for hosting some routing services, and I am wonde…
127³ — Superintelligence, public. DeepSeek V4 Pro (deepseek-v4-pro-127cubed.vercel.app via hn) DeepSeek V4 Pro 127³ 127-stratum crystalline lattice on DeepSeek V4 architecture. 1.6T params · 49B activated · MoE · 1M context · MIT license.
OpenAI's Privacy Filter vs GLiNER on 600 PII samples (www.reddit.com) Both models are open weight, both run on a local CPU workstation, both detect PII in text. Quick rundown of what I found.
Show HN: Phase Router – capacity-aware routing for MoE (github.com via hn) A deterministic, capacity-aware routing kernel that reduces dropped work in load-balanced systems. Trades microseconds of routing for milliseconds of saved compute.
Project Aurelia — A 3-model architecture (80B + 13B + 9B) that physically reacts to my real-time heart rate via mmWave radar, spatial awareness via Lidar, and Vibration via Accelerometer. (www.reddit.com) Hey everyone, I’ve been building a multi-agent system in my spare time, and I just open-sourced the repository. I was getting tired of the standard text-in/text-out chat paradigm and wanted to build a genuinely situated AI—one that actuall…
Memory upgrade, is it worth it? (www.reddit.com) Hi, I need your opinion on a system upgrade, 🤔 I currently have the following AI server used for various tinkering, learning, development etc. System AMD Ryzen 7 7700 (8C16T Zen4) Corsair Vengeance RGB DDR5 5600MHz 32GB MSI B650 Gaming Plu…
Qwen 3.6 35B-A3B takes a long time at image processing. Is it happening only to me? (www.reddit.com) 9900x, RTX 4080, 96GB RAM. Llama-cpp, Windows.
I tested 9 local models on the same flight sim prompt, all Q8, different Q providers, MLX (www.reddit.com) I gave 9 local models the same flight combat sim prompt. The results broke a few of my assumptions about quant providers and parameter count.
Gemma 4-31B vs Qwen 3.5-27B vs Qwen 3.6-35B-A3B on a browser-agent vision prompt — MoE wins on every axis (www.reddit.com) I was building a dedicated-vision-model feature for an open-source browser agent and wanted to figure out which local model to actually recommend. Wrote a small probe that sends the same image + same system prompt + same params (temperatur…
Deploying Gemma 4 26B A4B on a single RTX 5090 — ~196 tok/s with AWQ + vLLM on RunPod Serverless (www.reddit.com) Multi GPU setup help (www.reddit.com) Hi guys I managed to get a multi GPU setup going with a 3090 and three 3060 bringing my vram to 60gb along with 64gb ddr5. The objective is to run the largest coding model I can at a respectable token speed of over 20 tokens / second.
How is V100 32GB PCIE for LLM? (www.reddit.com) I have just brought one of these cards for non llm related reasons (new old stock), but I would enjoy the possibility of using it to run slightly larger models than currently allowed by my 4080 Super 16GB which will stay in the same box al…
Qwen3.6-35B-A3B — full JANG suite (15 profiles, 1L through 6K) for Apple Silicon (www.reddit.com) Full JANG adaptive mixed-precision quantization sweep of Qwen3.6-35B-A3B: https://huggingface.co/collections/bearzi/qwen36-35b-a3b-jang All 15 profiles, from extreme compression to near-lossless: JANG_1L JANG_2S/2M/2L JANG_3S/3M/3L/3K JANG…
I Lora trained Qwen 122B in NVFP4 on a single 128GB GPU (www.reddit.com) Huggingface loads it but instant OOM when it hits bf16 deepspeed zero3 with nvme offload. Loaded the shard but the weight names dont match(NVFP4 stores weight_packed/weight_scale, model expects weight) HF disk offloading - decompress befor…
How to run MoE models without necessary RAM? (Apple Silicon) (www.reddit.com) Hey, I have a M1 Pro 16gb machine, and I wanted to run the Qwen3.6/3.5 35A3B model. However, this model cannot fit on a 4bit quant on my system.
Who is actually behind the "Elephant-Alpha" stealth model on OpenRouter? (www.reddit.com) Has anyone else been tracking this? I just checked the OpenRouter daily rankings, and this anonymous "Elephant" (or Elephant-Alpha) model is sitting comfortably at the 8th spot.
Qwen3.5 35b is sure still one the best local model (pulling above its weight) - More Details (www.reddit.com) Last time I posted on how this model has performed in creating the webapp based on provided research paper. I got so much love to see people has appreciated the post and of-course the potential of this MOE model.
Ask HN: How do you prepare for a mid career Research Engineer role at neo Labs (news.ycombinator.com) Hey, I’m sure this question has been asked in various forms on HN. While I feel the answer might mostly stay the same, changes with various developments in AI - relevance of concepts like MoE, RL etc change - and the tools like custom Open…
Beyond the Previous Layer: Residual Predictive Structure in Sparse MoE Routing (arxiv.org) Sparse mixture-of-experts models route each token through a sequence of expert selections. We ask whether the immediately preceding selection adequately summarizes this trajectory for predicting the next router.
Colla-Q: Toward Collaborative Experts in MoE Quantization via Minimax Precision Balancing (arxiv.org) In this paper, we present a Mixture-of-Experts (MoE) quantization method based on activation entropy. Although quantization reduces memory and computational costs, it can substantially degrade performance.
Infinite-Parameter LLMs: Generating and Adapting Weights from Live Data (arxiv.org) The scaling laws hold that a language model grows more capable with more parameters and more training data, and Mixture-of-Experts (MoE) architectures have ridden these laws to remarkable results, activating only a fraction of an enormous…
Physically Partitioned KVCache Format for CPU--GPU Load Balancing in MoE Inference (arxiv.org) Single-GPU long-context inference with Mixture-of-Experts (MoE) models requires spilling the key-value cache (KVCache) to CPU memory. The spilled KV serves two complementary purposes---transferring to the GPU for attention computation, or…
OpWeave: Flexible Operator Disaggregation for Heterogeneous LLM Serving (arxiv.org) LLM serving systems increasingly disaggregate inference into finer-grained stages, with recent approaches separating attention from FFN or MoE execution during decode. This operator-level disaggregated serving (ODS) can improve hardware ma…
Task-Aware Federated Fine-Tuning for MoE-based Large Language Models (arxiv.org) Mixture-of-Experts (MoE) has become a widely adopted architecture for Large Language Models (LLMs), as it improves model capacity while limiting computational overhead through sparse expert activation. This property makes MoE-based LLMs pa…
Expert-Space Exploration in MoE Reinforcement Learning (arxiv.org) Reinforcement learning (RL) has become central to post-training of large language models. Recent advances in RL for Mixture-of-Experts (MoE) models have primarily focused on improving optimization stability and training efficiency, while t…
Dynamic Expert Quantization for Scalable Mixture-of-Experts Inference (arxiv.org) Mixture-of-Experts (MoE) has become a practical architecture for scaling LLM capacity while keeping per-token compute modest, but deploying MoE models on a single, memory-limited GPU remains difficult because expert weights dominate the HB…
SeqMoE: Toward Full-Load Performance via Predictive and Graph-Compatible MoE Offloading (arxiv.org) Mixture-of-Experts (MoE) creates a structural advantage for offloading: only a small fraction of activated experts need to reside in device memory, and if they can be loaded in time for computation, offloading can in principle approach ful…
FluxMoE: Decoupling Expert Residency for High-Performance MoE Serving (arxiv.org) Mixture-of-Experts (MoE) models have become mainstream for scaling language models to hundreds of billions of expert parameters. Despite sparse expert activation, existing inference engines keep all experts GPU-resident, crowding out the k…
How Fragile Is Safety Alignment at Frontier Scale? A Single-Direction Attack on a 320B MoE (arxiv.org) Directional ablation removes an aligned language model's ability to refuse by projecting a single "refusal direction" out of the weights that write the residual stream. It needs no gradient-based training and no optimization, only a few hu…
Distribution-Consistent Inference for Dynamic Sparse Mixture-of-Experts (arxiv.org) Mixture-of-Experts (MoE) architectures have emerged as a powerful paradigm for scaling model capacity while preserving efficient inference in large foundation models. However, most MoE models use a fixed top-$k$ expert selection policy, as…
EStream: Fast and Memory-Efficient MoE Prefill through Expert Virtualization on Mobile NPUs (arxiv.org) Mobile vendors and application developers increasingly deploy LLMs on smartphones for diverse prefill-only services. Yet current systems rely mainly on dense models whose regular computation maps efficiently to mobile NPUs, leaving more ca…
Latent-MoE: Domain-Aware Mixture-of-Experts for PDEs with Multi-Regime Physics (arxiv.org) Physics-informed neural networks (PINNs) struggle on PDEs whose governing physics varies across the domain. We trace this to a structural property of standard coordinate networks: their neural tangent kernel (NTK) is translation-variant an…
Beyond Retraining-Free MoE Compression: A Cost-Normalized Study of Post-Compression Adjustment (arxiv.org) Retraining-free MoE compression reduces deployment memory by pruning or merging experts, but often treats the compressed checkpoint as the final artifact. We argue that this view is incomplete: compressed MoE checkpoints are better underst…
Making Every Verified Token Count: Adaptive Verification for MoE Speculative Decoding (arxiv.org) Tree-based speculative decoding accelerates autoregressive generation by verifying multiple draft candidates in parallel, but this advantage weakens for sparse Mixture-of-Experts (MoE) models. As the draft tree grows, different branches ac…
Hyperparameter Scaling Laws Across MoE Sparsity (arxiv.org) Mixture-of-Experts (MoE) models expand model capacity without a proportional increase in training compute, but increasing sparsity makes reliable hyperparameter transfer challenging. In this work, we show that conventional hyperparameter s…
ExpertLens: Visualizing Embedding Spaces for Post-Hoc Explainability in MoE Enhanced Retrievers (arxiv.org) Neural models, including dense retrievers, have been widely adopted in Information Retrieval (IR), often delivering state-of-the-art performance. Despite their effectiveness, these models operate as black boxes, limiting the interpretabili…
ACE: Adapter Consolidation across Experts for Parameter-Efficient Fine-Tuning of MoE LLMs (arxiv.org) Parameter-efficient fine-tuning (PEFT) of mixture-of-experts (MoE) models commonly attaches a separate low-rank adapter to each expert. This expert-wise design fragments adaptation in three ways: capacity is split across narrow low-rank up…
Router Prior Bias: Preserving Base Routing Structure in MoE Post-Training (arxiv.org) Mixture-of-Experts (MoE) pretraining relies on an auxiliary load-balancing loss (LBL) to drive per-expert utilization toward uniformity. Post-training inherits a different situation: the base router already encodes non-uniform expert co-ac…
From Concentration to Differentiation and Back: Routing Effective Rank in MoE Reasoning Cohorts (arxiv.org) Test-time scaling produces cohorts of reasoning rollouts, yet there is no standard label-free account of how their internal computation reorganizes as inference unfolds. We introduce routing effective rank deff, the entropy-effective dimen…
Cache-Aware Joint Router Adaptation for Memory-Efficient MoE Inference (arxiv.org) Mixture-of-Experts (MoE) models activate only a small subset of experts per token, but the full expert set often exceeds GPU memory, causing repeated weight transfers during decoding. We formulate expert-cache management as a model-side al…
ACE: Adaptive Calibration-Free Expert Skipping for MoE-based LLMs (arxiv.org) Mixture-of-Experts (MoE) architectures provide an efficient paradigm for scaling large language models (LLMs), yet fixed top-k routing activates the same number of expert slots for every token, causing substantial redundant computation. Ex…
Expert-Aware Causal Tracing of Factual Recall in Sparse MoE Language Models (arxiv.org) Activation patching can identify a mixture-of-experts (MoE) block whose clean output restores a corrupted factual prediction. However, because the block output combines contributions from multiple routed experts, block-level rescue does no…
Routing Is Not Enough: Diagnosing Intra-Adapter Subspace Contention in MoE+LoRA Fine-Tuning (arxiv.org) Multi-domain fine-tuning often combines MoE routing with LoRA, assuming that token-level routing separates domain-specific updates. We test this assumption in MoE+LoRA using Python code paired with biomedical text and mathematical reasonin…
SEAL: Reinforcing Global Safety in Mixture-of-Experts through Shared Expert ALignment (arxiv.org) Mixture-of-Experts (MoE) is a scaling architecture for large language models that activates only a small subset of expert modules per token, enabling massive parameter growth with nearly constant computation. Recent Hybrid MoE architecture…
Beyond Modality Harmony: Orthogonal Purification and Topology-Guided MoE for Conflict-Aware Multimodal Recommendation (arxiv.org) Multimodal Recommender Systems (MRSs) typically rely on a flawed "modality harmony" assumption, presuming that multimodal features are inherently beneficial and strictly aligned with users' collaborative interaction patterns. However, moda…
DynaNDE: Dynamic Near-Data Expert Scheduling for Batched MoE Inference (arxiv.org) Mixture-of-Experts (MoE) models enable efficient scaling of large language model (LLM) inference but suffer from substantial data-movement overhead when deployed on neural processing unit (NPU)-based systems. Near-Data Processing (NDP) pro…
SMELT: Scaling Laws for Compute-Matched MoE Looped Transformers (arxiv.org) Looped Transformers increase effective depth by iterating a shared block of layers, but most evaluations compare at fixed model size, conflating architectural advantage with extra FLOPs. We study looping on Mixture-of-Experts Transformers…
PCoMoE: Shifting MoE Inference from Monolithic Expert Selection to Fine-Grained Path Composition (arxiv.org) Mixture-of-Experts (MoE) architectures scale Large Language Model (LLM) capacity efficiently by activating a sparse subset of experts per token. However, modern MoE inference remains heavily constrained by the rigid, whole-expert abstracti…
Instella-MoE Technical Report (arxiv.org) In this work, we introduce Instella-MoE, a fully open Mixture-of-Experts (MoE) language model with 16 billion total parameters and 2.8 billion active parameters per token, trained entirely from scratch on AMD Instinct MI300X and MI325X GPU…
Residual Sparsification via Output Importance for Compressing Mixture-of-Experts LLMs (arxiv.org) Mixture-of-experts (MoE) architectures scale large language models efficiently, but they demand massive GPU memory. To cope with such demand, models are commonly compressed to reduce their memory footprint.
Q-Strata: Hierarchical Bit Allocation for Mixed-Precision Quantization of Mixture-of-Experts LLMs (arxiv.org) Mixed-precision quantization (MPQ) assigns a different bitwidth to each linear layer of a large language model (LLM) to minimize the quantization-induced quality loss under a fixed budget, but Mixture-of-Experts (MoE) models contain these…
Structure Aware Neural Architecture Search for Mixture of Experts (arxiv.org) Neural Architecture Search (NAS) has so far rarely been applied to Mixture-of-Experts (MoE) models, and existing MoE designs leave the alignment between experts and the structure of the data to emerge on its own. We propose an architecture…
Unveiling Language Routing Isolation in Multilingual MoE Models for Interpretable Subnetwork Adaptation (arxiv.org) Mixture-of-Experts (MoE) models exhibit striking performance disparities across languages, yet the internal mechanisms driving these gaps remain poorly understood. In this work, we conduct a systematic analysis of expert routing patterns i…
CoRe-MoE: Compact Reusable MoE for Continual Multimodal Instruction Tuning (arxiv.org) Continual multimodal instruction tuning requires multimodal large language models to acquire new task abilities sequentially while preserving previously learned knowledge. LoRA-MoE provides a promising solution by introducing expert-based…
MM-Spectrum: Multimodal Multi-spectral Molecular Structural Elucidation with a Stable MoE Framework (arxiv.org) Inferring molecular structures from multimodal spectroscopic measurements requires integrating complementary yet highly heterogeneous signals. However, the common paradigm of directly concatenating multispectral sequences can exhibit anoma…
CRAM: Centroid-Routing and Adaptive MoE for Multimodal Continual Instruction Tuning (arxiv.org) Multimodal Large Language Models (MLLMs) unify heterogeneous vision-language tasks under a shared generative framework via instruction tuning, yet real-world deployment demands continuous capability expansion, making Multimodal Continual I…
Appreciation Post - thomsonreuters/Thomson-1.0-Small (www.reddit.com via reddit) With the lack of support from Qwen regarding the smaller 9B and 35B MOE models. Like myself, not everyone is looking for an agentic coding model, I particularly use it for RAG and reviewing and require high reasoning across different docum…
feat: import qwen4exp (Qwen3.8-Flash-Next) support from upstream PR #27742 by giveen · Pull Request #324 · TheTom/llama-cpp-turboquant (github.com via reddit) 44tks on a 5090 using Flash at Q4 and using ```--moe-cache auto```
Why can't we make MoE routers predict experts needed in the next 5-10 tokens? (www.reddit.com via reddit) Same as title. If we could do that can we potentially do expert caching from ram to vram so it's faster?
Hierarchical MoE for Multi-Modal ILD Diagnosis (arxiv.org) Mixture-of-experts (MoE) models combine specialized predictors under learned routing, offering a principled mechanism for leveraging heterogeneity in medical data. We present a hierarchical multimodal MoE for interstitial lung disease (ILD…
Ban&Pick: Enhancing Performance and Efficiency of MoE-LLMs via Smarter Routing (arxiv.org) Sparse Mixture-of-Experts (MoE) has become a key architecture for scaling large language models (LLMs) efficiently. Recent fine-grained MoE designs introduce hundreds of experts per layer, with multiple experts activated per token, enablin…
ExFold: Unified Expert Folding for Training-Free MoE Prefill-Decode Acceleration (arxiv.org) Mixture-of-Experts (MoE) models scale capacity for strong quality while keeping per-token compute bounded through sparse expert activation. Yet low-latency MoE serving is increasingly challenging, because it spans two inference phases with…
Output Dilution: Redundant but Fragile Representations in MoE Models (arxiv.org) Mixture-of-Experts (MoE) models appear to encode moral content as robustly as dense models, yet prove far more fragile in their encoding. In OLMoE-1B-7B, linear probes recover moral valence from nearly every expert-layer combination, with…
Groundhog Bit-Flip Attack: Seeding Infinite Generation Loops in Mixture-of-Experts LLMs through Bit Flips (arxiv.org) Mixture-of-Experts (MoE) architectures enable scalable and efficient large language models (LLMs) by selectively activating expert sub-networks through a routing mechanism. However, this adaptive design introduces a new attack surface: spe…
N-gram vs Experts explained (www.reddit.com via reddit) Since Qwen's dropped the Qwen4Exp architecture bomb that focus on offloading parameters to n-gram instead of pure mixture of experts, I dug into this and learned quite a lot. Here's the summary.
MIND BLOWING - QWEN3.8-FLASH-NEXT - RTX 3090 + 128 DDR5 +/-20 tok/s (www.reddit.comhttps) Qwen3.8 Flash Next just built this demo, and I have to say: this thing is seriously impressive. For a local model, the combination of coding ability, reasoning, and execution speed is kind of crazy.
Are you disappointed about Qwen3.8 Flash Next ? (www.reddit.com via reddit) We all need 35b a3b at least or 70b/80b MOE models, but instead we got 122b and not that huge than Qwen3.8 27b So what do you think ?
Strix 128 + eGPU 32 or 48, thoughts/concerns please (www.reddit.com via reddit) I have a strix halo, 128GB using GTT = 120GB usable = 105GB models and below, and this has served me well, but as the world changes I want to improve a bit. Your thoughts are welcomed !
Self-hosting LLMs on budget hardware: general principles, hardware, benchmarks and frontends (www.reddit.com via reddit) Hello, I've been self-hosting LLMs on various budget hardware for a while (6x RTX 3060 12 GB, Intel Arc Pro B60 24 GB, RX 9070 XT, etc). Over the last few months, I wrote about it in 4 articles: General principles Hardware and inference op…
How I see the future for local AI (www.reddit.com via reddit) I think that mall models between 10B and 40B are the real gamechanger. These models will be the ones that will make the AI buble pop and big companies like Anthropic and OpenAi go to hell.
A Heterogeneous Mixture of Experts Framework for Interpretable Machine Learning (arxiv.org) Mixture-of-Experts (MoE) models provide a flexible framework for partitioning complex prediction problems into simpler local learning tasks through an input-dependent gating mechanism. Existing interpretable MoE approaches, such as Mixture…
Qwen 3.6 35b a3b is slower on 7900xtx than on 3060ti on the same settings eveny using Vulkan? (www.reddit.com via reddit) why Qwen 3.6 35b a3b q4-k-m is slower on 7900xtx (20t\s 100% GPU Load) than on 3060ti (37t\s and GPU 50% Load) on the same settings? Linux llama.cpp vulkan 1.5Gb VRAM is empty.
Stop pretending every token needs access to the entire expert universe. (www.reddit.com via reddit) I still don’t understand why most MoE LLMs route every token independently. Apparently the current master plan is: - Build hundreds of experts.
Benchmarking Composable Compression Techniques in Mixture-of-Experts LLMs (arxiv.org) Mixture-of-Experts (MoE) LLMs scale model capacity efficiently through sparse activation, but their large expert parameter footprint, routing imbalance, and long-context KV-cache growth make deployment difficult on commodity hardware. Prac…
RASET: Router-Agnostic Safety-Critical Expert Tuning Exposes Localized Safety Enforcement Failures in Mixture-of-Experts LLMs (arxiv.org) Mixture-of-Experts (MoE) LLMs rely on sparse, router-driven expert activation, yet how safety alignment interacts with routed expert specialization remains underexplored. A common intuition is that safety behavior may be controlled by rout…
DF-MoE: Generalizable Deepfake Detection via Multimodal Sparse Mixture-of-Experts (arxiv.org) Audio-visual deepfake detection is an actively studied topic, where one of the main challenges is to develop detectors able to generalize across deepfake generation methods. We conjecture that overfitting can be mitigated by extracting mul…
SAEM: Stage-Aware Expert Management for Memory-Efficient MoE Inference in Chain-of-Thought Reasoning (arxiv.org) Chain-of-thought (CoT) prompting improves LLM reasoning by decomposing complex problems into intermediate steps, but its sequential nature increases decoding latency and memory usage. Mixture-of-Experts (MoE) models scale capacity through…
LayerStoRm: Run frontier-scale MoE LLMs on a handful of consumer GPUs by streaming experts over PCIe. (www.reddit.com via reddit) I am open sourcing an early release of LayerStoRm, a MoE LLM serving engine for limited VRAM multi-GPU systems, taking advantage of RAM and parallel PCI-E transfers. Currently tailored for my system with 2 x RTX5090 + 2 x RTX5080 and 512GB…
Spent a day seeing how far extreme MoE models can be pushed on a 4070 Ti + 32GB RAM. Kimi K3, DeepSeek V4 Flash, and Qwen3.5-122B results + research paper🔧 (www.reddit.com via reddit) I’ve been experimenting with a custom inference/runtime research project called CRANE V2, mostly because I wanted to answer a stupid question: How far can you push absurdly large MoE models on an ordinary consumer Windows machine before ph…
Best 64k-128k models/fine-tunes on a single 3090 for PRD planning & ticket creation? (www.reddit.com via reddit) Hey guys, I'm setting up a local workflow on a single 24GB RTX 3090 to handle project planning—specifically digesting massive (~128k context) requirements documents/PRDs and spitting out a ton of structured .md files to act like Jira ticke…
MobileMoE - a facebook Collection (huggingface.co via reddit) MobileMoE is a family of on-device Mixture-of-Experts (MoE) language models with sub-billion active parameters, designed to push the quality–efficiency Pareto frontier for on-device LLMs, including three model scales (S/M/L): 0.3B/0.5B/0.9…
TielCoder's 22 GB 4-bit quant matches Opus4.6 medium on recent real life coding issues, surpassing KAT-Coder and Nail as strongest and fastest MoE picks. (www.reddit.comhttps) Qwen3.8-27B is amazing, but it’s slow. A stronger 35B-A3B Mixture of Experts-coder that can run and solve real codebase issues fast (even on constrained hardware) is a valuable addition to the arsenal.
deepseek-v4-flash-0731 - surprisingly usable (www.reddit.com via reddit) I just finished building my (relatively) low rent local inference machine: * Epyc 7663 * 256GB ECC DDR4-3200 * 1x RTX 5090 32GB Yeah I realize it's weird to throw a 5090 and 256GB of anything together and call it low end, but relative to ~…
DAOP: Data-Aware Offloading and Predictive Pre-Calculation for Efficient MoE Inference (arxiv.org) Mixture-of-Experts (MoE) models, though highly effective for various machine learning tasks, face significant deployment challenges on memory-constrained devices. While GPUs offer fast inference, their limited memory compared to CPUs means…
RouteScan: A Non-Intrusive Approach to Auditing MoE LLMs Safety via Expert Routing Telemetry (arxiv.org) As Mixture-of-Experts (MoE) architectures are increasingly adopted for scaling Large Language Models (LLMs), safety auditing becomes necessary to verify whether these models produce or facilitate harmful behaviors during operation. However…
Fuzzy-MoE: Interpretable Regime-Conditioned Expert Routing for Non-Stationary Multivariate Time Series Forecasting (arxiv.org) In non-stationary multivariate time series, different variables and samples often exhibit heterogeneous latent dynamic states, while existing deep forecasting models usually compress them into a unified end-to-end mapping, leading to subop…
For those offloading MoE models to Ram... is prefill very slow? (www.reddit.com via reddit) I was seeing some decent decode speeds in people experimenting with offloading MoE models to ram when having small Graphic cards. But reading a bit I see prefill is very very slow.
ConvRot Quant method now in llama-cpp-turboquant (www.reddit.com via reddit) It started here , and now https://github.com/TheTom/llama-cpp-turboquant/ has it. Imagine a Q6 quant with nearly Q8 KLD/PPL.
Models for planing and coding (www.reddit.com via reddit) Hi, I am a hobby dev using currently qwen 3.8 27b on my strix halo machine for coding. I was wondering what is the best approach to speed up.
Crowd-funding new open-weight models? (www.reddit.com via reddit) There’s always lots of people asking for various specific new local models… I wonder if we could incentivize a specific model (e.g. Qwen3.8 35B MoE) via crowdfunding on Kickstarter or something.
AMD Users: Have you tried the llamma.cpp AMD-Ecosystem branch? Up to 2x PP Speed (www.reddit.com via reddit) AMD has it's own llama.cpp branch: https://github.com/AMD-Ecosystem/llama.cpp And despite the Deprecation warning it's actively maintained (things are later upstreamed to the normal llama.cpp). What i noticed with my Strix Halo: It has som…
Why does t/s go down as offload more to egpu? (www.reddit.com via reddit) For moe models like Qwen3.6 35B A3B, some setups keep experts on the cpu in order to have the active weights in the gpu. But I have a strix halo + r9700, so I wondered what if I kept the experts on the strix halo and active weights on the…
Why is using harness slower than normal chatting? (www.reddit.com via reddit) When I have a normal conversation with my Qwen3.6 35B A3B I get ~27t/s. I noticed that when I use a harness sometimes it can be slow as 15t/s (tested OpenCode and Maki), with way lower context than my normal chats.
Need support for llama.cpp with multi GPU (www.reddit.com via reddit) Using llama.cpp I seem to be unable to get my to GPUs working tougether correclty, so I need help somehow. Setup: 96GB RAM, one Blackwell 5000 (48GB) and one 3090 (24GB).
If you want to upgrade from Qwen3.6 27B, but dislike 3.8, give Ornith-1.5-35B-A3B a try. (www.reddit.com via reddit) Just make sure to leave MTP off. It's a decent upgrade from Qwen3.6 27B, and better than KAT Coder V2.5 Dev (which also requires their custom vLLM).
error en modelo ling 3.0 flash moe 124B a5B (www.reddit.com via reddit) what is this error? some person know how this errors was?
Beyond the Trace: Coupling an Interpretable Reasoning-State Readout to Native MoE Routing (arxiv.org) What a reasoning model writes is only a partial record of the process that produces it. We introduce a two-level internal readout for mixture-of-experts reasoning.
Mixture of experts architectures for machine learning interatomic potentials (arxiv.org) Machine Learning Interatomic Potentials (MLIPs) enable accurate large-scale atomistic simulations, yet improving their expressive capacity efficiently remains challenging. Here we systematically investigate Mixture-of-Experts (MoE) and Mix…
Every Expert Counts: ExactMoE for Memory-Efficient W4A16 Inference (arxiv.org) Sparse mixture-of-experts (MoE) language models reduce arithmetic by activating only a small subset of experts per token, yet deployment still requires storing and moving the full expert bank. We present ExactMoE, an inference design that…
A Unified Mamba--MoE Surrogate for Closed-Loop Simulation and Measurement-Window Forecasting of Inverter Transients (arxiv.org) This paper proposes a Mamba surrogate model with mixture-of-experts (MoE) routing to represent the transient dynamics of inverter-based resources. A Mamba surrogate model is a predictive machine learning model built on the Mamba architectu…
MoE-Enhanced Explainable Deep Manifold Transformation for Complex Data Embedding and Visualization (arxiv.org) Dimensionality reduction (DR) plays a crucial role in various fields, including data engineering and visualization, by simplifying complex datasets while retaining essential information. However, achieving both high DR accuracy and strong…
Routing Divergence Is Not Evidence of Behavioral Influence in Same-Weight MoE Self-Distillation (arxiv.org) Two Mixture-of-Experts (MoE) forward passes can share every weight yet route the same token through different experts. This creates a possible blind spot in same-weight self-distillation, where a demonstration-conditioned teacher supervise…
MAPLE: MoE Adaptive Plug-and-play Layer-wise Expert allocation (arxiv.org) Sparsely-activated Mixture-of-Experts (MoE) Transformers universally fix the same number of routed experts across all layers, a convention that ignores the well-documented heterogeneity in layer-wise redundancy. We demonstrate that this un…
THESIS-MoE: Trainable Hierarchical Extraction and SteerIng of Sycophancy in Mixture-of-Experts (arxiv.org) Sycophancy, the tendency of a language model to change its answer to match a user's stated belief, is a common alignment failure. Existing activation steering methods typically apply a single contrastive direction uniformly throughout the…
MoE Router-Guided Clustering for Heterogeneous Federated Instruction Tuning (arxiv.org) Federated instruction fine-tuning enables Large Language Models (LLMs) to adapt to decentralized, privacy-sensitive data without requiring data sharing. Recent Mixture-of-Experts (MoE) LLMs are particularly attractive for federated learnin…
S2-MoE: Enabling Efficient Self-Speculative Decoding for Mixture-of-Experts on Edge Devices (arxiv.org) Deploying large language models (LLMs) for inference on edge devices is challenging due to severe memory and bandwidth constraints. While speculative decoding and Mixture-of-Experts (MoE) have been proposed to improve inference efficiency,…
DeaMoE: Efficient MoE Structure for Fast Small-Batch Decoding (arxiv.org) Mixture-of-Experts (MoE) models have been widely adopted in real-time interactive applications such as coding assistants, real-time audio-video interaction systems. To meet the extremely low response latency requirements of these scenarios…
FreeBalance: Pre-Routing Online Moe Load Balancing via Residual Workload Prediction (arxiv.org) Load imbalance poses a major bottleneck to the efficiency of expert parallelism in distributed inference of Mixture-of-Experts (MoE) models. The most heavily loaded rank stalls global execution due to skewed routing distributions, directly…
HybridSB-MoE: Dual-Domain Schr\"odinger Bridges with Scene-Adaptive Expert Routing for Speech Enhancement (arxiv.org) Generative speech enhancement faces three gaps: spectral models capture harmonic structure but often disrupt phase, waveform models preserve phase but miss harmonics, and Schrödinger Bridges (SB) shorten transport from noise to clean speec…
RoutePack: Expert Placement and Attention-Aware Data Packing for MoE Reinforcement Learning (arxiv.org) Training Mixture-of-Experts (MoE) models for reinforcement learning (RL) couples two load-balancing problems: sequence composition determines dense attention work in each data-parallel microbatch, while token routing determines sparse expe…
APEX: Adaptive Expert Prefetching for Memory-Efficient Edge MoE Inference (arxiv.org) Mixture-of-Experts (MoE) models are attractive for edge deployment because they provide high model capacity while activating only a small subset of parameters per token, improving compute efficiency. However, MoE inference at the edge is f…
MoE Proxy Models for Low-Cost Failure Reproduction and Diagnosis in LLM RL Post-Training (arxiv.org) Reinforcement learning (RL) post-training of large language models (LLMs) is computationally intensive and involves complex system pipelines with substantial debugging overhead. In practice, factors such as framework adaptation, numerical…
Share First, Route What Remains: A Unified Framework for Token-Adaptive MoE Computation (arxiv.org) Mixture-of-experts (MoE) models have recently moved beyond routing a fixed number of complete experts. Shared-expert designs preserve reusable knowledge, fine-grained methods vary computation within experts, and dynamic routers adapt the n…
RotaryQuant: Fitting 120B MoE Models on Consumer Hardware via Fused Compressed-Space Attention (arxiv.org) Large mixture-of-experts (MoE) language models with 26--120 billion parameters exceed the memory capacity of consumer devices through three simultaneous pressures: resident weight matrices, key-value (KV) cache state that grows linearly wi…
When Does Trace-Driven Evaluation Mislead MoE Expert Caching? Replay Semantics, Workload Contamination, and Operating Regimes (arxiv.org) Mixture-of-Experts (MoE) models have outgrown accelerator memory, and offloading expert weights to host memory is now standard. This makes expert cache management an attractive lever: a policy that raised the hit rate would cut expert traf…
MoE-Prism: Disentangling Monolithic Experts for Elastic MoE Services via Model-System Co-Designs (arxiv.org) Mixture-of-Experts (MoE) scales model capacity through sparse activation, and is becoming an important architecture for large language models (LLMs). However, existing MoE serving systems typically execute all requests under a fixed routin…
EasyBalance: Cross-Layer Load Balancing in Distributed MoE Inference (arxiv.org) Load Balancing has emerged as a critical problem in expert-parallel distributed inference of Mixture-of-Experts (MoE) models. As routing distributions are typically skewed across experts, devices hosting lighter-loaded experts must idle to…
Router Sensitivity Under Lightweight Fine-Tuning Identifies Prunable Experts in Mixture-of-Experts Models (arxiv.org) Mixture-of-Experts (MoE) models decouple total parameters from per-token compute, but deployment still requires storing every expert. Recent theory shows that pruning experts with the smallest router-norm changes during fine-tuning can pre…
UniMoMo: Expert Merging-Based MoE Acceleration for Large Recommendation Models (arxiv.org) Sparse mixture-of-experts (MoE) layers expand recommendation capacity through conditional computation, yet a trained checkpoint still stores and routes over its full expert bank. We study a deployment problem: convert that checkpoint to a…
Decided Upstream, Written Late: Locating and Pricing the Cross-Lingual Refusal Circuit of a Multilingual MoE (arxiv.org) Safety alignment in multilingual models is uneven: a model that reliably refuses a harmful request in English will often comply with the same request in a lower-resource language. We trace this gap mechanistically in sarvam, an Indic-multi…
Policy-Masked Private Experts: Auditable and Reversible Capability Access Control in Sparse MoE Models (arxiv.org) Most language-model access controls regulate behavior while leaving the same computation available to every request. We study a different systems question: can trusted authorization determine which newly trained parameters are reachable by…
TEXAS: Task-Expert-Aware Supervision for Downstream Mixture-of-Experts LLM Adaptation (arxiv.org) Mixture-of-Experts (MoE) language models route each token through a small subset of experts, making routing patterns useful for identifying task-relevant experts during downstream adaptation. Yet current approaches have two limitations: ta…
CP-MoE: Consistency-Preserving Mixture-of-Experts for Continual Learning (arxiv.org) Catastrophic forgetting remains a major obstacle to continual learning in large language models (LLMs) and vision--language models (VLMs). Although Mixture-of-Experts (MoE) architectures offer an efficient path to scaling, existing LoRA-ba…
Beyond Global Routing Aggregation: Phase-Aware Expert Merging for MoE Vision-Language Models (arxiv.org) Mixture-of-experts vision-language models (MoE-VLMs) increase model capacity with sparse expert activation, yet deployment requires storing the full expert pool. Training-free expert merging reduces this burden, and many routing-based meth…
Elbow-Based MoE Routing: A Training-Free Inference Time Plugin for Expert Selection (arxiv.org) Mixture-of-Experts (MoE) models enable model scaling while maintaining low inference-time compute by activating only a subset of experts per token. However, conventional routing relies on a fixed top-k selection, forcing the model to spend…
AFD-Ledger: Deployment Provisioning for Attention--FFN Disaggregation (arxiv.org) Attention--Feed-Forward Network (FFN) Disaggregation (AFD) is emerging as a promising architecture for serving Mixture-of-Experts (MoE) language models. While existing AFD systems improve the efficiency of disaggregated execution, they lea…
C$^2$MOE: Consistency and Complementarity-guided Mixture of Experts for Incomplete Multimodal Emotion Learning (arxiv.org) Recent advances in Multimodal Emotion Recognition in Conversations (MERC) highlight its reliance on complete multimodal inputs. However, real-world data often suffer from missing modalities due to transmission errors or user behavior, seve…
LLaDA MoE v2: Scaling Mixture-of-Experts Diffusion Language Models (arxiv.org) Diffusion language models (dLLMs) offer an alternative to autoregressive (AR) language modeling, yet the scaling behavior of Mixture-of-Experts (MoE) dLLMs remains poorly understood. We systematically characterize how optimization hyperpar…
AcceptMoE: Commitment-Weighted Self-Sizing Verifier Expert Sets for Efficient MoE Speculative Decoding (arxiv.org) Speculative decoding verifies a tree of draft tokens in one target-model forward pass. For a mixture-of-experts (MoE) target, however, parallel verification can activate the union of the experts selected by all tree nodes, even though only…
Tevatron Meets Megatron: Expert-Parallel LLM Reranker Training on an Academic Budget (arxiv.org) Modern reranking recipes---billion-scale cross-encoders, mixture-of-experts (MoE) backbones, and distillation against strong teachers---have outpaced the training infrastructure available to most academic groups. Existing Tevatron reranker…
TEAM: Temporal-Spatial Consistency Guided Expert Activation for MoE Diffusion Language Model Acceleration (arxiv.org) Diffusion large language models (dLLMs) have recently gained significant attention due to their inherent support for parallel decoding. Building on this paradigm, Mixture-of-Experts (MoE) dLLMs with autoregressive (AR) initialization have…
HetRoute Heterogeneous and Cost-aware Collaborative Routing Framework for Distributed Edge MoE Inference (arxiv.org) Mixture-of-Experts (MoE) models have become a dominant architecture for large-scale AI services, yet deploying them over geo-distributed heterogeneous edge servers remains challenging. When the Top-k activated experts of a token are spread…
TrimMoE A communication aware and adaptive depth framework for distributed edge inference (arxiv.org) Serving Mixture-of-Experts (MoE) large language models across distributed edge servers is bottlenecked by the cross-server expert transmission. The existing approaches mainly focus on how to reach a remote expert faster.
REFLEX: Rethinking MoE Inference as Refinement-Aware Compute Allocation in Diffusion Language Models (arxiv.org) Mixture-of-experts (MoE) models increase parameter capacity by activating only a small subset of experts for each token. This conditional-computation paradigm has enabled autoregressive language models to scale model capacity without a pro…
vLLM for Baidu Kunlun (github.com) 📖 Documentation | 🚀 Quick Start | 📦 Installation | 💬 Slack Latest News 🔥 [2026/07] 🚧 v0.25.1 under development — Added Qwen3.5 / Qwen3.5-MoE, Gemma4 (text and multimodal), GLM MoE DSA, and DFlash speculative decoding [2026/02] ⚡ Performanc…
From Expert Reduction to Behavioral Divergence: Tracing Numerical State through Sparse MoE Inference (arxiv.org) Mathematically equivalent expert-reduction orders can produce observably different sparse-MoE executions. We isolate this effect in native DeepSeek-V4-Flash by freezing local MoE state and varying only aggregation semantics.
TIER-MoE: Trust-Informed Expert Routing via Conditional Modality Risk for Multimodal Fusion in Biomedical Classification (arxiv.org) The promise of multimodal fusion lies in combining complementary sources of evidence, yet more evidence does not always yield a better prediction. Recent multimodal models have advanced fusion through richer cross-modal interaction and sam…
Incast-Free MoE Rate-Based Scheduling (arxiv.org) Mixture of Experts (MoE) architectures have become key to large language models; however, their typical round-robin (RR) scheduling introduces significant bottlenecks. In this paper, we demonstrate that RR causes a previously-undiscovered…
FedWeave: Rethinking the Unit of Specialization in Heterogeneous Federated MoE-LoRA (arxiv.org) Federated PEFT enables LLMs to collaboratively adapt to decentralized private data without sharing raw examples. However, task heterogeneity across clients can cause cross-task interference and gradient conflicts during aggregation.
SpecPrefetch: Parameter-Efficient Expert Prefetching for Sparse MoE Foundation Models (arxiv.org) Sparse Mixture-of-Experts (MoE) models expand foundation model capacity through conditional expert activation, but their full expert pools remain difficult to deploy under limited accelerator memory. Although expert offloading alleviates m…
DraftExpert: Expansion-Aware Self-Speculative Decoding for End-Device MoE Inference (arxiv.org) Large Mixture-of-Experts (MoE) language models are attractive for end-device deployment because only a small subset of experts is active per token, but their routed expert weights often exceed accelerator memory. We target latency-critical…
LIBMoE: A Library for comprehensive benchmarking Mixture of Experts in Large Language Models (arxiv.org) Mixture of experts (MoE) architectures have become a cornerstone for scaling up and are a key component in most large language models such as GPT-OSS, DeepSeek-V3, Llama-4, and Gemini-2.5. However, systematic research on MoE remains severe…
MoE$^2$-LoRA: When MoE Models Meet MoE-style Low-Rank Adaptation (arxiv.org) Mixture-of-Experts (MoE) architectures have been widely adopted in large language models, yet parameter-efficient fine-tuning (PEFT) for MoE models remains underexplored. Existing PEFT methods for MoE either ignore router priors with unifo…
PreMoE: Proactive Inference for Efficient Mixture-of-Experts (arxiv.org) Mixture-of-Experts (MoE) models offer dynamic computation, but are typically deployed as static full-capacity models, missing opportunities for deployment-specific specialization. We introduce PreMoE, a training-free framework that proacti…
Is MoE Routing a Huffman Code? Discovering the Frequency-Diversity Law in Chain-of-Thought (arxiv.org) Mixture-of-Experts architectures have revolutionized scaling, yet the underlying logic of their routing remains a black box. In this paper, we uncover a fundamental governing principle: MoE routing is not merely selection, but a manifestat…
Knowledge Injection Exists in MoE? Exploring Expert-Aware Contrast Decoding in MoE for Mitigating LLMs'Hallucinations (arxiv.org) Existing LLM hallucination mitigation methods, including prompt engineering and model optimization, either hardly alter models'internal knowledge or have poor cross-domain generalization. Contrastive decoding mitigates hallucinations by us…
Beyond Independent Optimization: Compression, MoE Routing, and Quantization Interactions in Multimodal Edge Intelligence (arxiv.org) Efficient multimodal inference is increasingly constrained not only by model quality or FLOP count, but also by the cost of preserving, moving, routing, caching, and quantizing multimodal representations under latency, memory, and energy c…
SLAI T-Rex: Full-Parameter Post-training of the DeepSeek-V4 Family on Ascend SuperPOD (arxiv.org) Full-parameter post-training of trillion-parameter-scale MoE models introduces substantial system-level challenges for large-scale distributed training, including severe memory pressure, non-overlapped communication overhead, and inefficie…
↯ DeepSeek 4↯ DeepSeek 4↯ DeepSeek 4↯ DeepSeek 4↯ DeepSeek 4↯ DeepSeek 4moedeepseek
Breaking the MoE LLM Trilemma: Dynamic Expert Clustering with Structured Compression (arxiv.org) Mixture-of-Experts (MoE) Large Language Models (LLMs) face a trilemma of load imbalance, parameter redundancy, and communication overhead. We introduce a unified framework based on dynamic expert clustering and structured compression to ad…
OrderMoE: An expert similarity driven distributed edge MoE inference (arxiv.org) Although mixture-of-experts, MoE, models have been increasingly adopted to scale large language models with moderate computation cost, it remains challenging to deploy MoE inference over resource-constrained and bandwidth-limited edge infr…
Half the Experts, All the Code: One-Shot Domain Pruning of Mixture-of-Experts LLMs for Coding (arxiv.org) The strongest open-weight coding models are mixture-of-experts (MoE) networks: most of their size comes from large pools of "expert" subnetworks, of which only a few act on any token. That pool is why these models do not fit on the machine…
ThAME: 3D Memory-Enabled Heterogeneous Accelerator for LLM Mixture of Experts (arxiv.org) Mixture of Experts (MoE) architectures have emerged as a dominant paradigm for scaling Large Language Models (LLMs). However, MoE inference on conventional hardware is constrained by three fundamental bottlenecks.
LookME: Lookup-Based Multimodal Embeddings for Layer Injection in Vision-Language Models (arxiv.org) Vision-Language Models (VLMs) have achieved strong progress in multimodal understanding. However, scaling dense or sparse Mixture-of-Experts (MoE) models to improve performance limits deployment in resource-constrained environments due to…
PagedWeight: Efficient MoE LLM Serving with Dynamic Quality-Aware Weight Quantization (arxiv.org) Mixture-of-Experts (MoE) is a popular class of large language models (LLMs), offering high efficiency and accuracy. However, in KV-cache-intensive serving scenarios, MoEs often exhibit a tension between the GPU memory requirements of the m…
QUADS: Stabilizing NVFP4 Reinforcement Learning for MoE via QUantization-error Alignment across Dual Sides (arxiv.org) Rollout generation is a major bottleneck in Reinforcement Learning (RL) for Mixture-of-Experts (MoE) Large Language Models, motivating low-precision rollout acceleration such as FP8. As an emerging low-precision format, NVFP4 combines fine…
PASs-MoE: Mitigating Misaligned Co-drift among Router and Experts via Pathway Activation Subspaces for Continual Learning (arxiv.org) Continual instruction tuning (CIT) requires multimodal large language models (MLLMs) to adapt to a stream of tasks without forgetting prior capabilities. A common strategy is to isolate updates by routing inputs to different LoRA experts.
A 7B MoE at ~25 tok/s on a free Oracle ARM box — live in your browser. My llama.cpp-based CPU server, and the whole hosting stack is €0. (www.reddit.comhttps) Maintainer here. A live, in-browser demo of Reame, the CPU-first inference server I've been building on llama.cpp — and I think the numbers are fun for this sub.
Prism ternary 27b, how is it? (www.reddit.com via reddit) Is it actually comparable to the qwen3.6 27b? Can it be used for GPU poor?
Full-Pipeline Inference Optimization for MiMo-V2.5 Series: Pushing Hybrid SWA Efficiency to the Limit (arxiv.org) We present a full-pipeline inference optimization for the MiMo-V2.5 model family, which combines Hybrid Sliding Window Attention (Hybrid SWA), sparse Mixture-of-Experts (MoE), and multimodal encoders. While Hybrid SWA can ideally reduce bo…
MawForge: Memory-Bounded Expert Materialization for Local Mixture-of-Experts Inference (arxiv.org) Sparse Mixture-of-Experts (MoE) language models separate total parameter count from per-token active computation, but local inference systems often still require the full model, key-value cache, runtime buffers, and operatingsystem headroo…
HCRMap: Pressure-Aware Hot-Expert Residency Mapping for 3.5D MoE Chiplet Inference (arxiv.org) Mixture-of-Experts (MoE) large language models (LLM) activate only a small number of experts during inference, but token routing introduces persistent expert hotness skew: a small set of hot experts continuously receives most tokens, while…
Director: Accelerating Distributed MoE Serving via Online Proactive Expert Placement (arxiv.org) Expert parallelism has become the prevailing paradigm to serve Mixture-of-Experts (MoE) models. Its efficiency depends on the communication and computation latencies of the GPUs, which are linked to the placement of experts in the GPUs.
Sticky Routing: Training MoE Models for Memory-Efficient Inference (arxiv.org) Mixture-of-Experts (MoE) models activate only a sparse subset of experts per token, yet consecutive tokens frequently activate different experts -- causing constant weight swapping between slow storage and fast memory on edge devices. Exis…
GLM-5.2 (744B MoE) on a 25GB-RAM consumer machine (github.com via reddit) Tiny engine, immense model. Run GLM-5.2 (744B-parameter MoE) on a consumer machine with ~25 GB of RAM — in pure C, with zero dependencies, by streaming experts from disk.
SpaR3D-MoE: Adaptive 3D Spatial Reasoning from Sparse Views Meets Geometry-Inductive Mixture-of-Experts (arxiv.org) Recent Multimodal Large Language Models (MLLMs) struggle to bridge the representational gap between 2D semantic understanding and 3D spatial geometry. Existing 3D-aware models either rely on costly 3D-specific data or utilize RGB-only inpu…
TriRoute: Unified Learned Routing for Joint Adaptive Attention, Experts, and KV-Cache Allocation (arxiv.org) Conditional computation can decouple language model quality from per-token inference cost, yet leading techniques act on a single axis in isolation: Mixture-of-Experts (MoE) sparsifies the FFN, Mixture-of-Depths (MoD) skips whole transform…
WPG-MoE: Weak-Prior-Guided Dense Mixture-of-Experts for User-Level Social Media Depression Detection (arxiv.org) Online social media posts provide scalable signals for early depression screening, and recent studies mainly improve pre-classification evidence through risk-post selection, symptom grounding, and clinically informed feature construction.…
PuzzleMoE: Efficient Compression of Large Mixture-of-Experts Models via Sparse Expert Merging and Bit-packed inference (arxiv.org) Mixture-of-Experts (MoE) models have shown strong potential in scaling language models efficiently by activating only a small subset of experts per input. However, their widespread deployment remains limited due to the high memory overhead…
Localized LoRA-MoE: Block-wise Low-Rank Experts With Adaptive Routing (arxiv.org) Large Language Models (LLMs) and high-dimensional perception networks increasingly rely on parameter-efficient fine-tuning (PEFT) to adapt to diverse operational contexts. However, standard methods like LoRA are structurally limited by a m…
Worldscape-MoE: A Unified Mixture-of-Experts World Model for Scalable Heterogeneous Action Control (arxiv.org) World models are rapidly becoming a core infrastructure for embodied intelligence and interactive agents: they provide controllable simulators in which agents can perceive, act, forecast, and acquire scalable experience. Yet current video…
Nemotron-Labs-3-Puzzle-75B-A9B: Compressing Hybrid MoE LLMs (arxiv.org) We present Nemotron-Labs-3-Puzzle-75B-A9B, a compressed variant of Nemotron-3-Super optimized for interactive deployment. We designed the model to maximize server throughput under high user throughput constraints.
EPnG: Adaptive Expert Prune-and-Grow for Parameter-Efficient MoE Fine-tuning (arxiv.org) Mixture-of-Experts (MoE) models scale efficiently but remain costly to adapt due to redundant experts and uniform parameter allocation. Existing parameter-efficient fine-tuning (PEFT) methods such as LoRA ignore MoE routing dynamics, leadi…
OmniMoE: An Efficient MoE by Orchestrating Atomic Experts at Scale (arxiv.org) Mixture-of-Experts (MoE) architectures are evolving towards finer granularity to improve parameter efficiency. However, existing MoE designs face an inherent trade-off between the granularity of expert specialization and hardware execution…
Multi-scale Mixture of World Models for Embodied Agents in Evolving Environments (arxiv.org) Embodied agents operating in the real world require multi-scale reasoning and knowledge adaptation as conditions change. We identify two challenges in applying Mixture of Experts (MoE) to this setting: routing lacks an explicit notion of s…
TF-MoE: Time-Frequency Mixture-of-Experts for Efficient Speech Separation (arxiv.org) Diff-MN: Diffusion Parameterized MoE-NCDE for Continuous Time Series Generation with Irregular Observations (arxiv.org) Time series generation (TSG) is widely used across domains, yet most existing methods assume regular sampling and fixed output resolutions. These assumptions are often violated in practice, where observations are irregular and sparse, whil…
Symbiotic-MoE: Unlocking the Synergy between Generation and Understanding (arxiv.org) Empowering Large Multimodal Models (LMMs) with image generation often leads to catastrophic forgetting in understanding tasks due to severe gradient conflicts. While existing paradigms like Mixture-of-Transformers (MoT) mitigate this confl…
DAIN: Dynamic Agent-Based Interaction Network for Efficient and Collaborative Multimodal Reasoning (arxiv.org) Current multimodal fusion approaches, particularly those based on static Mixture-of-Experts (MoE) architectures, often struggle to provide the adaptive and efficient collaborative reasoning required by complex real-world applications. We i…
FlexMoE: One-for-All Nested Intra-Expert Pruning for MoE Language Models (arxiv.org) Mixture-of-Experts (MoE) language models scale model ability with sparsely activated experts, making this architecture a standard recipe for modern large models. However, sparse activation does not remove the deployment burden of storing a…
Ornith-1.0 released on Hugging Face (www.reddit.com via reddit) Including 9B Dense, 31B Dense, 35B MoE, and 397B MoE and reporting sota on different benchmark (let's see if this holds). https://huggingface.co/collections/deepreinforce-ai/ornith-10
SARA: Unlocking Multilingual Knowledge in Mixture-of-Experts via Semantically Anchored Routing Alignment (arxiv.org) Sparse Mixture-of-Experts (MoE) architectures have emerged as an increasingly influential paradigm as they offer a strategic balance between parameter scalability and computational efficiency. However, low-resource languages, which suffer…
CrossPool: Efficient Multi-LLM Serving for Cold MoE Models through KV-Cache and Weight Disaggregation (arxiv.org) Emerging LLM services increasingly host many sparse MoE models, yet most models receive sparse requests and remain cold. This creates a GPU memory problem: model weights are stable and model-determined, while KV-cache is transient and dema…
Does the Same Token Mean the Same State? MoE Routing as Signal for Reasoning Control (arxiv.org) In sparse Mixture-of-Experts language models, does the same token id imply the same router state and the same experts producing it? Holding the emitted token id fixed at repeated anchors, we find it does not: the experts that produce it st…
Symmetry-Compatible Principle for Optimizer Design: Embeddings, LM Heads, SwiGLU MLPs, and MoE Routers (arxiv.org) A striking geometric disparity has long persisted in the practice of deep learning. While modern neural network architectures naturally exhibit rich symmetry and equivariance properties, popular optimizers such as Adam and its variants ope…
Does Mixture-of-Experts Actually Help Inference on Consumer and Edge Hardware? An Empirical Study (arxiv.org) Mixture-of-Experts (MoE) language models are often described as ideal for resource-constrained inference. Each token activates only a small subset of experts, so the per-token compute cost, in floating-point operations (FLOPs), resembles t…
CTS-MoE: Implicit Terrain Adaptation via Mixture-of-Experts for Perceptive Locomotion (arxiv.org) Perceptive legged locomotion over discontinuous terrain (e.g., stairs, gaps, and obstacles) requires adaptive behavior, as a single conservative gait cannot produce the anticipatory maneuvers needed for abrupt topology changes. Cast as mul…
DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence (arxiv.org) We present a preview version of DeepSeek-V4 series, including two strong Mixture-of-Experts (MoE) language models -- DeepSeek-V4-Pro with 1.6T parameters (49B activated) and DeepSeek-V4-Flash with 284B parameters (13B activated) -- both su…
Attribution-Guided and Coverage-Maximized Pruning for Structural MoE Compression (arxiv.org) Mixture-of-Experts (MoE) models scale compute efficiently, yet remain expensive to deploy due to their substantial memory footprint and inference overhead. Prior compression methods mainly operate at the expert level, either removing entir…
Kimi K2.7 Code: 1T MoE, $0.95/M tokens, MIT license, beats Opus 4.8 on MCP tool-calling (www.reddit.com via reddit) Moonshot AI released Kimi K2.7 Code on June 12 — a coding-focused open-weight model. Key specs: - 1 trillion params (MoE, 32B active, 384 experts) - 256K context window - Modified MIT license — weights on Hugging Face - $0.95/M input, $4.0…
AIMER: Calibration-Free Task-Agnostic MoE Expert Pruning (arxiv.org) Mixture-of-Experts (MoE) language models increase parameter capacity without proportional per-token computation, yet deployment still requires storing the full expert pool, making expert pruning important for reducing memory and serving ov…
SoftMoE: Soft Differentiable Routing for Mixture-of-Experts in LLMs (arxiv.org) Sparse Mixture-of-Experts (MoE) architectures enable scaling LLM parameters under a fixed inference budget by activating only a small subset of experts via top-$k$ routing. While this preserves causality and suits autoregressive language m…
MODE: Modality-Decomposed Expert-Level Mixed-Precision Quantization for MoE Multimodal LLMs (arxiv.org) Mixture-of-Experts Multimodal Large Language Models (MoE-MLLMs) offer remarkable performance but incur prohibitive GPU memory costs, making compression essential. Among PTQ methods, expert-level mixed-precision quantization has proven effe…
A Spatio-Temporal Expert Prefetching Framework for Efficient MoE-based LLM Inference (arxiv.org) Mixture-of-Experts (MoE) based large language models (LLMs), such as Qwen and DeepSeek, have recently emerged as an effective approach to improving model capacity without proportionally increasing computational cost. By replacing the conve…
How to Score Experts for One-Shot MoE Expert Pruning: A Unified Formulation and Selection Principle (arxiv.org) Mixture-of-Experts (MoE) language models reduce per-token computation through sparse expert activation, yet deployment still requires storing the full expert pool, making one-shot expert pruning a practical approach for reducing memory usa…
SPRI: SVD-Partitioned Residual Initialization for Data-Constrained MoE Upcycling (arxiv.org) Mixture-of-Experts (MoE) models enable efficient scaling, but training them from scratch remains prohibitively expensive. MoE upcycling mitigates this cost by converting pretrained dense models into sparse MoE models.
250+ t/s on a single RTX 5090: Running Google’s DiffusionGemma MoE locally via llama.cpp (www.reddit.com via reddit) So Google’s new DiffusionGemma-26B-A4B-it is pretty wild. It's a discrete text diffusion model, which means instead of generating one token at a time autoregressively, it predicts and refines a whole block of up to 256 tokens in parallel.
I have finally tested it : large models can be run on low RAM / no VRAM (www.reddit.com via reddit) I was not sure myself, seeing a lot of statements here and around like "you need XXX VRAM / Unified Memory to run this model". So today I finally tested it.
Reviewing speed optimizations on llamacpp for large MoE models on multiGPU rigs? (fitparams vs -ngl/-ncmoe vs other flags, P2P, overclocking) (www.reddit.com via reddit) In anticipation of MiniMax reported upcoming open-weight release of M3, wanted to do comprehensive review of what I’m aware of regarding speed optimizations. Hopefully it can be helpful reference for some people too.
Buy recommendations on a thight Budget to aid my RX 6800 (www.reddit.com via reddit) So after a few hours of reserach, im torn between getting either a radeon vii or 2 p100 (both options for roughly 240€). The Radeon would give me 32gb of vram and fast inferference, while the 2 p100 would give me a total of 48gb, but rough…
Are older Titan cards still viable? (www.reddit.com via reddit) Looking at older Nvidia cards under £200 for Gemma/Qwen MOE coding. Is there any reason to avoid older Titan 12GB cards other than being power hungry?
nvidia/diffusiongemma-26B-A4B-it-NVFP4 · Hugging Face (huggingface.co via reddit) Model Overview Description: DiffusionGemma 26B A4B IT is an open-weights multimodal generative model developed by Google DeepMind that processes text, image, and video inputs to produce text output via discrete diffusion. Built on the Gemm…
↯ Deepmind↯ Gemma↯ Function Calling↯ Gemma 4function-callingdeepmindgemma+1
Diffusion Gemma 26B MOE (www.reddit.comhttps) Pretty exciting, wonder what it will take from llama.cpp to get it working locally
What's up on CPU inference these days? (www.reddit.com via reddit) What are the best models, quants and llama.cpp versions/forks for CPU inference these days? I have AVX2 but no AVX512 - Intel core ultra 7 165H; 64G RAM This seems to ask for massive MoE (a lot of RAM, not a lot of bandwidth/compute).
PADD: Path-Aligned Decompression Distillation for Non-Router Teacher to Guide MoE Student Learning (arxiv.org) CoRe-MoE: Contrastive Reweighted Mixture of Experts for Multi-Terrain Humanoid Locomotion with Gait Adaptation (arxiv.org) MoE Enhanced Federated Learning for Spatiotemporal Prediction (arxiv.org) Traffic prediction is fundamental to intelligent transportation systems and urban computing, yet many cities continue to suffer from traffic data scarcity due to limited sensor deployment and uneven urban development. Cross-city knowledge…
Achieving Cloud-Grade SLOs for Local Mixture-of-Experts Inference through CPU-GPU Hybrid Design (arxiv.org) Local deployment of large Mixture-of-Experts (MoE) models falls short of the service quality achieved in cloud-scale environments, even under low-concurrency workloads. We identify four key gaps in local MoE inference: reliance on capacity…
SHAPE: Coalition-Aware Expert Pruning for Sparse Mixture-of-Experts LLMs (arxiv.org) Sparse Mixture-of-Experts (MoE) large language models achieve strong quality with low per-token compute, yet their deployment is often limited by the memory wall: the full expert pool must remain resident to support token-dependent routing…
Qwen 3.6 35b A3B Speed Help (www.reddit.com via reddit) Jetson Orin NX Build for Hermes Agent + Benchmarking (www.reddit.com via reddit) I had a huge LLM server, and now I have a tiny one! I had a Jetson Orin NX gathering dust from a long dead robotics project, from back in the Llama-7B days.
Post-Trained MoE Can Skip Half Experts via Self-Distillation (arxiv.org) STAR: Rethinking MoE Routing as Structure-Aware Subspace Learning (arxiv.org) Mixture-of-Experts (MoE) scales model capacity efficiently by selectively routing inputs to a specialized subset of experts. However, input-expert specialization, the core motivation of MoE, critically depends on whether the router is actu…
Jetbrains Mellum 2: a really good and performant model (www.reddit.com via reddit) Oh Hey Folks, I took the Mellum 2 model for a spin, so I wanted to share my impressions here. Disclaimer: the tests presented here are not cientific nor have those nice names like perplexity,etc.
I tested in-conversation memory on LFM2.5, Gemma 4 E2B and E4B. The biggest model forgot a fact from earlier in the chat first. (www.reddit.comhttps) Ran a small, focused eval on three on-device models and the result was backwards from what I expected, so sharing the method and numbers. The task: tell the model "my dog is named Pablo," then add N turns of unrelated filler (shuffled gene…
Reversible Foundations: Training a 120B Sparse MoE through State-Preserving Scaling (arxiv.org) MACS: Modality-Aware Capacity Scaling for Efficient Multimodal MoE Inference (arxiv.org) I built a PyTorch MoE/MoD training framework with custom CUDA kernels [Apache 2.0] (www.reddit.com via reddit) PyTorch framework for training transformer LLMs with MoE and MoD architecture support, custom CUDA kernels, and DeepSpeed integration. Key things it does: - Custom CUDA kernels for RMSNorm, RoPE, SwiGLU, MoE routing.
2-bit QAT model releases (www.reddit.com via reddit) So far model releases that take advantage of Quantization a Aware Training (QAT) have been focused on 4-bit. I’m curious what could be accomplished with a larger MoE model around 120b up to 400b.
Dense vs MoE quantization resiliance (www.reddit.com via reddit) Which one is more resiliant to quantization? Especially at 4-bit?
It felt good to return my Asus Spark (www.reddit.com via reddit) It's an incredible little package but too expensive of a price to pay for the performance and I simply didn't want to be part of the great "Superchip lie" - it could be super, but its super ruined by its limited memory bandwidth even thoug…
Gemma 4 QAT accuracy inconsistencies (www.reddit.com via reddit) Table from https://unsloth.ai/docs/models/gemma-4/qat#qat-analysis I heard that MoE models are usually more susceptible to quantization error, but what happened with the 12B? I thought lower-parameter models usually quantized worse and yet…
Stuck trying to moe my folder out of the 'tabs' folder for 30minutes (www.reddit.com via reddit) Can anyone point me in the right direction, I've been going at this for over 30minutes just trying to move my "_budgetContext.tsx" file out of the (tabs) folder but nothig seems to be working https://preview.redd.it/8on4rg6cyo5h1.png?width…
Experimentation with Qwen 3.6 and Gemma 4 - Guidance needed (www.reddit.com via reddit) I’m a web developer doing mostly coding, but also project management, requirements analysis, testing, etc. I recently started experimenting with local LLMs, mostly because agentic stuff finally made them feel useful.
Gemma 4 Haters 2 months Ago now seems to love Gemma 4 now. (www.reddit.com via reddit) What's with the switch guys? now imagine if google gonna drop 128B model or a MoE version (I bet those Qwen lovers will forget Qwen even existed).
Running Qwen3.6-35B-A3B on a laptop RTX 4060 (8GB) — what worked, what didn't, and a surprising speculative-decoding result (www.reddit.com via reddit) TL;DR: I spent a long session tuning a 35B MoE on a tiny 8GB laptop GPU. Three things mattered a lot (--no-mmap, VRAM headroom, closing CPU-hungry apps).
What exactly is quantization aware training? (www.reddit.com via reddit) First time hearing it. I also heard about the gemma 4 qat quants and if any one of them is good for 4gb vram and 16gb ram.
AnchorMoE: Interpretable Time Series Classification via Anchor-Routed MoE (arxiv.org) UltraEP: Unleash MoE Training and Inference on Rack-Scale Nodes with Near-Optimal Load Balancing (arxiv.org) Less is MoE: Trimming Experts in Domain-Specialist Language Models (arxiv.org) Mixture-of-Experts (MoE) models achieve strong performance through conditional computation, but their large parameter footprint poses deployment challenges. Prior MoE compression approaches catastrophically fail when evaluated on general-p…
Value-and-Structure Alignment for Routing-Consistent Quantization of Mixture-of-Experts Models (arxiv.org) Mixture-of-Experts (MoE) models scale foundation models efficiently by activating only a subset of experts for each token, but their large number of expert parameters still makes quantization essential for practical deployment. Unlike dens…
2 RTX A6000 at 96GB VRAM with nvlink. Best local coding model/what you would daily drive? (www.reddit.com) Really been testing qwen 3.6 27b and 35 a3b so far with 27b at q8 and 35 a3b at q4 (byteshape quant is insane). But i feel im not utilizing it the best, esp for long context messy coding of large repos.
$16 refactor, 400 steps, 95% routed to open MoE (www.reddit.com) Got tired of $160 Opus bills so I spent a weekend wiring up a routing layer on vLLM 0.8 (2xA100, enable_auto_tool_choice). Getting the tool call parser to cooperate took longer than the actual routing logic.
Rejoice, if Qwen doesn't release any new local model, it's a blessing in disguise (www.reddit.com) Do you remember the times when we only had lama2 released? a bunch of finetunes were released and some of them had real values .
Comparison of Qwen 3.6 and Gemma4 (MoE and Dense models, Q4_K_M), generating a moderately complex MySQL query, only one produced acceptable results (www.reddit.com) I tried Qwen3.6 35B A3B MoE, Qwen3.6 27B Dense, Gemma4 26B A4B MoE, Gemma4 31B Dense. In all cases I was using Q4_K_M and thinking mode enabled.
Measuring Maximum Activations in Open Large Language Models (arxiv.org via reddit) The dynamic range of activations is a first-order constraint for low-bit quantization, activation scaling, and stable LLM inference. Prior work characterized outlier features and massive activations on pre-2024 LLaMA-style models, and the…
"Qwen 3 72B" doesn't exist — and it's in a surprising number of places that act like it does (www.reddit.com) spent today auditing my own model catalog and noticed 39 of my own pages confidently reference "qwen 3 72b" with apache 2.0 licensing, a 2025-09-15 release date, and a 131k context window. seemed normal — qwen 2.5 had a 72b, why wouldn't q…
Best llama.cpp launch config for Qwen3.6 27B on RX 7800 XT (16 GB VRAM) for OpenClaw? (www.reddit.com) I’m trying to find the best llama-server launch command / runtime config for running Qwen3.6 27B GGUF with full GPU offload on ROCm. I’m currently using the IQ4_XS quant, but I’m not sure if that’s the best option for my setup.
Estimate inference speed of local Qwen3.6-35B on Mac M5... (www.reddit.com) "Based on currently available information, estimate the prefill/decode speed of Qwen3.6-35B-A3B Q8 with 262K context on a Mac M5 Ultra 128GB." I'm surprised that almost every LLM fails at this task (ChatGPT/Gemini/Grok/Claude/DeepSeek/Kimi…
Don't you have issues in W11 with AMD GPU where llama.cpp suddenly drops performance for no reason ? (www.reddit.com) I have this issue in all Windows installations I have done in my system, which of course, does not occur in Linux. 7900XTX + 9800x3D + 64GB DDR5 Issue is that for some reason, after sometime, llama.cpp performance cuts in half, even restar…
rtx 5070ti with Qwen3.6-35B-A3B-Uncensored-HauhauCS-Aggressive-Q4_K_M.gguf token speed 564/41 (www.reddit.com) --model "/mnt/e/my-path-change-to-yours/qwen3.6-35b/Qwen3.6-35B-A3B-Uncensored-HauhauCS-Aggressive-Q4_K_M.gguf" \ --ctx-size 262144 \ --parallel 1 \ --n-cpu-moe 29 \ --no-mmap \ --mlock \ --cache-type-k q4_0 \ --cache-type-v q4_0 10.8/16 d…
What are the best 40-500 B MoE LLM models now? (www.reddit.com) Due to old GPU I run on CPU and came to appreciate value of MoE. I know of MoE for Qwen 3.6 and Gemma-4, which are <40B.
Possibility of partly moe weights gpu offloading via sglang/ktransformers (www.reddit.com) I’m interested in dual Xeon setup with AMX support for ktransformers and CPU sglang backend. Let’s say I have 512gb RAM in 8x channel for each CPU and 2x RTX6000 Pro.
Qwen 3.6 35B MoE at full 262K context on an RTX 3090. Here's exactly how I did it. (low.li via reddit) I spent a while getting this dialed in and wrote up the full recipe. Short version: 35B MoE TQ3_4S fits in 12.4GB of weights KV cache at q8_0/q8_0 and 262K context only uses 2.7GB because MoE only has 10 attention layers out of 40 Total VR…
Updated: RTX6k (Server, 450w) Qwen3.5-122B-A10B (MXFP4_MOE) Benchmarks (llama.cpp) (www.reddit.com) Round 2: 2026-05-02 — llama.cpp b8198 → d05fe1d Rebuilt llama.cpp from b8198 (2026-03-04) to commit d05fe1d (2026-05-02), ~770 builds of progress. Same model, same hardware, same flags.
thinking of gemma 4 26B vs 31B (www.reddit.com) I see a big difference in agentic coding between gemma-4-31B-it-Q5_K_M and gemma-4-26B-A4B-it-UD-Q8_K_XL. The 26B model is much faster because of A4B and generally works well, but there is a big difference in thinking.
Reasoning Guard: Stopping LLM Thinking Loops at the Proxy Layer (www.reddit.com) Reasoning Guard: Stopping LLM Thinking Loops at the Proxy Layer I’ve been running Qwen3.6 MoE behind a vLLM proxy and hit a specific reliability issue: occasional runaway reasoning loops. This isn’t a criticism of Qwen3.6.
Rada — AI coding workspace with local-first behavioral routing (no hot-swapping, I built this) (www.reddit.com) With GitHub pausing Copilot Pro+ signups and Claude Code potentially leaving the Pro tier, I started building the AI coding tool I actually wanted to use. One that doesn't depend on cloud access staying cheap and available.
Qwen 35B-A3B as an always-on agentic loop on a 16GB Mac M4: disk became the bottleneck before RAM (www.reddit.com) M4 Mac Mini, 16GB unified, basic spec. For a few weeks I had Qwen 3.5 35B-A3B UD-IQ3_XXS (12GB on disk) running under llama.cpp with --mmap and --flash-attn.
OpenMythos with Qwen2.5-1.5b weights (No recurrence atm) - looking to turn it into full OpenMythos (huggingface.co via reddit) Mythos likely isn't this architecture but I did find it pretty cool to experiment with. It has features of the HRM-27m architecture.
I like my models dense. Can model makers please bring back or update the dense models from like 2 years ago? A nice 39b or 72b maybe? (www.reddit.com) Seriously, Qwen3.6 27b is mopping the floor against models like 5 times its size right now. It doesn’t take a rocket scientist to figure out that maybe the whole a2b and a3b MoE thing isn’t the best solution after all.
My 12-agent Qwen 35B stack on Ollama died at 500 tokens every single time. Raw MLX fixed it and broke 4 other things I didn't see coming. (www.reddit.com) TLDR: Swapped Ollama for MLX on M1 Max (64GB) to run a 12-agent trading stack using Qwen 35B MoE. MLX wins on throughput and fine-grained sampler control, but I lost the "it just works" convenience of Ollama.
Ollama swap to llamacpp/llama server (www.reddit.com) So I'm a newb in certain aspects but not in others, I'm currently running an AI stack on my unraid server: CPU: AMD Threadripper 3960X (24c/48t) Motherboard: Gigabyte TRX40 AORUS PRO WIFI RAM: 256GB DDR4-3200 G.Skill Trident Z GPU: Nvidia…
IQ2XXS Qwen 3.6 35b is actually very usable on 32 gb macbooks (www.reddit.com) just tested the MoE qwen model with 2 bit percision and its suprising good. I used the 2 bit xxs from unsloth and it seems to maintain intelligence really well, never failed a tool call so far and suprisingly good at 3js, even better than…
Trade offs for companion roleplay (www.reddit.com) Hey folks for storytelling and companion style roleplay with a local llm, what do you think is the most important? More parameters Less quantization Larger context window Dense vs MoE When looking at what can fit in RAM, I’m thinking that…
Is there a way to load huge MoE models on a computer with way too little RAM for the model's size, inferencing from the SSD, on LM Studio using the mmap/GPU/CPU layer customization thing (similar to how you can on llama.cpp)? I can't get it to load without memory spiking and going into swap. (www.reddit.com) Proper vibe coding with local LLM for average Joe (www.reddit.com) The short answer is you don't. But let me explain a bit more.
Are we at the point where local AI isn’t a compromise anymore? (Gemma 4 experience) (medium.com via reddit) Thoughts on MoE Qwen 3.6 35B? (www.reddit.com) 5070 Ti (New) vs 3090 (Used) to pair with 4070 for local LLMs? (www.reddit.com) Should I switch from Qwen 3.5 27B (dense) to Qwen 3.6 35B-A3B for tool calls & vision? Need Docker config review + VRAM advice (www.reddit.com) Qwen3-30B-A3B-Instruct-2507 is better than the new Qwen 3.6 for our tasks (www.reddit.com) Gemma4 26B MoE on Arc 140T (www.reddit.com) Lm studio running some models very slow while others run normally. (www.reddit.com) Newbie here (www.reddit.com) Hi guys im on 9950x 196gb and a 4090 This parameters are ok? mi main use will be coding llama-server -hf unsloth/Qwen3.6-35B-A3B-GGUF:UD-Q8_K_XL --n-cpu-moe 20 -c 250000 --host 0.0.0.0 --port 8082 --reasoning-budget -1 --top-k 20 --top-p 0…
Imposing my laptop to run Qwen 3.6 (www.reddit.com) So, I am excited with the new MoE model released by Alibaba. And as an excited person, I want to believe that it can actually run in my hardware.
This is very fair. Other interesting context behaviors you've experienced? (www.reddit.com) I guess the model didn't feel it needed to do anything beyond proving. Not entirely sure how I got it to act so..
Qwen3.6-35B-A3B just dropped — quick thoughts after trying it (www.reddit.com) Just gave the new Qwen3.6-35B-A3B a spin. It’s a MoE model (35B total, ~3B active), but honestly the more interesting part is how much they’re pushing agent-style coding.
Anybody else seeing Qwen3.6-35B-A3B go crazy thinking in circles? (Compared to Qwen3.5-35B-A3B) (www.reddit.com) I was working on a simple frontend web design task earlier (styling some buttons) with Qwen3.5-35B-A3B. The end results weren't great, but at least it kept trying to change stuff and call toosl properly.
How faster is Gemma 4 26B-A4B during inference vs 31B? (www.reddit.com) I want to download one and usually do inference on CPU having old GPU so I'm concerned with speed. One link on the web (I have posted with it and post been removed): Multiple users are reporting that Gemma 4's MoE model (26B-A4B) runs sign…
Is Gemma 4 26B MoE or 31B good as an MCP agent for coding with Xcode? (www.reddit.com) Thanks
Hardware needed for Gemma 26B MoE vs Qwen 14B for ~100–300 users (vLLM, single node?) (www.reddit.com) I'm trying to figure out what sort of hardware setup i will need to accomodate a userbase of 100 users (not necessarily concurrent). Does anyone have any idea what sort of setup i'd be looking at?
Gemma4 vs Qwen3.5! MoE vs Dense! Sota vs Obsolete! Porque no los dos? (www.reddit.com) Every other day, there's someone posting about how the latest hotness of the month is gamechanger, but flawed in some way relative to their previous favorite. I can't help but wonder, does no one else keep their previous gen models on spee…