DiffusionGemma made me rethink what memory bandwidth means for local agent inference (www.reddit.com via reddit) agentic 22w
event
Qwen 3.5
-
Been testing DiffusionGemma 26B A4B for the last few days and the bottleneck profile is completely different from autoregressive models. With autoregressive models you are compute-bound during prefill and memory-bandwidth-bound during deco…
-
How do i prevent llama.cpp from offloading on Swap? (www.reddit.com via reddit) llamaqwen 22w
I have tried preventing this issue by using llama.cpp flags. However, I still have the issue: whenever I'm close to my 96GB of RAM, llama-server / llama.cpp decides to offload the KV cache onto my swap.