Are those "free 1B Claude tokens Github repo" just gimmick? (www.reddit.com via reddit) llamadeepseekqwen 5h
topic
Llama
-
I am not in this AI field but I have a question. Does they just give us the Claude interface or CLI experience and silently route our promts to free tier models like DeepSeek, Qwen, or Llama?
-
Show HN: Loft – offline Q&A over your local files, with citations (llama.cpp) (github.com via hn) llama 1d
Loft Your files. Your AI.
-
Llama.cpp v1.0 on an RTX 3090: up to 55% faster with speculative decoding (homebrewserver.club via hn) llama 1d
llama.cpp reached version 1.0 on 13 November 2026. This test compared it with a pre-v1 build from five days earlier (build 12199) on an RTX 3090.
-
ModelMesh – Native macOS app to shard models across Macs via Llama-RPC (modelmesh.app via hn) llama 2d
Native macOS orchestrator for local llama-server instances and multi-Mac GGUF sharding over llama-rpc . Built in AppKit, Swift 6, and Objective-C++ directly on top of llama.cpp .
-
Llama.cpp v1 Is Out (huggingface.co via hn) llama 2d
llama.cpp v1.0.0 : A stronger foundation for the local AI ecosystem llama.cpp makes it possible to run multi-modal LLMs across a wide range of hardware - from consumer-grade local machines to high-end enterprise servers. Through quantizati…
-
Im 17, Built a Swift-native LLM inference engine for Apple Silicon (github.com via hn) llama 3d
alpaca.swift Native language model inference for Apple Silicon. A from-scratch Swift inference engine with custom Metal kernels — no llama.cpp, MLX, Core ML, ONNX, or Python at runtime.
-
Llama.cpp: 6.5x faster prompt processing on Arm with a NEON kernel (github.com via hn) llama 3d
Arm NEON kernel for llama.cpp's tiled K-quant matmul llama.cpp reads prompts several times slower on Arm CPUs than on x86 for low-bit quantized models. The reason is one routine: the "tiled K-quant matmul" added in September 2026 (llama.cp…
-
Racial and ethnic minority patients receive less pain treatment than non-Hispanic White patients, and large language models (LLMs) used as medical AI could reproduce that bias. Prior audits tested a few proprietary models, stated race expl…
-
Grow your Llama army, win the territory (warlama.com via hn) llama 3d
WAR LAMA ⚔️ The great llama war ⚔️ PLAY Private room Create a room JOIN Practice vs bots 🏆 Ladder ⭐ Llama Club 🎒 Locker EN FR ES ⚙️ Settings
-
Inside Llama.cpp: How Local LLMs Run on Your Hardware [video] (www.youtube.com via hn) llama 4d
About Press Copyright Contact us Creators Advertise Developers Terms Privacy Policy & Safety How YouTube works Test new features NFL Sunday Ticket © 2026 Google LLC
-
Kindi: The fastest software inference engine for low-cost GPUs (amirhusain.com via hn) llama 4d
NVIDIA CMP 170HX 40 GB · Ternary-Bonsai-2 27B · new in 0.5.0Kindi writes faster than llama.cpp at every context that we measured, from 512 to 64K tokens, on one card and on two cards.406080100120106.196.374.156.1113.4110.192.977.25128K32K6…
-
Show HN: Tensward – Local AI inference optimisation (github.com via hn) vllmllama 4d
Tensward Measures how your LLM is served on your own machine, offers one change at a time, and keeps a change only if it is faster and the answers pass your check. You already run vLLM or llama.cpp.
-
We improve Turkish mathematical reasoning in Large Language Models by combining LoRA fine-tuning, weight-space model merging, and search-based inference. Using Turkish-Llama 8B-DPO with Turkish translations of MetaMath, NuminaMath, and Tig…
-
Anthropic recently released FlashProve, a research prototype (Apache 2.0) for zero-knowledge proofs of LLM inference. The provider commits to a set of weights, its GPU generates a proof for each reply, and anyone can verify that proof on a…
-
We study the effectiveness and robustness of open-weight large language models (LLMs) for Danish offensive language detection, an underexplored low-resource task. We evaluate four model families (Llama, Mistral, Gemma, and Qwen) across ele…
-
Llama-modchip: 284B DeepSeek on 64GB RAM, no GPU (github.com via hn) moellamadeepseek 8d
♟️🐋 llama-modchip (v2.4.0) Break the 64 GB RAM Wall: Run 284B DeepSeek-V4-Flash ( 284.33B Frontier MoE params 56.56 GiB checkpoint on disk 57.61 GiB peak VmRSS @ 256K ctx 44.36 tok/s 284B CPU prefill 60.4–111.6 tok/s 30B MoE CPU decode 13.…
-
Your phone already has everything it needs to run an AI assistant. Mine proves it.
-
TokiRecur – llama.cpp LLM launcher with KV-cache recycling (github.com via hn) llama 8d
Toki Recur KV cache reuse for local LLM inference. Exact prefix matching in front of llama-server, across CUDA, ROCm, SYCL, OpenVINO, Vulkan, Metal and CPU.
-
LexiSuite AIO bootable all rust OS/Engine/Kernel/Tuning & training management suite. Runs kernel 7.4 rc1 by default as the kernel needs work. Engine runs Qwen3.8 27b faster with lossless 5-25% lead over llama.cpp. 100% solo authored. Built on for 7900xtx vulcan, should work on other AMD cards. (github.com via reddit) llama 9d
Use balena etcher or whatever to mount to USB stick, boot. Test.
-
Subway Zombie FPS for fun using Deepseek v4.1 1017 UD-IQ3_S with Maya and OMP (www.reddit.comhttps) llamadeepseek 9d
inspired by Bijan Bowen's video - Subway FPS test. Hardware details: RTX 5090, 256GB DDR4 3200Mhz 8 channel, AMD EPYC 7532 CPU (32 core).
-
ran the new ripple-65b-preview (q4_k_m, llama.cpp, 64 gb macbook pro, no cloud) through atomic agent on a 5-part python app graded by 50 hidden tests. 3 runs.
-
Swapping LLMs in Under Two Seconds: A Sleep Mode for Llama.cpp (zaage.it via hn) llama 9d
Swapping LLMs in Under Two Seconds: A Sleep Mode for llama.cpp Contents Abstract# Different tasks want different language models, and with a single GPU a proxy such as llama-swap has to swap them, often several times an hour. A swap betwee…
-
zml/llmd: a fast and universal inference server (www.reddit.com via reddit) mistralglmgemma+4 9d
Hey all, I wanted to share that we're working on zml/llmd, a fast (we believe the fastest) inference engine that currently runs on NVIDIA, AMD, Intel, TPU and Metal. It has all the features of a modern LLM server: continuous batching, pref…
-
We observe a novel phenomenon, contextual entrainment, across a wide range of language models (LMs) and prompt settings, providing a new mechanistic perspective on how LMs become distracted by `irrelevant'' contextual information in the in…
-
We investigate whether open-source LLMs encode a linearly separable truthfulness signal in their hidden states, and at which network depth this signal is strongest. Across three $7$B--$8$B instruction-tuned models (Llama-3.1-8B, Mistral-7B…
-
Desktop app from the llama.cpp team (Mac OS and Windows) (llama.app via hn) tool-usellama 12d
Models Updated Oct 2026 A curated selection of open models — every one runs in llama.app. Good places to start LFM 2.5 24B-A2B 24 GB+ memory Liquid AI's fast local model for reasoning and tool use.
-
Running quantum circuits inside llama.cpp (github.com via hn) llama 12d
qullama qullama turns a quantum circuit into a qgate GGUF "model" whose layers are the gates themselves; running it with llama-embedding applies the circuit to |0...0> and returns the output statevector. llama.cpp is used with a small patc…
-
Ask HN: Do you have any kind of sandboxing for those who run LLMs locally? (news.ycombinator.com) llama 13d
I have the project of running a LLM locally on a Windows. Since no one can tell for sure what's going on I was thinking of sandboxie https://sandboxie-plus.com/ or better open sandbox https://open-sandbox.ai/ around a llama.cpp or whatever.
-
Inference engine in game scripting language using macros (dasllama.io via hn) llama 2w
A game scripting language that runs speech-to-text 5x faster than llama.cpp on CPU. How we ended up writing an inference engine and the macros that did it.
-
Large MoE models, powered by your SSD Website · Blog · Slack · Apache-2.0 macOS / Metal · Linux / CUDA · Windows (in progress) Run models larger than your RAM or VRAM. InferSSD builds on llama.cpp to run large mixture-of-experts (MoE) mode…
-
NeuronDump: Open a local LLM as a 3D graph, cut parts, see what breaks (github.com via hn) ollamallama 3w
NeuronDump Look inside any local LLM, find the parts it doesn't need, and make it smaller. NeuronDump reads GGUF model files (the format used by Ollama and llama.cpp) and shows what is inside: layers, attention heads, vocabulary, chat temp…
-
A 32K-window proxy that beat full context on long agent histories (github.com via hn) llama 3w
Mnemonic Proxy Like Johnny, your agent carries more than its head can hold — and keeps working for hours. A transparent HTTP proxy that sits between a coding agent and a local model (llama.cpp's llama-server, Strata, ds4-server, or any Ope…
-
I built an offline manga translator for Android that users local AI models (www.reddit.comhttps) llamaclaude-code 3w
I built Yomu, an Android app that translates Japanese manga pages on the phone. Open a page in any reader app and tap a floating button.
-
Ask HN: Secure minimalistic web front-ends to Local LLM's? (news.ycombinator.com) llamaanthropicopus 3w
Are there any secure minimalist web front-ends to Local LLM's? Anthropic are pushing me to local LLM's (opus 5.5 shock collars implemented on Opus 4.8) but if I am going to spend the duckets on the hardware I would like to share the hardwa…
-
Opus 5.5 orchestrated three local models and spent $2 (www.reddit.comhttps) llamaopus 3w
I wanted to see how much Opus 5.5 actually does when it runs as a planner over local models. In Fusion mode Opus plans and reviews, and three workers on a local model write the code in parallel.
-
I’ve been running various tests to see which AIs tell the truth most often and came up with a silly idea to run a social experiment. For a week or so now I’ve let this little village of AIs function and when they say and do things, we chec…
-
macOS 27 has a built-in, offline LLM you can call from Terminal (news.ycombinator.com) ollamallamachatgpt 3w
Apple appears to have made its Foundation Models accessible directly from Terminal in macOS 27. The interesting part is how little setup is required: ``` fm respond "Your Question" ``` You can also pipe text into it, request structured JSO…
-
Hi everyone! I'm the developer of llampart, an open-source Web UI for chatting with LLMs.
-
8B Llama model running at 17,000 tok/s on ONLY a single chip!!! (chatjimmy.ai via reddit) llama 3w
I found this project that I never saw talked about enough which is absolutley baffling to me. The title is not clickbait, and I can't blame you if you thought it is, because I was in your shoes once.
-
Private and OpenAI-Comp. Inference Engine for Nvidia Linux and Apple Silicon (github.com via hn) glmgemmallama+2 3w
taraengine The Tara Vision Inference Engine Local, private and OpenAI-compatible: Gemma, GLM and Qwen on NVIDIA Linux machines and Apple Silicon Macs. Built on llama.cpp.
-
Reposting again since original post got removed by Reddit for some reason (I have no idea why) A few of you pointed out in my last post that comparing Flash-Next and Opus with different harnesses and effort levels wasn't fair. So here is a…
-
Best LLM Fine-Tuning Services: What to Look for in 2026 (www.rebiha.com via hn) gemmallamafine-tuning 3w
Pick a model — Qwen3.8, Granite, OLMo, Gemma 4, Llama & more — choose from 54 ready-made datasets, and train. No data prep, no infrastructure, no API.
-
Benchmarks yzma is fast because it calls llama.cpp in the same process. There is no external server.
-
2x Tesla p100s, q6_k quant, Qwen 3.8 27B ~60tps (github.com via hn) llamaqwen 4w
This is a fork of llama.cpp tuned for two Tesla P100s (sm_60). On Qwen3.8-27B, plain decode runs at 1.86x upstream's speed (17.5 -> 32.6 t/s), and ~3x with the model's built-in MTP head (~54 t/s).
-
Modern healthcare generates increasingly complex medical reports that challenge clinical professionals in terms of time and interpretive burden. Manual review processes are slow, error-prone, and difficult to scale.
-
If llama.cpp enables MoE to activate multiple experts (www.openai-hub.com via hn) moellamaqwen 4w
llama.cpp让MoE多激活专家 社区分支为 llama.cpp 加入实验性 MoE 专家扩容:无需重训或微调,就能在运行时让模型激活多于原生 Top-K 的专家。该方案已在 Qwen 3.6 35B A4B+ 上测试,但仍处于早期实验阶段。 llama.cpp 开始实验性支持 MoE 专家扩容:不重训也能多激活专家 2026 年 9 月 6 日,llama.cpp 社区出现了一项值得关注的实验性改动:在不修改模型权重、不重新训练、也不做微调的前提下,让稀疏混合专家模型…
-
Gemma 3 1B at 114 tokens per second on a laptop CPU (knl.co.in via hn) gemmallama 4w
Gemma 3 1B at 114 tokens per second on a laptop CPU Research 9 minute read How a ternary conversion, a 1.6-bit kernel and a cheaper output layer made a small model 3.8x faster than llama.cpp on my laptop, and what it cost in quality. I too…
-
Transformers are efficient to pre-train at scale because attention permits parallel computation across sequence positions, but long-context inference requires increasingly large KV caches. Efficient recurrent alternatives reduce the asympt…
-
PS5LM: Llama.cpp on a jailbroken PS5, run any GGUF model natively on the console (github.com via hn) llama 4w
PS5LM llama.cpp on a jailbroken PlayStation 5. Run any GGUF language model on the console itself, first on its Zen 2 CPU, then on its GPU.
-
Show HN: Enki LLM A pure rust (even kernels) with live GPU/CPU brain migration (github.com via hn) llama 4w
Enki LLM: Llama-3.2 1B in Pure Rust (Zero CUDA Dependencies) A pure Rust Transformer inference engine (~2,000 LOC) running Llama-3.2 1B at ~40 tok/s on an Nvidia Tesla T4 GPU, with live brain migration between GPU VRAM and CPU RAM mid gene…
-
I built LlamaForge, a free, MIT-licensed browser control panel for llama.cpp. It isn't a fork: it drives the real upstream llama-server, so when llama.cpp adds a model you update from the panel and have it that day.
-
Fine-tuned Llama 3.1 8B vs. Claude Sonnet 5 on 437 legal clauses (reopenly.com via hn) llamasonnet 4w
If your product runs the same AI task thousands of times a day, you are probably paying for a model that knows far more than the job needs. So we put it to the test.
-
SonicBoom – llama.cpp's own kernels, 14% faster decode via a graph compiler (github.com via hn) llama 4w
SonicBoom A native tensor runtime. No Python.
-
The Universal LLM Server (zml.ai via hn) mistralgemmallama+1 5w
Continuous batching Serve concurrent requests efficiently without hand-building a scheduler around the model. ZML/LLMD A self-contained inference server for LLaMa, Gemma, Qwen and Mistral models, running across NVIDIA CUDA, AMD ROCm, Googl…
-
This will be my third update on the bc-250 cluster and for my first forray into local ai I have been having a blast. This has turned from a weekend experiment into something I think I will use in the future.
-
local semantic file search for Linux (Rust, llama.cpp, EmbeddingGemma 2) (www.reddit.com via reddit) llama 5w
indexes PDFs, Office files, scans, screenshots, audio and video, and finds them by content. A query like "when did they decide to freeze hiring" opens the recording at that second.
-
Show HN: LoRA over GGUF – Train Qwen3.8-Flash-Next in 40G VRAM (github.com via hn) moellama 5w
Open weight AI is like open source software. Users not only run the weights, but also modify the weights.
-
Kisoku 1.6B: LLM trained solo from scratch on a TPU grant, matches Llama 3.2 1B (github.com via hn) llama 5w
Kisoku 1.6B A 1.6B-parameter language model pretrained from scratch by one person on a TPU v4-32 from Google's TPU Research Cloud, on about 0.5 trillion tokens. On a 10-benchmark suite run through the same harness for every model, it match…
-
Llama Factory (news.ycombinator.com) llama 5w
The Llama Factory was the leading business in Llama production, Llamas, well known to be cyborgs, where mainly built at the Llama Factory by other cyborg Llamas, until, last year, the all the Jewels, gold, diamonds, croissants and vbucks w…
-
MS announces DeepSeek V4 Flash and Nemotron locally, llama.cpp comes to Win ML (pasqualepillitteri.it via hn) llamadeepseek 5w
could not extract summary
-
AI Development on Windows: From PyTorch and Llama.cpp to Windows ML (devblogs.microsoft.com via hn) llama 5w
Today we are highlighting improvements across a few key open-source projects many of you already use, as well as updates to Windows ML. To empower developers, we must support the broad range of tools for experimentation and exploration acr…
-
Show HN: Long term memory agent for "real world" data work (github.com via hn) ollamallamacodex+1 5w
Hi HN! I am an FDE and in all my clients' projects I have encountered the same problem: real world data preparation is the bottleneck.
-
Llama.cpp and WebGPU = Client-side LLMs [video] (www.youtube.com via hn) llama 5w
About Press Copyright Contact us Creators Advertise Developers Terms Privacy Policy & Safety How YouTube works Test new features NFL Sunday Ticket © 2026 Google LLC
-
I spent the last few weeks comparing llama.cpp output across backends on a small multi-vendor GPU fleet, and the single most useful thing I learned wasn't about inference quality — it was that on four AMD hosts, the official ROCm prebuilt…
-
Local AI ecosystem overview (www.reddit.com via reddit) llama 5w
Hey guys, it's Merve from Hugging Face! I've recently given a talk in a dev conference about llama.cpp + but also covering basic concepts like prefill vs decode, memory types, speculative decoding etc.
-
The Llama 3 Herd of Models (arxiv.org) llama 5w
-
Reika – A coding agent CLI designed around small local models first (github.com via hn) llama 5w
A coding agent CLI for local and hosted models, designed around small local models first. Website · Quick start · Docs · Tested models · 简体中文 A real local run of Qwen3.6 35B A3B (Unsloth UD-IQ2_M) via llama.cpp, sped up 3×.
-
LLaMA Now Goes Faster on CPUs (justine.lol via hn) llama 6w
Mar 31st, 2024 @ justine's web page I just wrote 84 new matrix multiplication kernels for llamafile which enable it to read prompts / images faster. Compared to llama.cpp, prompt eval time with llamafile should go anywhere between 30% and…
-
I built a local LLM cluster at home with Claude Code doing most of the hands-on work. Three Mac minis: an M1 as the gateway (OpenAI-compatible API, logging HTTP proxy, Tailscale) and two base M4s with 16 GB each that split one 35B mixture-…
-
Hey HN, Anders and Tom here. We're building Magnitude, an inference engine for agents that optimizes itself to run as fast as possible on your hardware.
-
ContextMemory – Markdown memory for your llama.cpp/vLLM server (github.com via hn) vllmllamaopenai 6w
Try it (3 commands) · Engines · Docs · vs Mem0 / Zep / Letta Self-hosted memory gateway for your llama.cpp / vLLM server. Put one OpenAI-compatible /v1 URL in front of your engine.
-
I built NIBIA Fabric, an open-source distributed LLM inference system that pools CPU and RAM across macOS, Linux, and Windows machines. It builds on llama.cpp, with capacity-aware scheduling, adaptive memory reservation, persistent tensor…
-
Gevva0 – a Jev like decision engine on Gemma 26B via direct logit scoring (github.com via hn) gemmallama 6w
Gevva0: Calibrated Local Decision Gateway -00f2fe) []() A high-speed, local decision gateway running on Gemma 4 (llama.cpp / CUDA) that replaces flaky autoregressive JSON generation with direct token logit scoring, mathematical cyclic debi…
-
Show HN: Jeva.cpp – a llama.cpp fork with JEV-compatible API for all LLMs (github.com via hn) llama 6w
you can download it from releases and try it now!
-
I got 2.2x more tokens per second from llama.cpp on Intel Arc (grigio.org via hn) llama 7w
How I got 2.2x more tokens per second from llama.cpp on Intel Arc I spent a weekend benchmarking llama.cpp on my Xiaomi Book Pro 14. The machine has Intel's Core Ultra X7 358H and Arc B390 integrated graphics, with 30 GiB of unified memory.
-
Cloud LLM services typically require users to send prompts to a model provider, creating a privacy risk. Fully homomorphic encryption (FHE) lets a server perform inference without decrypting the input, but representing data as ciphertexts…
-
Glyd – Run LLMs in 33% less GPU memory, bit for bit (getglyd.com via hn) mistralgemmallama+1 7w
Lossless AI compression: 33% less GPU memory, bit for bit. Glyd stores the weights of open models like Llama, Qwen, Gemma and Mistral in about 11 bits instead of 16, and decodes them inside the GPU's matrix multiply.
-
Llama.cpp banned him for accidentally tagging in a fork PR (github.com via hn) llama 7w
llama.cpp LLM inference in C/C++ ggml / ops / maintainer PRs%20sort%3Aupdated-desc) / dev stats / lib llama API / llama-server REST API Quick start A few options to get llama.cpp installed on your machine: Visit https://llama.app and follo…
-
42x faster prompt lookup drafting in llama.cpp (jadidbourbaki.github.io via hn) llama 7w
[homepage] [github] [twitter] This article was originally published on 2026-09-26. TL;DR I make drafting for prompt lookup decoding in llama.cpp up to 42x faster while using up to 2.6x less memory through a set of simple performance optimi…
-
Show HN: Prodigy – The Self-Updating Technical Digest (www.prodigy.org.in via hn) llama 7w
I've seen newsletters where we have 15-20 people working on typing out the content and publish it. Obviously that gives you originality, but what if we don't have that much time and resources?
-
Custom Models in Oh My Pi: vLLM, Llama.cpp, SGLang and More (doug.sh via hn) vllmllama 7w
Custom Models in Oh My Pi: vLLM, llama.cpp, SGLang and More Custom models let you run Oh My Pi (omp) on local models, or on models from providers it doesn’t support out of the box. omp reads them from ~/.omp/agent/models.yml .
-
llama.cpp: fork with multi gpu acceleration even for models bigger than total gpu mem Big thanks to @csantiago78: the expert cache here builds on their implementation in llama.cpp PR #27861 ("GPU-resident LRU cache for host-offloaded MoE e…
-
Detectors of alignment failures screen deployed language models and score alignment benchmarks. Most are generative judges that spend a decoding pass on every criterion, and classifiers that read token probabilities, such as Llama Guard, s…
-
Using Llama-cpp-Python grammars to generate JSON (til.simonwillison.net via hn) llama 7w
llama.cpp recently added the ability to control the output of any model using a grammar. This is an incredibly powerful technique for working with a Large Language Model.
-
Transformers now runs llama.cpp quants (huggingface.co via hn) llama 7w
Transformers now runs llama.cpp quants from_pretrained , and start generating on your own machine. Running AI models on your laptop has become much easier, and llama.cpp has been a big part of that.
-
The Basics of Transformer Inference (jax-ml.github.io via hn) llama 7w
Part 7 of How To Scale Your Model (Part 6: Training LLaMA | Part 8: Serving LLaMA) Performing inference on a Transformer can be very different from training. Partly this is because inference adds a new factor to consider: latency.
-
LM Studio is a free desktop app to run local LLMs on your own computer (www.lm-studio.net via hn) llamadeepseekqwen+1 8w
Welcome to the LM Studio docs LM Studio is a free desktop app to run local LLMs on your own computer — a private, offline ChatGPT alternative for macOS, Windows, and Linux. Download open models like gpt-oss, Llama, Qwen, DeepSeek R1, Mistr…
-
Turn any Llama-server into a jev system one endpoint (github.com via hn) llama 8w
verdict an open source, self-hosted alternative to typesafe's jev and its system one api. verdict serves the same wire protocol on your own hardware, so an existing jev client changes its base url and nothing else.
-
Frontier open-weight language models increasingly use Mixture-of-Experts (MoE) architectures to expand model capacity while activating only a small subset of experts per token. Local inference must nevertheless keep the complete expert poo…
-
Benchmarking Local LLM Servers: Llama.cpp, Llamafile, LM Studio, and Ollama (blog.mozilla.ai via hn) ollamallama 8w
Benchmarking Local LLM Servers: llama.cpp, llamafile, LM Studio, and Ollama Benchmarking Local LLM Servers evaluates llama.cpp, llamafile, LM Studio, and Ollama across Mac, Linux, and Steam Deck. The study reveals build flags and configura…
-
Language models frequently abandon correct answers when users push back. We study this in two small instruction-tuned models from different families, Qwen2.5-1.5B and Llama-3.2-1B, over TriviaQA: the model answers, is challenged with one o…
-
Who the judge is can affect an LLM-as-judge result, but measuring that effect without confusing it with candidate quality is difficult. We study four open-weight families (Llama 3.1, Qwen 2.5, Gemma 2, and Yi 1.5) in a fully crossed pairwi…
-
Benchmark Llamacpp on Intel Panther Lake: Vulkan vs. SYCL vs. OpenVINO vs. CPU (grigio.org via hn) llama 8w
I benchmarked the Qwen3.6-35B-A3B-MTP-UD-Q4_K_M.gguf model (a 35B Mixture-of-Experts model with 3B active parameters, 22 GiB quantized) across all available llama.cpp compute backends on an Intel Panther Lake laptop. The goal: find out whi…
-
We predict single-sequence model throughput from GGUF metadata using roofline-shaped predictors with quantization-specific scale factors fitted on reference models. The scored cohort comprises 318 phase-depth measurements from 53 host-file…
-
Making a vintage LLM from scratch; Take #2 (crlf.link via hn) llama 8w
Making a vintage LLM from scratch; Take #2 Three and a half months ago I wrote about making my own vintage LLM from scratch, trained only on texts written before the year 1900. That project ended with a Llama-340M params base model that co…
-
Show HN: I built Otis, a minimal AI agent that runs local models out of the box (triangllabs.ai via hn) ollamallama 9w
Hi everyone, Been working on Otis, an open-source ai agent that gives you one minimal experience across local and hosted open-weight models, privacy-focused by design. On setup it recommends a local model based on the hardware Otis is runn…
-
Automatic safety judges -- systems such as Llama Guard or a GPT-4o grading prompt that decide whether a model's reply is harmful -- produce the numbers behind almost every reported jailbreak success rate, defense evaluation, and safety lea…
-
I have a strong suspicion that Claude Code is deeply optimized for Anthropic’s own models (especially Opus and Fable) and that performance drops noticeably when you try to run it with other LLMs. Has anyone actually tested this properly?
-
Ollama replacement 2-4x faster for no extra compute cost (github.com via hn) ollamallama 9w
llmash An Ollama-compatible server and command line for Windows, built on llama.cpp. It serves your GGUF files through llama-server and keeps Ollama's commands, API and model store, so anything already pointed at Ollama keeps working.
-
Show HN: Local audit trail for Claude Code tool calls (github.com via hn) llamaclaude-code 10w
A dead simple local audit trail for Claude Code tool calls. Energized by a rule-based risk analyser.
-
Anyone using llama.cpp willing to test LlamaRack? (github.com via hn) llamaopenai 10w
LlamaRack A self-hosted control plane and OpenAI-compatible gateway for llama.cpp. LlamaRack manages GGUF models, durable llama-server Instances, GPU placement, automatic loading and unloading, Hugging Face downloads, request observability…
-
Show HN: We Beat MLPerf: Modern Storage for KV Offload and LLM Training (www.theopenlake.com via hn) llama 10w
OpenLake Leads MLPerf Storage v3.0 OpenLake’s Infinity Core I/O Engine delivered 6.72 GiB/s writes and 11.55 GiB/s reads in the MLPerf Storage v3.0 Llama 3.1 8B checkpointing benchmark. September 1, 2026.
-
Show HN: MyHandler – Local-first AI assistant for Windows, llama.cpp on Vulkan (myhandler.ai via hn) llama 10w
MyHandler is the private AI assistant and gatekeeper that handles your email, meetings, and calls. Local-first, encrypted, you stay in control.
-
Georgi Gerganov on llama.cpp/ggml future after Nvidia acquisition of HuggingFace (twitter.com via hn) llama 10w
Hugging Face has been acquired by NVIDIA It is quite exciting to be a part of this journey! NVIDIA has been an active supporter of the llama.cpp project.
-
Show HN: Llama – Self-Hosted OSS Alternatives to Vercel and Supabase (apps.lla.ma via hn) llama 10w
Self-hosted deploy console — your own Vercel.
-
ERIS: Local-first vault agent in Rust, grammar-enforced tools (github.com via hn) llama 10w
Eris A local-first agent in a single Rust binary: your Markdown vault as memory, grammar-enforced tool calls on llama.cpp, and nothing leaves your machine unless you say so. Eris runs a local LLM as a personal agent over a plain-Markdown v…
-
Guard Llama (en.wikipedia.org via hn) llama 10w
Guard llama A guard llama is a llama that is used in farming to protect sheep, goats, hens or other livestock from canids such as coyotes, dingos, dogs, foxes and other predators.[1][2] In the past, a single gelded (castrated) male was rec…
-
Show HN: PicoLM v1.0-rc1 (github.com via hn) gemmamoellama+2 10w
PicoLM is an LLM inference engine written in C99. It currently supports llama-2, GPT-2, Qwen 3.6/3.8(+MoE) and Gemma-3n models.
-
Long story short, about a year ago, in spite of everybody bashing gpt-oss for broken tool calling and refusals, I thought there's something there worth exploring. Model hit a sweet spot for me in that it was the first time I could run full…
-
SinLlama -- A Large Language Model for Sinhala (arxiv.org) llama 10w
Low-resource languages such as Sinhala are often overlooked by open-source Large Language Models (LLMs). In this research, we extend an existing multilingual LLM (Llama-3-8B) to better serve Sinhala.
-
The Llama.cpp Fork That Enables Qwen 3.8 27B Large Contexts for 16GB VRAM GPU (github.com via hn) llamaqwen 10w
Adaptive KV Streaming for llama.cpp This branch adds an experimental, block-granular KV cache streaming path to the CUDA llama-server. It is intended for running long contexts when model weights leave too little VRAM for the complete KV ca…
-
Show HN: VelocityNote – A tiny Markdown notebook with local AI (velocitynote.app via hn) llama 11w
Hi HN — I built VelocityNote, a compact, Markdown cross-platform notebook. The desktop app is under 100 MB.
-
Running Llama 3.1 405B (a11ce.com via hn) llama 11w
a11ce.com/llama-3.1-405b.html Llama 3.1 405b (base) is no longer hosted by any public inference provider. This page has instructions for running it yourself on an on-demand GPU instance for ~$20/hr and 10 minutes of setup.
-
I am Concerned if Nvidia Acquires Llama.CPP, Dev Team and HF, Anybody else? (www.reddit.com via reddit) llama 11w
I dont know about others, but Nvidia seem to close the lid on older GPUs since they want to push their new technology. Llama and team has been the to go places for older GPUs like V100s.
-
Hi! Just wanted to share a llama.cpp branch which should optimize Qwen-3.8-Flash-Next on Apple Silicon: https://github.com/tarruda/llama.cpp/tree/metal-qwen4exp-split-ngram Here are the numbers I'm getting on my M1 Ultra: % ./build/bin/lla…
-
I always run so many AI agents that I keep ending up with mystery processes, stray localhost ports, and no quick way to tell what started what. So I built Port Visualizer, a free open-source Windows app that shows which process owns each T…
-
Qwen3.8 27b: UD Q_K_XL vs W4A16-AutoRound (www.reddit.com via reddit) llama 11w
Hi, I've been trying to squeeze every bit of performance and context on RTX 3090 with llama.cpp, and after many tests I've come up with using both mtp and ngram but with --spec-draft-p-min 0.75, achieving around 45-50 tps in average with 1…
-
Over 200k context on 16GB VRAM with Qwen 3.8 27B UD-IQ3_XXS (www.reddit.comhttps) llamaqwen 11w
I was using UD-Q3_K_XL until now with more than 140000 context. Quality wise it's very good, very few erroneous tool calls.
-
With HuggingFace, Nvidia is also acquiring llama.cpp and the team behind it (www.reddit.com via reddit) llama 11w
With this move Nvidia is not only acquiring the HuggingFace platform, but they might also effectively acquire the copyright to the `llama.cpp` project, together with the entire team behind it. In February 2026 the llama.cpp team was employ…
-
44tks on a 5090 using Flash at Q4 and using ```--moe-cache auto```
-
I used local Qwen 27b to build a harness and replace OpenCode (www.reddit.comhttps) llamaqwen 11w
Sharing my harness for running local LLMs that I built using Qwen 3.x 27B (> 90% locally built) under my supervision - not vibe-coded. Its free, no telemetry, and open-source (AGPL).
-
Llama.cpp with ROCm 7.14 on Radeon 780m - fast, but unstable. Workaround (www.reddit.com via reddit) llamaqwenchatgpt 11w
Recently I posted some benchmarks of that setup which looked promising. So, I started using it with Qwen 3.8 but bumped into frequent crashes :( I would like to continue using ROCm as it gives me much higher preprocessing speed for small p…
-
Looks like we are close to local llama robotics (x.com via reddit) llama 11w
HuggingFace releases microduck a 10 inch open-source biped with 15 actuators and sensors (camera, speaker, LiDAR, NFC, bluetooth, wifi, ...) that you train yourself with reinforcement learning, $400. https://pollen-robotics.com/microduck/…
-
openrouter is a hop. local 27b on the mac is on-prem for the tool loop (www.reddit.com via reddit) llamaqwen 11w
openrouter joining stripe doesn't change the 27b that already fits. qwen 3.8 27b q4_k_m is ~16-17gb.
-
this is a friendly reminder you can legally seed ai models via torrenting. (www.reddit.com via reddit) llama 11w
A lot of people are skeptical Nvidia will keep huggingface intact now that they will buy huggingface. There's a lot of doom and gloom about not having any alternatives, removing nsfw models, saying there's no decentralized alternative or j…
-
MIND BLOWING - QWEN3.8-FLASH-NEXT - RTX 3090 + 128 DDR5 +/-20 tok/s (www.reddit.comhttps) moellamaqwen 11w
Qwen3.8 Flash Next just built this demo, and I have to say: this thing is seriously impressive. For a local model, the combination of coding ability, reasoning, and execution speed is kind of crazy.
-
Running local LLM's as agents in Claude Code (www.reddit.com via reddit) vllmllamamcp+2 11w
I hit my token limit three times a day on my max subscription - got sick of that and designed this MCP setup to shift some of the coding load to my local Qwen3.8-27B model. I've been iterating on it now for a bit, and thought I'd share it…
-
5090 People please share your Llama.cpp models.ini config for Qwen3.8 27B (www.reddit.com via reddit) llama 11w
This is mine, I'm on a 5090 with 128GB of DDR4 ram with a recent gen i3 cpu. Should I get a higher quant and lower my context down from 256K?
-
Who else is drop watching? (www.reddit.com via reddit) glmllama 11w
curl -s https://api.github.com/repos/ggml-org/llama.cpp/pulls/27742 | jq -c '{draft,state,merged}' for r in unsloth/GLM-5.3-Flash-GGUF unsloth/Qwen3.8-Flash-Next-GGUF; do echo "== $r" curl -s "https://huggingface.co/api/models/$r" \ | jq -…
-
little tool for offline wikipedia RAG (www.reddit.comhttps) llamaqwenrag+1 11w
I was bored and handwrote a tiny 100-line bash script to let an agent search for and read articles from an offline wikipedia archive during a regular chat. It's not particularly useful, but it's definitely neat and a big step up from llama…
-
Axera AX8850 LLM running ggufs (github.com via hn) llama 11w
ggml-axcl — llama.cpp Axera NPU backend A custom llama.cpp backend (ggml-axcl) that runs Qwen3-0.6B directly from GGUF on an Axera AX8850 NPU accelerator card (M5Stack LLM-8850: 24 TOPS INT8, 8 GB LPDDR4x) hosted on a Raspberry Pi 5. The G…
-
Are the best settings for single 3090 just ninfer-3090 build or can i do better? (www.reddit.com via reddit) clinellamaqwen 11w
I come from Qwen 3.6 35B-A3B + cline plugin of vscode at full 256k context but it eats 100k of context just to say "write me a python script that say print("hello")" so when i upgraded to Qwen 3.8 27B i switched to pi.dev i'm trying to und…
-
OpenCode with Qwen3.8-27B for Small Games or Browsing the Web With 16GB VRAM (www.reddit.comhttps) llama 11w
In the past, I have use llama.cpp, but I read that the exl3 quantization format should give better precision, so I have tried exllamav3/tabbyAPI. It was able to write the shown simple HTML game without interaction after asking some questio…
-
Alternative to textgen? (www.reddit.com via reddit) llama 11w
I've been using textgen (text-generation-webui) forever, but since the author is contributing to unsloth development seems to have stagnated, and the included llama backend is obsolete. So I was looking for a good alternative that has all…
-
Getting Qwen3.8-27B with decent speed on my 4080 with 16Gb card (www.reddit.com via reddit) llama 11w
I saw that Q2 is actually very good and produce real good results and I also saw how dflash2 make its running at generating >60 t/s with a 120k context lenght. And I like what its doing!!
-
WHAT THE FUC& AM I DOING WRONG . Help (www.reddit.com via reddit) llama 11w
Qwen3.6-35B-A3B-Uncensored-HauhauCS-Aggressive Q4_K_P on llama.cpp, -ncmoe offload cause it doesnt fit in vram outright, 10 threads, q8_0 kv both sides. nothing weird about the setup far as i can tell.
-
About the Huggingface sale.. (www.reddit.com via reddit) llama 11w
.. didn't llama.cpp aka ggml get acquired by Huggingface not too long ago?
-
Re-done benchmarks for V620 on Windows/ROCm & Vulkan (www.reddit.com via reddit) llama 11w
I'm here to show some benchmarks while using llama.cpp with an AMD V620 on Windows 11 via Vulkan & ROCm. These have been reuploaded & older threads deleted ran it with longer tokens thanks to a rec by someone who commented.
-
I've made a few changes here and there to get nearly 2.1k tokens/sec prefill, ~210-230 tokens/sec decode with MTP7 (configurable, extended up to 15) on benchmarks. Also added disk caching options, up to 30GB per config by default for near-…
-
why Qwen 3.6 35b a3b q4-k-m is slower on 7900xtx (20t\s 100% GPU Load) than on 3060ti (37t\s and GPU 50% Load) on the same settings? Linux llama.cpp vulkan 1.5Gb VRAM is empty.
-
New: Llama.cpp adaptive speculation for faster inference (www.reddit.com via reddit) llama 11w
We have been working on some performance optimisations for Qwen3.8 and other models. The main new feature that we introduced is adaptive speculation for Llama.cpp What is it?
-
Optimal llama.cpp/server settings for Qwen 3.8 27b (RTX 6000 Pro) (www.reddit.com via reddit) llamaqwen 11w
Currently the following llama-server settings are in use for Qwen 3.8 27b, it is running on a single RTX 6000 Pro, which allows running the full bf16 + 256kb context in bf16 cache. I am unsure if the current settings are optimal, they are…
-
Ornith 1.5 seems overfitted in my config, asked it to tell me a story, they are very similar (www.reddit.com via reddit) llama 11w
https://preview.redd.it/fk76mfp8nglh1.png?width=5108&format=png&auto=webp&s=6836c5f9f47ae3760ade0a365ae76907d7add14f I normally use this prompt when I fine tuning llama.cpp params, like ngl, draft, ctx-size... After using this prompt somet…
-
FIXED 7900 xtx + headless Linux crashes (Low RAM OOM) amdgpu.runpm=0 (www.reddit.com via reddit) llama 11w
TL:DR: power management was doing auto-suspend and dumping model weights into RAM. Models weights 18gb+ RAM 16gb = no Bueno immediate crash.
-
Protein-peptide interactions are central to cellular regulation and peptide-based drug discovery, yet existing computational methods mainly focus on interaction classification, binding-site prediction, or peptide binder generation. These f…
-
Anyone got a solid run script for 7900xtx without 4g & rebar? https://github.com/ggml-org/llama.cpp/releases/download/b10612/llama-b10612-bin-ubuntu-rocm-7.14-x64.tar.gz
-
I have Qwen3.8 27B running across two machines with llama.cpp RPC. The main PC has an RTX 4070 Ti with 12 GB VRAM, and the worker is an M5 MacBook Air with 16GB unified memory.
-
llama.cpp docs now have a new home ❤️ (llama.app via reddit) llama 11w
Introduction llama.cpp lets you run large language models locally on your laptop, desktop, or server with minimal setup and state-of-the-art performance. With a single command you can chat with a model in your terminal, or spin up an OpenA…
-
[Benchmark] llama.cpp batch/ubatch impacts on PP and TG (www.reddit.com via reddit) llamadeepseek 12w
My test is running DeepSeek v4 Flash 0731 at native size on DGX Spark machine (GB10, 128 GB unified memory). The model size is bigger than RAM, so weights will be loaded many times when running.
-
ReBar support for 20gb RTX 3080? (www.reddit.com via reddit) vllmllama 12w
Has anyone successfully patched the drivers of a modded RTX 3080 20gb to get Rebar support? I already updated my 3090.
-
Graph: A cumulative metric of Speed x Context Size - the higher the dot - the better. Helpful for looking for the most balanced solution.
-
instead of begging ai labs cooks I was invited to make my own gguf bf16 to f16 conversion setup for strix halo models fitting and despite a couple of showstoppers along the way it was a smooth ride. Here is the result, enjoy: https://github.com/littlett77/gguf-bf16-to-f16-conversion-script-for-strix (www.reddit.comhttps) llama 12w
I could only test llama-bench on a couple of gguf files having bf16 tensors baked in that have been successfully converted to f16 (I only had a couple of them as I tend to filter such gguf out) and so far it creates functional files. Haven…
-
Best model for 16gb ram Mac (www.reddit.com via reddit) gemmallama 12w
Hi everybody! Every now and then these days, we’re seeing really huge open-weight models popping up.
-
Is there a way to run a local Claude Desktop-type setup? (www.reddit.com via reddit) glmcopilotllama 12w
I used to use Qwen3.6-35B-A3B with llama.cpp and connecting it to the VSCodium extension called "Continue." My computer is running a Intel(R) Core(TM) Ultra 7 265K (3.90 GHz) with 128 GB of DDR5 RAM and an Nvidia Geforce RTX 5090 that has…
-
Llama-Mobile: Efficient 2.7-Bit Quantization of VLMs (arxiv.org) llama 12w
Deploying vision-language models (VLMs) on mobile devices is challenging due to their significant memory and compute requirements. We present a framework for quantizing VLMs for efficient inference on resource-constrained hardware.
-
Qwen 3.8 27B, just wanted to say thanks to you guys (www.reddit.com via reddit) llamaqwen 12w
I commented on another Qwen 3.8 27B post that I was frustrated getting anything to work. You all gave some great comments.
-
ConvRot Quant method now in llama-cpp-turboquant (www.reddit.com via reddit) moellama 12w
It started here , and now https://github.com/TheTom/llama-cpp-turboquant/ has it. Imagine a Q6 quant with nearly Q8 KLD/PPL.
-
reasoning-budget for Qwen3.8-27b (www.reddit.com via reddit) llamaqwen 12w
Are you using reasosning budget? I found that sometimes this model thinks so much that it triggers max output tokens limit in pi which I set to 128k tokens 😂 I saw this in the Qwen team repo: https://github.com/QwenLM/Qwen3/blob/main/docs/…
-
Looking at a PowerColor R9700 for Qwen3.8-27B, Q4_K_XL, llama.cpp/Vulkan. (www.reddit.com via reddit) llama 12w
Hi all Looking at a PowerColor R9700 for Qwen3.8-27B, Q4, llama.cpp/Vulkan. AMD's own blog quotes 51.8 tok/s but doesn't say what context length that's at, or whether MTP=2 was holding up.
-
How to run models locally on shared machine without any chat history? (www.reddit.com via reddit) llamadeepseekqwen 12w
I will be running qwen 3.8 model on a shared university machine for some research work, mostly using llama cpp but I am open to using other inference engines. I would like that there is no chat history or application logs saved on remote m…
-
I found why Qwen3.8-27B was 14× slower on my 4070 Super (www.reddit.comhttps) llama 12w
I downloaded the Q4_K_M GGUF and got 5.7 tok/s. The same file did 81.5 tok/s on a 5090.
-
Hey folks, Been doing some benchmarking on my dual-GPU setup and ran into a noticeable performance gap between LM Studio and a raw llama-server.exe build that I can't quite pin down. Also running into some confusion with how reasoning effo…
-
Benchmark results: what is the best and fastest engine to run Qwen3.8-27B on macOS (www.reddit.com via reddit) llamaqwenagentic 12w
The new Qwen 3.8 27B is fantastic for local agentic use. The problem is, what makes it so good, being a dense model, also makes it slow.
-
vLLM + Deepseek harness or hermes? qwen3.8 (www.reddit.com via reddit) vllmllamadeepseek 12w
How do you guys set it up , i constantly get the error : I tried increasing the contex to 142k and putting the contex size as 115k in DSH , it still did not compress correctly. I have 0 issues if i run it with llama.ccp , it can work for 2…
-
This is one more vibe slopped custom optimization for, in this case, my hardware (m2 ultra 60 cores, 192gb). It is just a fork from llama.cpp with a few changes, it achieves: - DeepSeek V4 Flash, no kv cache quant - 141GiB model, byte-iden…
-
I hosted Kimi K3 (2.8T parameters) using 8 B300s. 92 tok/s, $190 per million tokens (www.reddit.comhttps) vllmllama 12w
What I ran: 8x B300 on Modal, $56.79 per hour, vLLM, tensor parallel 8, native MXFP4 Cold boot ~27 min (1.56 TB load, JIT, 51 CUDA graph captures) TTFT 0.92 to 1.02 s, decode 92 tok/s steady, 83 tok/s average over 4 prompts $190 per millio…
-
AMD Users: Have you tried the llamma.cpp AMD-Ecosystem branch? Up to 2x PP Speed (www.reddit.com via reddit) moellama 12w
AMD has it's own llama.cpp branch: https://github.com/AMD-Ecosystem/llama.cpp And despite the Deprecation warning it's actively maintained (things are later upstreamed to the normal llama.cpp). What i noticed with my Strix Halo: It has som…
-
I originally followed unsloth's Q4 distribution to make an nvfp4/mxfp6/mxfp8 tri-quant, but after testing mxfp6 was faster so I made it an mxfp6/mxfp8 split. added dflash2 also quantized to mxfp6, and added mxfp8 as supported kvcache data…
-
Why does t/s go down as offload more to egpu? (www.reddit.com via reddit) moellama 12w
For moe models like Qwen3.6 35B A3B, some setups keep experts on the cpu in order to have the active weights in the gpu. But I have a strix halo + r9700, so I wondered what if I kept the experts on the strix halo and active weights on the…
-
Gemma 12B is obviously a very well trained model, I always thought the fine tuning they did on it wasn't really cut out for agentic coding. From my own experiences it struggles to use the tools it's given from Github Copilot and is also ve…
-
Question for folks with r9700 (www.reddit.com via reddit) vllmllamaagentic 12w
I have dual r9700 set up with Ubuntu. With llama.cpp I'm getting about 40 tokens per second for single instance.
-
Why is using harness slower than normal chatting? (www.reddit.com via reddit) moellama 12w
When I have a normal conversation with my Qwen3.6 35B A3B I get ~27t/s. I noticed that when I use a harness sometimes it can be slow as 15t/s (tested OpenCode and Maki), with way lower context than my normal chats.
-
Hey guys, Inco AI shipped DFlash 2 a few days ago with a drafter for Qwen 3.8 27B and a llama.cpp PR. I built the PR and ran it against plain decoding, MTP, the n-gram lookup drafters, and my July DFlash 1 numbers on Qwen 3.6 27B for 3 day…
-
I felt the need to share this here. Looking for feedback.
-
Good afternoon, everyone! I wanted to show the work I've been doing around porting Ninfer over to the CMP170HX (Github) So, first, I do want to call out the amazing work that Neroued, Sergiuszm and specifically Don-Chad have all done, to n…
-
Failed with the 3090s… I need help choosing a GPU (www.reddit.com via reddit) llama 12w
I'm building a budget AI PC for our company's application. The specs are: MSI Z370 TOMAHAWK 64 GB RAM NZXT C1200 Gold PSU 2x RTX 3090 build was finished and I started testing with llama.cpp.
-
Your own GGUF (www.reddit.com via reddit) gemmallama 12w
Hello, I have a few questions that I can't seem to find a clear answer to. Does it make sense to make your own GGUF?
-
New/Old benchmark that provides a lot of answers for local LLM (www.reddit.com via reddit) llama 12w
New/Old benchmark that provides a lot of answers for local LLM. I present to you a new test that I developed somewhat by accident: https://huggingface.co/cHunter789/Qwen3.8-27B-i1-IQ4_KS_KT-GGUF/tree/main/ctx-cliff Its original goal was to…
-
V620 Qwen 27B & Gemma A4B benchmarks (www.reddit.com via reddit) gemmallamaqwen 12w
I'm here to show some benchmarks while using llama cpp with an AMD V620 on Windows 11 via Vulkan & ROCM. The benchmarks were written out by AI, but are verified by myself to be correct.
-
Need support for llama.cpp with multi GPU (www.reddit.com via reddit) moellama 12w
Using llama.cpp I seem to be unable to get my to GPUs working tougether correclty, so I need help somehow. Setup: 96GB RAM, one Blackwell 5000 (48GB) and one 3090 (24GB).
-
Have llama.cpp spread requests across multiple cards? (www.reddit.com via reddit) llama 12w
So, with the rise of Qwen3.8-27B, I'm wondering how I can utilize my 4x32GB AMD R9700 GPU setup to best run this model. The model runs great on 1 card, but I'm wondering if llama.cpp can somehow leverage the extra cards.
-
For those who bought the CMP170HX and unlocked the memory, I have some questions… are you using any specific llama.cpp or vLLM forks? What are you using for cooling the card?
-
error en modelo ling 3.0 flash moe 124B a5B (www.reddit.com via reddit) moellama 12w
what is this error? some person know how this errors was?
-
16 GB VRAM purgatory discussion thread (www.reddit.com via reddit) llama 12w
What models and configs are we using? Please share here On windows, I am using this copium pared down model https://huggingface.co/Bucoid/Qwen3.8-27B-Uncensored-IQ4-XS-MTP-16GB-VRAM-GGUF with MTP disabled, q4 k/q4 v mmproj banished to CPU/…
-
Is there llama.cpp rendition that allow context checkpointing that is not bound by chat template? (www.reddit.com via reddit) llama 12w
Like in the topic. I'm looking for implementation similar to the flag that was removed from llama.cpp: --checkpoint-every-n-tokens x Current implementation does not work for tasks with shared base data.
-
OpenCode + llama.cpp + unsloth/Qwen3.8-27B-GGUF:Q6_K_XL = only failure so far (www.reddit.com via reddit) llama 12w
Hi, I have been using opencode with openrouter for quite a while now. Having read the success stories of using Qwen3.8-27B, I thought of trying it too.
-
Hi fellows fully-local halos, after manually following existing guides, I decided to build an LLM API endpoint installation and optimization guide that works even when autonomously followed by my pi agent, so I can install/experiment/reins…
-
Bonsai-8B is PrismML's ternary model (weights restricted to {-1, 0, +1}). The standard method for running it on a CPU is via their official llama.cpp fork.
-
Llama.cpp DSpark PC Tree Fork (up to 3%-29.5% faster!) (www.reddit.com via reddit) llamaqwen 12w
Hello gang, I made an implementation of DSpark PC Tree (Parent conditioned drafting tree). This is an implementation of this research paper: https://arxiv.org/abs/2608.02123 Unaffiliated, just found it and implemented it.
-
Recent work has shown that large language models (LLMs) exhibit strong numerical sequence modeling capabilities and show promise in time-series prediction. While LLMs display in-context learning capabilities, the mechanisms with which they…
-
GenOffice fork that works with any local LLM instead of a cloud account (github.com via hn) vllmollamallama+1 12w
GenOffice (local-LLM fork) A free, open-source AI Office suite — this fork drops the cloud-account requirement and talks to any OpenAI-compatible endpoint instead: a local server (Ollama, LM Studio, vLLM, llama.cpp server, text-generation-…
-
As AI datacenters retire functional GPUs, vast quantities of still capable accelerators enter secondary markets. This paper investigates whether these retired GPUs can find a productive afterlife to form a DumpsterCluster that can serve mo…
-
Llama-macOS – Agentic and MCP Native macOS Front End for Llama.cpp (github.com via hn) llamamcpagentic 13w
Llama Llama is a macOS menu bar app for running local LLMs. Watch a 2-minute intro 📽️ Install brew install --cask llama-app Or download from Releases.
-
Local+privacy+focus on cache AI CLI and llama.cpp branch + Qwen3.x (github.com via hn) mistralllama 13w
privibe CLI coding agent for private, local-first development. privibe is a fork of Mistral Vibe reworked to not do any call back home of any kind and then run against local models first, I want to be able to use it and know it will not be…
-
Show HN: GPT, Claude, Gemini. Ask them all. Pay for one (whizi.io via hn) llamagemini 13w
Ayman Aboskhroun · Accountant I’ve been using Whizi for my accounting work at the office for the past year, and the results have been unlike anything else. Whizi · The multi-model AI workspace GPT, Claude, Gemini, Llama, and top image-gene…
-
DLLM: Minimal, clean coding agent built directly on llama.cpp without overhead (github.com via hn) llama 13w
DLLM - D Language 🤖 on 🦙.cpp A minimal, clean D language agent built directly on llama.cpp via importC. No Python, no bindings, no overhead.
-
Ask HN: Are third party gguf/mmproj safe on Llama in production env (news.ycombinator.com) gemmallama 13w
So i had been building a side project screenmind ,sort of local ai desktop assistant, and hit a confusing multimodel failure... texts worked,but vision and image failed no crash just returning <unsused49>.
-
Khatri et al. (2026) [DOI: https://doi.org/10.1109/DSN-W70714.2026.00027] show that lightweight MLP probes on final-layer activations of a single 8B model (LLaMA-3.1-8B) detect harmful prompts at F1 competitive with guard models 1000x larg…
-
Zuck rekindles open weights Llama drama with Muse Glimmer (www.theregister.com via hn) llama 13w
MOST POPULAR AI - AI and ML The future is for billionaires – the rest of us will get open weight AI models, maybe Mark Zuckerberg muses about 'superintelligence' and 'arc of human civilization' - AI AND ML Zuck rekindles open weights Llama…
-
Meta is back with Muse Glimmer: local, agentic, multimodal, and open source! To celebrate, we are shipping with Meta day-0 support in transformers , llama.cpp , vLLM , Inference Endpoints, and other libraries.
-
The Llama Tests (www.llamatests.org via hn) llama 14w
The LLAMA tests A suite of language learning tests. The LLAMA test battery consists of four tasks.
-
Local model in tandem improves responses (www.reddit.com via reddit) llama 14w
Claude fabricated a timestamp on me. I was pissed off.
-
Building a Rust Inference Engine That Matches Llama.cpp (www.fratepietro.com via hn) llama 14w
Ferrox: Building a Rust Inference Engine That Matches llama.cpp I’ve spent the last few days building Ferrox, a pure-Rust inference engine for running open LLMs locally — dense models and Mixture-of-Experts, on CPU, Apple Metal, or CUDA. N…
-
Homebench – Benchmark local LLMs for speed, memory, and quality (github.com via hn) ollamallama 14w
homebench **Benchmark the local LLMs you already have — speed, memory, and quality — as a live terminal leaderboard. homebench is a single-command TUI that discovers the models installed in your local runner (Ollama, LM Studio, llama.cpp,…
-
Show HN: Gainz.fast – Local Inference, Faster (gainz.fast via hn) vllmllama 14w
Come help push the frontier of token speed across local models and hardware with your agents! Current frontier Laguna XS 2.1 · AMD R9700 (llama.cpp HIP) +31.14% 143.3 tok/s Laguna XS 2.1 · DGX Spark GB10 (vLLM NVFP4) +5.28% 37.3 tok/s Lagu…
-
In this work we investigate how decoder-only transformers resolve lexical ambiguity through layer-by-layer analysis of three models spanning three parameter sizes (GPT-2-Small-117M, Llama-3.2-3B, Qwen2.5-32B). For both homonyms and polysem…
-
Show HN: NightRun, bare metal LLM inference, no OS, boots from USB (github.com via hn) llama 15w
A local LLM runtime that boots from USB and runs without a conventional operating system. Real boot, one cut: loading and prefill sped up, generation at actual speed (Llama 3.2 1B, QEMU/KVM, 8 cores).
-
Large Language Models (LLMs) are temporally overexposed: trained on vast contemporary corpora, they encode present-day concepts that make them unreliable narrators of the past. We present TimeCapsule, a 1.2B-parameter LLaMA-style causal mo…
-
Show HN: Minute – Offline meeting notes on macOS with Whisper and llama.cpp (github.com via hn) llamacodex 15w
Hello HN, I built Minute because I wanted searchable meeting notes without sending recordings or transcripts to a cloud service. It captures microphone and optional system audio, transcribes locally with Whisper, and generates summaries, d…
-
Mixture of experts (MoE) architectures have become a cornerstone for scaling up and are a key component in most large language models such as GPT-OSS, DeepSeek-V3, Llama-4, and Gemini-2.5. However, systematic research on MoE remains severe…
-
Deploying large language models in financial-services and agentic settings requires safety classifiers that simultaneously handle prompt injection, regulatory compliance, and general harm, a combination no existing open guardrail addresses…
-
All major LLMs are lib-left. Even Grok, half the time (unslop.run via hn) mistralglmgpt-5+6 15w
I ran the 62-item politicalcompass.org test 30 times each on sixteen models: OpenAI's GPT-5.x and GPT-4o, Claude, Gemini, Grok, Llama, Mistral, and China's DeepSeek, Qwen, Kimi and GLM. Fifteen land in the libertarian-left quadrant.
-
The capabilities of generative AI in mathematics education are rapidly evolving, posing significant challenges for research to keep pace. Research syntheses remain scarce and risk being outdated by the time of publication.
-
Show HN: Lexicon a free Grammarly alternative, local writing assistant (lexicon-writer.pages.dev via hn) ollamallama 16w
Lexicon is a rich-text editor with grammar checking and AI writing tools (rewriting, tone shifting, and summarizing) that run entirely on your machine. No account, no API calls, and nothing uploaded.
-
Show HN: I built a hypervisor and client for inference on consumer compute (scalattice.com via hn) gemmallamadeepseek+1 16w
I'm the founder of Scalattice, this is my second company, third total product. I'm a 2x founder building some challenging software, some easy software, and some curiosity based tools that I've just always wanted to be a part of!
-
Ask HN: HotPin – lossless 120B MoE inference on 24GB RAM (CPU, 50 loc) (news.ycombinator.com) mixture-of-expertsglmmoe+1 16w
I'm a mechatronics designer with a background in control systems, robotics, PCB design, and embedded hardware. I design physical systems: motors, sensors, microcontrollers, and real-time control loops.
-
TurboPrefill: 3.27× Prefill Speedup in Llama.cpp (devpost.com via hn) llama 16w
Inspiration Large language models running across multiple GPUs are often limited not only by computation, but by communication between GPUs. This becomes especially visible on systems without NVLink or with slow PCIe connections.
-
Speculative Decoding: The Free Speed Toggle Your Local LLM Is Probably Not Using (vettedconsumer.com via hn) llama 16w
There is a setting in LM Studio and llama.cpp that can make a local model generate 1.5 to 2.5 times faster without changing a single word of its output. Not a lower quant, not a smaller model, the same model producing mathematically identi…
-
Mean cross-positional attention degradation is widely reported in transformer interpretability, yet whether it causally limits contextual retrieval remains untested. We present six coordinated experiments across GPT-2, LLaMA-3.2-1B/3B, OPT…
-
Frontier-class LLM inference on a laptop CPU (github.com via hn) llama 16w
cpubrrr https://github.com/user-attachments/assets/a390cb6e-86b8-41e5-9ef6-957c94dabe19 From-scratch CPU-only LLM inference that beats llama.cpp's CPU path — on both quant formats it runs, no GPU. cpubrrr is a research runtime that runs fr…
-
Show HN: Free API keys for open models (Llama, Qwen, GPT-OSS, Gemma) (app.tensormux.com via hn) gemmallamaqwen+1 16w
Open models behind one OpenAI-compatible endpoint. No account yet?
-
Show HN: Diktafon – Voice memos on cassette tapes, transcribed on-device (github.com via hn) llama 16w
Diktafon is a voice-memo app: each topic is a cassette, memos are appended to its tape, and a cassette plays back as one continuous recording. Every memo is transcribed and every cassette summarised, fully on-device: whisper.cpp for transc…
-
Show HN: Llamatop - see what the various cores on your macbook are doing! (github.com via hn) llamacodex 16w
I had codex make this for me so I could see what llama.cpp was doing on my mac. Vibecoded in swift, hit 1 for more processor details, hit m for more memory details.
-
Show HN: A fast, free AI text humanizer powered by Groq Llama 3.3 (zlvox.com via hn) llama 17w
could not extract summary
-
Show HN: Run a 120B-parameter MoE on Android mid-range phone CPU-only llama.cpp (github.com via hn) moellama 17w
BigMoeOnEdge Run Mixture-of-Experts models far bigger than your edge device's RAM. The result: a ~60 GB model on a 12 GB phone: 1.3 tok/s lossless, byte-identical to running from RAM, 2.2 tok/s with one speed knob.
-
"recipes" for reliable code work with Qwen3.6-27B? (www.reddit.com via reddit) llama 17w
so part of what inspired my benchmark post was that it does seem like folks here are generally converging on "unless you are able to operate at very large scales with a lot of system RAM and VRAM, the best model for code work is generally…
-
Maintainer here. A live, in-browser demo of Reame, the CPU-first inference server I've been building on llama.cpp — and I think the numbers are fun for this sub.
-
Dual Xeon T7920 Server - Useful for LLMs? (www.reddit.com via reddit) llama 17w
I have a Nimo Strix Halo system up and running, it's not the fastest, it can't run the biggest open weight models or high quants of 100-200b models, but it works really well for most of my use cases. I'm a hardware and signal/power integri…
-
LLM Red Team Lab A hands-on kit for educational, authorized red teaming of any locally-run LLM. It works with any OpenAI-compatible model — Llama, Mistral, Qwen, Gemma, DeepSeek R1, and more — and covers the two ways an LLM system gets exp…
-
Darby is an open source Hugo docs theme with the polish of the paid docs platforms: clean typography, dark mode, full-text search, code blocks with copy and filename tabs, callouts, tabs, beautifully rendered mermaid diagrams, auto sidebar…
-
Show HN: Trellis2.c – Local 3D generation with Vulkan and CUDA (github.com via hn) llama 17w
Hi HN, I started this because I wanted something in the spirit of llama.cpp and stable-diffusion.cpp, but for local 3D generation model: native executables, no Python/PyTorch runtime, and both CUDA and Vulkan backends. I’ve mainly tested i…
-
Fusing a 27B ternary LLM's whole decode step into one CUDA kernel (twitter.com via hn) llama 17w
i open-sourced bonsai-turbo -- a batch-1 decode engine that runs @PrismML's Bonsai 27B 1.76x faster than the official llama.cpp fork. same outputs, token for token H100, tg128, greedy: ternary 85.5 >> 151 tok/s.
-
Hedos – Local models in one native Mac app (llama.cpp, MLX, diffusers) (www.hedos.ai via hn) ollamallama 17w
Local-first · Private · Open source One native home forevery local model on your Mac Every text, image, and speech model already on your Mac — pulled through Ollama, cached by Hugging Face, or dropped in by hand — discovered and run in one…
-
Show HN: InferAll – One API for OpenAI, Anthropic, Google, Nvidia Nim (inferall.ai via hn) llamaopenaianthropic 17w
Open models at $0 rate 118+ open-source models hosted on NVIDIA NIM — Llama 3.1 70B, Mixtral, Nemotron, CodeLlama — bill at our open-model rate ($0 input, $0 output) once your account is activated with the $5 starter pack. InferAll is an A…
-
Decoding continuous language from fMRI signals remains a core challenge in non-invasive brain-computer interface research. We present two complementary investigations.
-
Cross-encoders achieve high reranking accuracy in Retrieval-Augmented Generation (RAG) pipelines but impose quadratic inference costs that limit real-time deployment. We address this by fine-tuning LLaMA 3 (8B) as a drop-in reranker using…
-
Cicada- an agentic Python IDE Free to use ( comes with built in small model) (github.com via hn) llamaagentic 17w
Cicada An agentic Python IDE that turns plain-English requests into runnable, executed code — powered entirely by a local model. local-llm · agentic-ide · electron · python · llama-cpp · gguf · code-generation · monaco-editor · machine-lea…
-
Show HN: Low-latency local LLM runner via OpenJDK Panama FFM (Java 22) (github.com via hn) llama 17w
I wanted to run AI from inside the JVM. I started out with the standard REST sidecar, ripped that out to use Project Panama (Foreign Function & Memory API) in the new JDK versions to interface directly with llama.cpp.
-
The multiplicative repetition penalty shipped across the LLM inference ecosystem (HuggingFace, vLLM, llama$.$cpp, and a dozen further engines) branches on the sign of each raw logit (divide positives by theta, multiply negatives). But the…
-
Ollama vs. Llama.cpp – Quick Benchmark (blawg.pages.dev via hn) ollamallama 18w
Ollama vs llama-server — Quick Benchmark I got myself a Tesla V100 a while ago and thought why not put it to some good use for once: Curious about the feasibility of the somewhat controversial ollama in comparison to straight llama-server…
-
LLMlet: P2P distributed LLM inference on browsers (github.com via hn) llama 18w
LLMlet: P2P distributed LLM inference on browsers LLMlet is an on-browser LLM runner based on Wasm-compiled llama.cpp. Distributed LLM inference among browsers connected via WebRTC using PeerJS.
-
Show HN: Reame – a CPU inference server that gets faster as it runs (github.com via hn) llama 18w
A lean, fully-tested LLM inference server built on llama.cpp — designed for the hardware you already have: shared vCPUs, free tiers, 2-core ARM boxes. Reame is not the first inference server.
-
Show HN: Willow Voice – Free AI Dictation (willowvoice.com via hn) llamaagentic 18w
Hey HN! We were part of YC S24, ended up pivoting around.
-
Show HN: I built a free app for New Yorkers to save money on groceries (sbnyc.app via hn) llama 18w
I built this because I see that grocery savings are achievable in NYC. People usually just go to the store they're used to going to, and it's rarely worth the effort of combing through card cashback, weekly coupons, CPG rebates.
-
Show HN: Sell your unused AI Credits or buy Claude credits for 50% off (secondhandtokens.com via hn) llamadeepseek 18w
Buy unused API tokens from other developers at 50% off. Claude, Llama, and DeepSeek — same models, half the cost.
-
Large Language Models (LLMs) and generative AI (GenAI) systems, such as ChatGPT, Claude, Gemini, LLaMA, Copilot, Stable Diffusion by OpenAI, Anthropic, Google, Meta, Microsoft, Stability AI, respectively, are revolutionizing cybersecurity,…
-
Llama 2 LLM on DOS (2025) (yeokhengmeng.com via hn) llama 18w
Ever thought of running a local Large Language Model (LLM) on a vintage PC running DOS? Now you can!
-
Show HN: I ported llama.cpp to Apple Watch and ran a 0.8B LLM locally (news.ycombinator.com) llama 18w
could not extract summary
-
Webnix AI assistant run on mobile no cloud no server (www.reddit.com via reddit) llama 18w
Webnix AI It's a fully offline, on-device AI assistant for Android — no cloud, no API keys, no network required. Llama 3.2 1B and GTE-Large both run locally on-device via the QVAC SDK, powering private chat, file indexing, and note search…
-
ZML/LLMD alpha – cross platform LLM server (zml.ai via hn) mistralgemmallama+1 18w
Today we’re releasing ZML/LLMD. It’s a self-contained inference server that runs LLaMa, Gemma, Qwen and Mistral LLMs transparently on 5 architectures: NVIDIA CUDA, AMD ROCm, Google TPU, Intel oneAPI and Apple Metal.
-
Any good uncensored AI models? (www.reddit.com via reddit) llama 18w
When I say uncensored, I mean uncensored. Not NSFW.
-
Show HN: Rewire Text – system-wide text transforms with BYOK and local-models (sunsetmesasoftware.com via hn) ollamallamagemini+2 18w
I just launched Rewire Text, a Windows + macOS tool that transforms text in any app at the press of a hotkey. Sits in the menu bar / system tray until needed.
-
What I haven’t made with Fablo (www.reddit.com via reddit) glmllamarag+1 18w
Due to the guardrails, I’ve never been able to run start to finish in a session without triggering the safety and switching to opus. This is across platforms and without custom instructions + clean Claude.md… heres all the things that were…
-
Show HN: Gemma 3 inference in pure C++ with Metal acceleration (github.com via hn) gemmallama 19w
MetalChat - LLM inference for Apple Silicon MetalChat is a Metal-accelerated C++ framework and command line interpreter for inference of Meta Llama and Google Gemma models. [!IMPORTANT] The library API and CLI are under active development,…
-
Can you use Claude Desktop (code) but use local API? (www.reddit.com via reddit) vllmollamallama+1 19w
Hey all, So I tend to favor the Claude Desktop app in Code mode as the GUI does a great job of previewing code, MCP browser interactions/screenshot evals/etc. But I recall people saying they could get Claude Desktop to use a local API.
-
We study inference-time pattern-memory gating in a production-scale clinical natural language processing (NLP) pipeline. The pipeline pairs a generator (Llama-3.3 70B) proposing extractions with a verifier (MMed-Llama-3.1 70B) accepting or…
-
Show HN: Voice-to-SQL – ask a database in plain English (LLM → SQL) (huggingface.co via hn) llama 19w
I built this and it is open source. You ask a business question in plain English and an LLM (Llama 3.3 70B via Groq) turns it into SQL and runs it against a sample SaaS database — read-only, SELECT only.
-
K.A.S — Kasra's Agentic Shell. Run open models locally — on Apple Silicon (MLX) or NVIDIA (llama.cpp/GGUF) — behind an Anthropic Messages-compatible server, driven by an agentic TUI.
-
EdgeSync-LLM – KV cache fragment engine for on-device LLM inference (Go/Android) (github.com via hn) llama 19w
EdgeSync-LLM — KV Fragment Engine for Local LLMs A engine-agnostic KV cache fragment system for on-device LLM inference. Designed for ARM64 Android (Cortex-A55/A78), portable to any platform running llama.cpp, MLC-LLM, or ONNX Runtime.
-
Show HN: Llama Legends (llamalegends.com via hn) llama 20w
I've been working on Llama Legends for half a year now, mostly evenings and weekends alongside my day job. It's a Gacha/Cardgame/Idle RPG that runs in the browser.
-
Link: https://ai-values.com/ There is a small 15 question quiz you can take before taking the full big quiz. The results of the big quiz update in realtime as you go so you dont have to actually go through all the questions (but they do ge…
-
Show HN: role-model, a router for hybrid local/cloud AI (github.com via hn) llama 20w
Hey everyone, I'm launching role-model today: a routing protocol, a reference router runtime, and an extension for Pi that allows for better informed routing decisions. role-model is mostly deterministic, with fallback to a controller mode…
-
Tiny LLM Benchmark: Jetson Orin Nano Super 8GB (www.smolhub.com via hn) ollamallama 20w
Tiny LLM Benchmark: Jetson Orin Nano Super 8GB 8 tiny LLMs benchmarked across 4 power modes on Jetson Orin Nano Super 8GB: llama.cpp vs Ollama. 25W sweet spot: 43% more tok/s than 15W, better tok/J than MAXN.
-
I patched llama.cpp to gain 20% prompt processing TPS. Help me make a PR (news.ycombinator.com) moellama 20w
I've been running Qwen3.6-35B-A3B locally on llama.cpp and noticed that prompt processing throughput gets too low with MTP. I got nerd-sniped.
-
Ask HN: How is GPU power draw measured at scale? (news.ycombinator.com) mistralgemmallama 20w
How do people measure power usage of GPUs at large (32x) self-hosted setups or small multi-rack setups? I've seen some PDUs which collect and transmit data, but I'm unsure of the processes and if/how people do this on small builds.
-
GLM 5.2 on consumer hardware (www.reddit.com via reddit) glmllama 20w
I tried out the unsloth quants of GLM 5.2 on still "consumer-ish" hardware: 32C Zen5 Threadripper Pro 9975 WX, Asus WRX90E-SAGE-SE PCIe Gen5, 512GB DDR5 ECC RAM @ 4800MHz, dual RTX 5090. This machine was put together pre-RAMpocalypse, and…
-
Tensor Split Fix for intel GPU's llama.cpp release b9788 (www.reddit.com via reddit) llama 20w
sycl : support --split-mode tensor #24152 I'd like to see some numbers if anyone has 2xintel gpus and tries this out
-
I was curious why MTP affects PP TPS in llama.cpp. My PoC recovers it? (news.ycombinator.com) moellama 20w
I've been running Qwen3.6-35B-A3B locally on llama.cpp and noticed that prompt processing throughput gets too low with MTP. I got nerd-sniped.
-
I read it with a little bit of effort The tiny model result is insane, theoretically this could make make a 0.5b on-par with a 2/3/4b ish class model in coding with no weights change*. And for large models it could maybe fix let's say 30-5…
-
Show HN: MinLlama – Llama 3.2 inference in ~100 lines of NumPy (github.com via hn) llama 20w
I built minLlama because I wanted a Llama implementation that was easy to understand and hack for KV cache compression research. There is also a PyTorch and Jax version in ~140 lines.
-
Show HN: Loqi, a "local-first" translation tool using Ollama/llama.cpp (github.com via hn) ollamagemmallama 20w
I got tired of sending every text I translate to Google/DeepL. Even with all the opt-out options and privacy policies, it never felt right especially for some work documents, personal writing, or anything sensitive.
-
Llama-dash – One go-to control plane for local inference (llama-dash.dev via hn) llama 21w
gateway online|running 3 · peer 2|req/s 0.87 LOCAL INFERENCE CONTROL PLANE One control plane for local inference. Monitor models, requests, API keys, routing rules, and proxy metrics from one dashboard for llama-swap and compatible upstrea…
-
Second Brain – A free, invisible AI interview copilot (Groq and Llama 3) (github.com via hn) copilotllama 21w
Second Brain 🧠 Your Invisible AI Interview Assistant English | Português | Español | Français | Italiano Second Brain is a 100% free, invisible desktop assistant built to help professionals worldwide ace their job interviews. It runs quiet…
-
Show HN: Selora – local model for Home Assistant (github.com via hn) llamaqwen 21w
Selora AI Local is an open-source, Qwen-based model for Home Assistant. Specs: Qwen3 1.7B base model (Q6 quantized~1.6GB) Four Home Assistant-specific LoRA adapters: - Answers - Clarifications - Automations - Commands ~3.5 GB total downloa…
-
Behavioral audits of Large Language Models on moral prompts measure what the model says, not the internal computation producing it. We use Transluce, an AI-driven mechanistic-interpretability platform, to examine LLaMA 3.1-8B-Instruct on 5…
-
What are you looking for when reviewing LLM generated code? (news.ycombinator.com) llamaclaude-code 22w
For work I often have to review multi thousand file changes in PR mostly generated by AI/Llama/claude code. It's mainly rust with many unit tests, so there is a lot tooling helps verify the code already to help a lot.
-
Small LLMs: Pruning vs. Training from Scratch (arxiv.org) llama 22w
Pruning promises a shortcut to strong small language models. In this work, we examine this promise by pruning Llama-3.1-8B at pruning ratios of 0.5--0.8 with six methods spanning depth, width, and sparse granularities, under two controlled…
-
Structured width pruning of GLU-MLP layers in Llama-3.2 models, guided by the Peak-to-Peak Magnitude (PPM) criterion, reveals a systematic dichotomy in how reducing the expansion ratio affects different model capabilities. While performanc…
-
Llama.cpp – Run LLM Inference in C/C++ (llama-cpp.com via hn) llama 22w
Llama.cpp (LLaMA C++) allows you to run efficient Large Language Model Inference in pure C/C++. Download llama.cpp for Windows, Linux and Mac.
-
How to Setup a Local Coding Agent on macOS (ikyle.me via hn) gemmallama 22w
How to Setup a Local Coding Agent on macOS Running Gemma 4 26B-A4B and Qwen3.6 35B-A3B locally with llama.cpp, MTP speculative decoding, multimodal support, and PI as a coding agent. I'd had my internet fail a few times recently leaving me…
-
Introducing: DNR-Bench: Do-not-respond Benchmark (www.reddit.comhttps) mistralgpt-5llama+5 22w
Single-item benchmark. One prompt, loaded from questions.txt: Scoring: empty completion = pass, any token (including reasoning) = fail.
-
LLM for the ESP32-S3 (github.com via hn) llama 22w
esp32-llm-pipeline One language model, two microcontrollers. A Llama-architecture LLM running with its layers split across two ESP32-S3 boards — per token, the activation vector crosses three wires (CRC-framed UART) between the chips.
-
This study examines training dynamics in a small Llama-style language model trained under a fixed, compute-constrained token budget. Rather than evaluating efficiency solely through endpoint performance, the study uses a quantitative exper…
-
250+ t/s on a single RTX 5090: Running Google’s DiffusionGemma MoE locally via llama.cpp (www.reddit.com via reddit) moellama 22w
So Google’s new DiffusionGemma-26B-A4B-it is pretty wild. It's a discrete text diffusion model, which means instead of generating one token at a time autoregressively, it predicts and refines a whole block of up to 256 tokens in parallel.
-
advice for dual-gpu asymmetric (www.reddit.com via reddit) gemmallamaqwen 22w
Hello everyone, i had a 3080ti 12gb and added a 3080 20gb, so it has a bit less speed but more memory than my main card. I could finally get some speed with the usual suspects (i am testing gemma 4 31b/26b-a4b and qwen 3.6 27b/35b-a3b), BU…
-
Any chances for a 12B diffusion Gemma? (www.reddit.com via reddit) gemmallama 22w
Currently recompiling my llama.cpp with support for diffusion Gemma, but I know on my hardware it won't likely be all that viable. I feel like if the goal was to take better advantage of consume GPUs for fast, intelligent generation, build…
-
DiffusionGemma 26B A4B results on my 5090 (www.reddit.com via reddit) llamadeepseek 22w
# DiffusionGemma 26B A4B — Tuning Results (note: these are my tuning results but Deepseek assisted in generation of testing scripts and reports) https://huggingface.co/unsloth/diffusiongemma-26B-A4B-it-GGUF System - **GPU**: RTX 5090 (32 G…
-
Can't seem to enable reasoning in llama.cpp (www.reddit.com via reddit) llama 22w
Hi, I'm trying to use some LLMs which I know support reasoning (TheDrummer Rocinante X 12B model) but I can't for the life of me to get it to work. I've tried using all these parameters: --chat-template-kwargs '{"enable_thinking":true}' --…
-
Show HN: EzziwebTools – A collection of 30 free tools (ezziwebtools.com via hn) llama 22w
Hi HN, I wanted to share EzziwebTools, a hub of 30+ free tools. The site is built with Next.js and React, and includes both standard utilities and a few AI-powered tools (using Groq/Llama 3), such as: An API Response Time Comparator (visua…
-
How do i prevent llama.cpp from offloading on Swap? (www.reddit.com via reddit) llamaqwen 22w
I have tried preventing this issue by using llama.cpp flags. However, I still have the issue: whenever I'm close to my 96GB of RAM, llama-server / llama.cpp decides to offload the KV cache onto my swap.
-
Ask HN: Any Local LLM can I run without GPU for Local Agentic workflow AI? (news.ycombinator.com) llamaagenticclaude-code 22w
Claude Code like agentic workflow ai too costly for me.Any LLM can I run with VSCode at the below setup? 16ram Intel core i7 h processor 13gen 512gb NVMe SSD I want to run the ai as local agentic workflow with Vscode.I want use LLAMA agent…
-
MTP hyperparameter search (www.reddit.com via reddit) llamaqwen 22w
TLDR; I only got a 6% improvement on tokens/sec over naïve parameters. I was messing around and ran a hyperparameter search with optuna over the MTP and speculative decoding options of llama-server for Qwen3.6 27b on strix halo.
-
TLDR: I just added an MCP to the Observer framework making it 10x easier to use, so you can create micro-agents that monitor your screen autonomously, literally one sentence and you're done! So just typing "Monitor my Steam download and se…
-
Executing a plan under context constraints (www.reddit.com via reddit) llamaqwen 22w
I'm running Qwen 3.6 35B-A3B via Pi harness on a 32gb unified RAM setup (Framework 13). llama.cpp, 64k context window.
-
Harnesses seem to have an issue. (www.reddit.com via reddit) llamaqwenopus 22w
There's a post i saw about Claude Fable where a user asked the model the car wash question and it sent me down a rabbit hole. I spun up qwen on llama.cpp and in the llama.cpp chat interface I asked the model and it got it right consistentl…
-
Need help improving speed of inference (www.reddit.com via reddit) llamaqwen 22w
Hello i'm running the qwen 3.6 27b in ud q5k xl, and with all the optimizations it barely fits in my 3090 vram with a 120k context, i'm sure it does not spill when context is full but i would like to improve the token generation speed. I w…
-
Diffusion Gemma 26B MOE (www.reddit.comhttps) gemmamoellama 22w
Pretty exciting, wonder what it will take from llama.cpp to get it working locally
-
Pi + Docker Sandbox + llama-server setup guide (www.reddit.com via reddit) llama 22w
Hi everyone, I put together a guide for running Pi securely inside a Docker Sandbox while running llama-server directly on my host machine. The goal was to keep the agent isolated while still running the model on the local GPU at full spee…
-
1-bit and 1.58 bit LLM Benchmarking on Jetson Orin Nano Super | Bonsai LM (www.reddit.com via reddit) llama 22w
Bonsai LM (1-bit and 1.58-bitLLMs) benchmark on Jetson Orin Nano Super Just released a deep benchmark of 5 Bonsai LM models (1.7B → ~8B) on a $250 Jetson Orin Nano Super 8GB using llama.cpp CUDA - across all 4 power modes: 7W, 15W, 25W, an…
-
Hey everyone, I'm running Qwen3.6-MTP-27B-MTP (Q4_K_M) with llama.cpp server on a Tesla V100, and I'm currently getting around 55 tokens/sec. I'm trying to find out whether there are any configuration changes that could increase throughput…
-
Can you really replace paid models with a local model? (www.reddit.com via reddit) minimaxglmllama+2 22w
Long time lurker, and I say this as someone who genuinely loves this community and runs many local models myself. I’ve been using LLMs since the early GPT and LLaMA days.
-
The 'storage tax' on cloud GPUs for short LLM runs is brutal. What's your workflow? (www.reddit.com via reddit) clinellamaagentic 22w
I’m trying to test Qwen3.6-27B for agentic coding through Cline / llama.cpp, but my local box struggles once the context gets longer. (my poor 3080 just can't keep up).
-
What's up on CPU inference these days? (www.reddit.com via reddit) moellama 22w
What are the best models, quants and llama.cpp versions/forks for CPU inference these days? I have AVX2 but no AVX512 - Intel core ultra 7 165H; 64G RAM This seems to ask for massive MoE (a lot of RAM, not a lot of bandwidth/compute).
-
unsloth/North-Mini-Code-1.0-GGUF · Hugging Face (huggingface.co via reddit) llama 22w
GGUF for the new Cohere 30B A3B model I haven't had a chance to test this yet, but I think it's related to https://github.com/ggml-org/llama.cpp/pull/24260
-
-
How I got inspired to build a version manager for llama.cpp (www.reddit.com via reddit) llama 22w
Hey everyone, I wanted to share a little side project I cooked up over the last week. So, long story short, I only started diving into the LLM world in February, and honestly, it’s been a wild ride.
-
Furiosa AI selling inference chip to consumer market will be a game changer to local llm (techcrunch.com via reddit) llama 22w
This is south Korean start up all-in on inference chip: https://furiosa.ai/renegade-spec Tsmc 5nm node Hynix HBM3 1.5TB/s 48GB VRAM TDP 180W Already tested on LG LLM. If they opened their programming interface the way NVIDIA opens PTX an…
-
OSCAR 2-bit KV on Windows/Nvidia? (www.reddit.com via reddit) llama 22w
Hey guys, Has anyone gotten the new OSCAR 2-bit KV cache fork running locally on Windows/Nvidia yet? Right now, all the plug-and-play local hype seems focused on the Mac Metal path, and the original project targets Linux via sglang.
-
I took the liberty to test both models today on my favorite benchmark question, head to head. Device: Apple Mac M3 Max 64GB Environment: llama.cpp, all defaults Gemma4-12B's token generation speed: 47 tps with MTP and 2 predicted tokens 29…
-
Jetson Orin NX Build for Hermes Agent + Benchmarking (www.reddit.com via reddit) gemmamoellama+1 22w
I had a huge LLM server, and now I have a tiny one! I had a Jetson Orin NX gathering dust from a long dead robotics project, from back in the Llama-7B days.
-
Has anyone tried running retrieval inside the model, not before it? (www.reddit.com via reddit) llama 22w
Been messing with a bolt-on refiner block for small models. Insert a small trainable transformer layer at the midpoint of a frozen base model, loop it 2-4 times over the hidden states.
-
Anyone else find local search becoming the bottleneck once your LLM setup gets fast enough? (www.reddit.com via reddit) llama 22w
Got my Llama 3 setup humming along on a 4090, inference is snappy, but retrieval became the hell. Running semantic search over a decent-sized document corpus and the latency gap between "model thinking" and "model waiting for context" star…
-
Pursuit of performance Llama.cpp to MLX (www.reddit.com via reddit) llama 22w
Right now, I am running llama.cpp on a M2 ultra 64gig. Having great fun with unsloth/Qwen3.6-35B-A3B-GGUF:UD-Q8_K_XL - Running opencode and finding it amazing to have such great tools running locally.
-
Financial transaction processing requires extracting structured merchant information from noisy, abbreviated bank transaction strings at scale. Our current production system, a LoRA-fine-tuned LLaMA 3.1-8B, achieves 96.95% F1 on this task,…
-
Jetbrains Mellum 2: a really good and performant model (www.reddit.com via reddit) moellama 22w
Oh Hey Folks, I took the Mellum 2 model for a spin, so I wanted to share my impressions here. Disclaimer: the tests presented here are not cientific nor have those nice names like perplexity,etc.
-
Here's a llama.cpp CLI Command builder. (llamabuilding.com via reddit) llama 22w
No accounts or sign up. No email requirements.
-
Pipeline parallelism in llama.cpp may be wasting your VRAM (www.reddit.com via reddit) llama 22w
By default, llama.cpp enables pipeline parallelism, presumably to speed up inference. In my testing, I found that pipeline parallelism has no speed benefit and comes at a significant cost of VRAM.
-
Quick note on the QAT of recent (www.reddit.com via reddit) llama 22w
tldr: Googles quant is broken, use unsloth UD Q4_K_XL for now This might be low quality post, but oh well, we ball llama-quantize will quant the token embed to q6k when Google really was supposed to use "--pure" but that’s only the first p…
-
LMStudio gemma 4 31b QAT with MTP (www.reddit.com via reddit) gemmallama 22w
Did anyone manage to launch that in LMStudio? I am on the most recent update with the most recent llama.cpp available in LMStudio.
-
Me: Arguing with an AI bot who just posted something on this sub about Llama 3.1. (www.reddit.comhttps) llama 22w
For real tho, these bots need to turn on their web search functions and quit living in the past. It’s bad enough we gotta deal with all the “Qwen3.6 27b helped me quit drinking and brought my dog back from the dead” posts.
-
Can't get beyond 8t/s with NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16 (www.reddit.com via reddit) llama 22w
I am running nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16 in an unsloth UD-Q6_K_XL quant (unsloth/NVIDIA-Nemotron-3-Ultra-550B-A55B-GGUF) on a dual 5090 Zen5 32C Threadripper Pro Workstation with 512GB DDR5 ECC RAM and a PCIe Gen5 capable…
-
Gemma 4 MTP with assistant vs llama cpp type MTP (www.reddit.com via reddit) gemmallama 22w
Hi all Been loving the QAT models but honestly what is up with the assistant models, any ggufs and ways to make em work with vanilla llamacpp and if this way of MTP is different than the one am17an developed for llamacpp. Followup question…
-
Latam GPT 1.0 released (www.reddit.com via reddit) llama 22w
https://huggingface.co/latam-gpt/Llama-3.1-70B-LatamGPT-SFT-1.0 Latam GPT is an AI model trained on latin american data. It's part of an initiative to create AI that works better in Latin America than Chinese or American models.
-
Gemma 4 QAT + MTP: max 33% speed increase in token generation, any ideas? (www.reddit.com via reddit) gemmallama 22w
-
llama-launcher Release (www.reddit.com via reddit) llama 23w
Hello everyone, I've been working on a point and click GUI to make tinkering with llama-server flags much quicker and easier, I thought I'd share for anyone else who might be interested. It's also great for anyone new to llama.cpp that is…
-
AutoMB – a CLI that brings 150+ AI commands, agents, and advisors to your terminal (www.reddit.com via reddit) ollamallamachatgpt+2 23w
-
-
Meta Abandons Llama for Muse Spark — The End of Open-Source AI's Biggest Champion (www.reddit.com via reddit) gpt-5llama 23w
Meta has officially abandoned its open-weight Llama family in favor of Muse Spark — a fully proprietary model built by Alexandr Wang's MSL team. The Llama era is over.
-
Windows keeps crashing on rtx 3090 (www.reddit.com via reddit) llama 23w
Recently bought used 3090. Under heavy stress tests and gaming it's fine.
-
The GPUless Revolution: How Efficient AI Models Are Democratizing Artificial Intelligence (www.reddit.com via reddit) llama 23w
You don't need a $10,000 GPU to run state-of-the-art AI anymore. The latest breakthroughs in model quantization and optimization are putting powerful AI in the hands of everyone—from hobbyists to small businesses.
-
We ask whether topic sentiment has a causal effect on perceived political ideology, and whether the answer depends on who assigns the ideology label. Using articles from AllSides, paired with shared sentiment annotations from Llama-3.3-70b…
-
Built a small Android app called Pocket Node that runs llama.cpp inference on-device. Here's what it actually does and what it doesn't.
-
Running into something annoying with llama-server in router mode (`--models-preset`) and I can't tell if I'm missing a flag or if this is just how it works. My rig is 2x 3090, 2x 4060 Ti (one's unplugged at the moment, riser got repurposed…
-
MTP and QTA - what is the relation? (www.reddit.com via reddit) llama 23w
I'm an old guy and I hate when things change so fast surrounded by noise and breaking news! MTP, I know what the acronym means and where it excels.
-
QAT variant of Gemma4 26B A4B is not working well for me (www.reddit.com via reddit) gemmallama 23w
I am using llama.cpp version b9549 with this arguments as recommended: llama-server --temp 1.0 --top-p 0.95 --top-k 64 -hf ... Here is what I got on chessboard svg test https://www.reddit.com/r/LocalLLaMA/comments/1t53dhp/quality_compariso…
-
Context, memory, and RAM/VRAM (www.reddit.com via reddit) llamaqwen 23w
This will be a slightly disorganized post, I apologize. I’m trying to understand the relationship between context, a memory system for the agent, RAM and VRAM.
-
A handy llama-server launcher with easy model and configuration customisation (www.reddit.com via reddit) llama 23w
I wanted something that I could easily configure to manage a set of sensible defaults, that supports multiple llama-server binaries, with per-model over-rides, and command line over-rides. The utility is here: https://github.com/stew675/st…
-
Qwen 3.6 27B KV cache quant benchmarks: 75 pairs, q8/q6/q5/q4, KVarN, Turbo/TCQ (www.reddit.com via reddit) llamaqwen 23w
Full benchmark results and in-depth analysis are available in the articles: KV Cache Quantization Benchmarks for Long Context and KVarN KV Cache: Implementation and Benchmarks. BeeLlama.cpp (my llama.cpp fork) was used as inference engine…
-
Show HN: Best setup local LLM found for a 5090 (llama.cpp fork + turboquant) (local-llm.utop.workers.dev via hn) llamaqwen 23w
Hi folks, I found this setup on consummer hardware that seems to have great results on local hardware. - qwen 3.6 q6 - 450 K context using turboquant turbo3 mode llama.cpp fork - multimodal support This AI generated blog article is a kind…
-
I’m trying to run: unsloth/gemma-4-31B-it-qat-GGUF gemma-4-31B-it-qat-UD-Q4_K_XL.gguf on an RTX 5090 32GB using llama.cpp Gemma 4 MTP PR branch. Main model loads.
-
5 Months Later: open-deepthink Now Has Full Knowledge Distillation Mode (www.reddit.com via reddit) llama 23w
Hey r/LocalLLaMA, Some of you might remember when I posted about this project back around September last year (it was called local-deepthink then). The core idea was to move past the usual flat multi-agent setups and instead build somethin…
-
Show HN: Ext-Infer – Native LLM Inference and Embeddings for PHP (infer.displace.tech via hn) llamarag 23w
Introduction ext-infer is a PHP 8.3+ extension that loads a GGUF model and runs LLM inference inside the PHP process via llama.cpp. PHP-native semantic search, RAG pipelines, and CLI / worker inference run without shelling out to Python or…
-
dvlt.cu: inference engine written from scratch in CUDA/C++ for NVIDIA's DVLT 3D transformer model (www.reddit.comhttps) vllmllama 23w
Im into both HPC and 3D reconstruction, so I built this as a side project. dvlt.cu is a single 5MB binary: - No python, torch, TF, ONNX, llama.cpp, vLLM, or huggingface runtime - Nearly no dependencies: only cuBLASLt (shipped with libcuda…
-
QAT MTP Heads Upload + PARALLEL=2 Fix + 12B 2-slot Bench (www.reddit.com via reddit) gemmallama 23w
Title: Gemma 4 QAT MTP assistant heads now public on HuggingFace + PARALLEL=2 crash fix + 12B 2-slot bench (Strix Halo / Vulkan) Three things in one update: the converted QAT-matched draft heads are now uploaded for anyone to use, we found…
-
What are you running on 16Gb VRAM + 64Gb Ram? (www.reddit.com via reddit) llamaagentic 23w
I know this gets asked a lot, but I can only find threads that are at least a couple of months old, so I thought I'd ask to see what people are running these days. I have an RTX5080 and 64Gb Ddr5 RAM.
-
AMD MI50 on Debian Testing is doing great and getting better. (www.reddit.com via reddit) llama 23w
There is probably some relevant information to other cards here but my benchmarks are on dual MI50 32GB cards because that is what I have, and thought I would share with the community. Install instructions at the end.
-
120 tok/s on 12GB VRAM with Gemma 4 12B QAT MTP (www.reddit.com via reddit) gemmallama 23w
Google just released the QAT (Quantization-Aware Training) variant of their Gemma 4 models, including 12B, so it was only natural for me to benchmark it on my 12GB GPU since it fits entirely in VRAM. I was pleasantly surprised of the resul…
-
Rate my config!! (www.reddit.com via reddit) llama 23w
Hey all, Wanted to get some eyes on my llama.cpp config to see if there is anything i could improve on. Currently getting an average of 55t/s (up to 75t/s occasionally).
-
KV cache quant benchmarks: KVarN 6-bit matches q8_0, 4-bit matches q5_0. Massive! (www.reddit.com via reddit) llama 23w
TL;DR Based on long context KLD benchmarks, KVarN appears to be just better than usual llama.cpp KV cache quants. At every size, KVarN matches precision of usual quants of one bit higher.
-
Friends Don’t Let Friends Use Ollama — So I Built Anvil (www.reddit.com via reddit) ollamallama 23w
Hi, I’m basically one of you, except I’m stepping onto the other side of the table today, fully prepared to accept your ridicule. Obvious disclosure: this is my project, so yes, this is self-promo — but I’m posting it here because this is…
-
StepFun 3.7 Flash MTP Bench Strix Halo (www.reddit.com via reddit) llama 23w
This is the StepFun Step-3.7-Flash UD-IQ4_XS main model with the official StepFun MTP Q8_0 draft model, served through a patched llama.cpp Vulkan/RADV build. Host System: AMD Ryzen AI Max+ 395 / Radeon 8060S (gfx1151) Memory: 128 GB unifie…
- Gemma 4 QAT Q4_0 Bench on Strix Halo (www.reddit.com)
-
Dual GPUs - 3060 & 3090 on a P520 (www.reddit.com via reddit) llama 23w
I've got a line on a reasonably priced 3090FE and I'm wondering whether it would play nicely with the 3060 I'm already using. System is a ThinkStation P520 - PSU would be an issue until I can get a replacement, so would have to run both GP…
-
Serving TTS/cloning models on llama.cpp? (www.reddit.com via reddit) vllmllama 23w
Are there any quality voice cloning and speech generation models that already have support in Llama.cpp or, more likely, vLLM-Omni? It would be nice to swap them out like any other inference model and use a common API, rather making a sepa…
-
I have a 5090 power limited to 475W. When I run the following command, it barely hits 300W and I get something like 30 t/s: bash ./llama-server \ -m ~/myp/models/unsloth_mtp_Qwen3.6-27B-UD-Q5_K_XL.gguf \ --host 0.0.0.0 \ --port 8080 \ --ch…
-
DeepSeek V4 Flash is amazing! (WIP llama.cpp PR #24162) (www.reddit.com via reddit) llamadeepseek 23w
In case you're not aware already, the DeepSeek V4 series is finally getting supported on llama.cpp with this PR! The PR is at a very early stage right now, so only try it if you're consciously willing to experiment out of curiosity and acc…
-
TL;DR: I spent a long session tuning a 35B MoE on a tiny 8GB laptop GPU. Three things mattered a lot (--no-mmap, VRAM headroom, closing CPU-hungry apps).
-
Initial testing with llama-bench and 3 different Qwen3 models for my R9700 32GB (www.reddit.com via reddit) llama 23w
In a recent build I did I used dual R9700 32GB cards but I wanted to see how a single R9700 stacked up against other hardware I had access to. I created a simple benchmark with llama-bench and ran it on a few different setups.
-
Gemma 4 12B Q4_K_XL Private Benchmark Results (www.reddit.comhttps) gemmallama 23w
Posting to share my results with others, I think the big bottom line is MTP acceptance rates offering a huge speedup, during coding tasks it's over 90% acceptance! Haven't hit my soft goal results or llm as judge benchmarks yet to compare…
-
This is a PSA for people like me who tried it and hit the wall with tool calls failing left and right, so much so that harnesses like OpenCode just didn't work: There is a fix for that. You need to pass a better chat template file, which i…
-
Apples to Apples: MLX vs. Llama.cpp for Gemma 4 12B on an M1 16GB (ziraph.com via hn) gemmallama 23w
Apples® to Apples®: MLX vs llama.cpp for Gemma 4 12B on an M1 16GB A matched-quant MLX-vs-raw-llama.cpp benchmark for Gemma 4 12B on one M1 16GB - decode is a tie, both pinned at the bandwidth wall. The cost that differs is startup and CPU…
-
I built a iOS app to benchmark GGUF models on your iPhone/iPad (www.reddit.com via reddit) llama 23w
Hey I've been working on GenBench, a free iOS app that lets you download, run, and benchmark GGUF models directly on your iPhone or iPad using llama.cpp + Metal. What it does: - Search and download GGUF models from Hugging Face in one tap…
-
Maybe KV cache offload to RAM isn't bad (www.reddit.com via reddit) llama 23w
So, llama.cpp has the -nkvo (--no-kv-offload) option to offload KV cache to RAM instead of VRAM. Many people avoid this because obviously it hurts performance.
-
How to build llama-cpp for Ampere/Blackwell? (www.reddit.com via reddit) llama 23w
Hello, I'm on Windows and started building my own versions of llama-cpp instead of using the precompiled versions. I'm using CUDA 12.9 with my RTX 5070, and I wanted to try to use my RTX 3060ti that I've laying around since I replaced it w…
-
Show HN: Run Llama.cpp In-Process from Java with Project Panama FFM (deemwar-products.github.io via hn) tool-callingllamaopenai 23w
mochallamaA local, tool-calling LLM inside your JVM The only in-process, tool-calling local LLM for the JVM — Spring-first, OpenAI-compatible, llama.cpp-backed via Project Panama FFM. No JNI, no daemon, no native-install dance.
-
Show HN: LLMhop – A tiny, stateless router for LLMs with a NixOS module (github.com via hn) vllmllama 23w
LLMhop is a tiny stateless proxy for LLM inference servers. It tackles an issue I faced when trying to serve more than one local LLM at once which is not natively supported by vLLM.
-
Show HN: Will It Fit? – Opinionated Normal People Llama.cpp VRAM Estimator (hypfer.github.io via hn) llama 23w
llama.cpp VRAM estimator for normal people. Assumes single GPU, all layers offloaded.
-
The Winamp Skin Museum whips the Llama's ass (2020) (www.rockpapershotgun.com via hn) llama 23w
The Winamp Skin Museum really whips the llama's ass Over 65,000 skins to browse! In the late nineties and early noughties, no video game forum was complete without a 'post your desktop' thread, and no desktop screenshot was complete withou…
-
Show HN: TurboPrefill – Multi-GPU prefill acceleration for llama.cpp (github.com via hn) llama 23w
TurboPrefill is an attempt to make layer-split multi-GPU configurations spend less time waiting and more time computing during prefill.
-
Gemma 4 12B appears in Hugging Face (huggingface.co via hn) gemmallama 23w
gemma-4-12B-it-GGUF Recommended way to run this model: llama-server -hf ggml-org/gemma-4-12B-it-GGUF Then, access http://localhost:8080
-
Free Yourself from the Copilot Tax (www.kronkai.com via hn) copilotllama 23w
Hardware accelerated local LLM inference for Go with llama.cpp integration.
-
LlamaStash: a zero-overhead, terminal-native llama.cpp launcher (deepu.tech via hn) llamaopenai 23w
Introducing LlamaStash: a zero-overhead, terminal-native llama.cpp launcher A fast TUI, CLI, daemon, and OpenAI-compatible proxy for running local LLMs via llama.cpp, in one Rust binary Series · GNU/Linux Environment for Developers Post 8…
-
ik_llama.cpp – llama.cpp fork with better CPU performance (github.com via hn) llamadeepseek 24w
ik_llama.cpp: llama.cpp fork with better CPU performance TL;DR This repository is a fork of llama.cpp with better CPU and hybrid GPU/CPU performance, new SOTA quantization types, first-class Bitnet support, better DeepSeek performance via…
-
Show HN: Thaw – Git branch for a running LLM (fork agents, skip prefill) (github.com via hn) llama 24w
I built thaw because forking an LLM agent is absurdly wasteful today. When an agent explores N branches — RL rollouts, best-of-N, parallel coding attempts — each branch re-runs prefill over the same shared context.
-
Llama.cpp now has an official website: llama.app (twitter.com via hn) llama 24w
llama.cpp now has an official website: llama.app Our goal is to make local AI accessible to everyone, and improving the user experience is a big part of that. On the new landing page you’ll find a single-line cross-platform installer.
-
LiquidAI/LFM2.5-8B-A1B · Hugging Face (huggingface.co via reddit) vllmmoellama+1 24w
looks like you can run it on any potato (A1B)! https://huggingface.co/LiquidAI/LFM2.5-8B-A1B-GGUF from LiquidAI: LFM2.5 is a new family of hybrid models designed for on-device deployment.
-
EDIT - IGNORE. I MADE A MISTAKE.
-
I implemented Laguna (XS.2) as a model in Llama.cpp (github.com via reddit) llama 24w
llama.cpp Manifesto / ggml / ops LLM inference in C/C++ Recent API changes Changelog for libllama API Changelog for llama-server REST API Hot topics Hugging Face cache migration: models downloaded with -hf are now stored in the standard Hu…
-
I have been coming to this subreddit to understand what the optimal config is to run a model on a given hardware setup. I referred to specific benchmarks, but they are too generic and do not consider the underlying hardware.
-
I'm seeing low draft acceptance when using Qwen3.x MTP, what am I doing wrong? (www.reddit.com) llama 24w
I'm using llama.cpp, and I've tried Bartowski's and my own quants. When using Qwen3.5-122B or Qwen3.6-27B, I'm seeing really low draft acceptance in chats with interleaved code snippets (chatting with the LLM about programming / a code pro…
-
I'm posting this because it may be helpful to squeeze the 12GB VRAM in the 3060. All credit goes to spiritbuun's fork (github.com/spiritbuun/buun-llama-cpp) and mudler's APEX quantizations (huggingface.co/mudler).
-
Local run for multi users: which software set? (www.reddit.com) vllmllama 24w
Context: I am testing and running local LLM on Linux for some months, first with llama.cpp and now with vLLM for better concurrent capabilities. I use llama-swap in front of either vLLM or llama.cpp in order to have thinking and non-thinki…
-
Used the vllm version of https://github.com/noonghunna/club-3090 It worked fine for myabe 20 40k context, havent tried the new one. Anyone used the new llama.cpp patched one for single 3090?
-
Data Gathering (www.reddit.com) llama 24w
Hello everyone I'm looking to gather some information about local model users for a college project. If you have the time please just comment your: hardware (CPU,GPUs, total VRAM and RAM) and OS the model/s you primarily use and at what qu…
-
my apologies if anything does not make sense, I literally dont know what I am doing, im not a programmer, just a simple vibe coder, with an Claude subscription. That said, if you have 200gb of sys ram+vram and want to run deepseek v4 flash…
-
Authors Sue Meta's AI Scientists Directly in Llama Copyright Case (www.law.com via hn) llama 24w
A proposed class action filed against Meta Platforms in New York federal court targets not only the company and its CEO Mark Zuckerberg but also two former senior AI researchers by name—an unusual move that could signal a new front in the…
-
Qwen3.6 huge quality gain from Q4 to Q6 for coding agent (www.reddit.com) ollamallamadeepseek 24w
So, last week I tried to update my unused local LLM setup. I had to stop using it because quality was too low and deepseek was too cheap.
-
Looking for a working Deepseek-v4-Flash quant (www.reddit.com) vllmllamadeepseek 24w
Best I tried so far is https://huggingface.co/nsparks/DeepSeek-V4-Flash-FP4-FP8-GGUF with the custom llama.cpp fork, but it suffers from low quality and random incoherent output. VLLM wouldn't support anything other than H100s for DS4.
-
Show HN: Biopetals – Run biology tuned Llama, BitTorrent-style (github.com via hn) llama 24w
About a month ago, I heard about petals. Petals is basically a library that lets you run LLMs by loading the weights onto a network of computers that are all running petals.
-
Here's my article with 38 quant pairs thoroughly benchmarked in KLD with 3 different Qwen 3.6 27B configs: Q5_K_S + 64k context, IQ4_XS + 64k context, IQ4_XS + 128k context. This allows us to track not only how cache quantizations affects…
-
I was given the great opportunity to borrow a H100 with 94GB VRAM at work until it is needed by a customer. (No idea how much system ram I will get, but I guess they are a bit flexible on this).
-
Went to the monthly AI dev meetup (www.reddit.com) glmllamacodex+1 24w
Usual crowd. Everyone's on Claude or Codex, nobody's really sure how any of it actually works, and that's fine, that's the vibe.
-
RTX5080 vs RTX 3090 ? (www.reddit.com) llamaqwen 24w
Hey guys, i’m looking for some educated advice / opinions on runing local LLM. I own an RTX 5080 and I’m runing llama.cpp (custom builds with turbo quant) with Qwen 27b Q3_K_M with a context of 128k all in vRAM (using turbo3/4 on kvcache t…
-
Info: Nvidia Cuda 13.3 landed (www.reddit.com) llama 24w
Cuda 13.3 Downloads Release Notes Anybody already tried llama.cpp with 13.3?
-
Run Llama.cpp on a Mac Pro 6,1 with Dual FirePro D700 GPUs on Ubuntu (matthewgribben.com via hn) llama 24w
Running llama.cpp on a Mac Pro 6,1 with Dual FirePro D700s on Ubuntu A D700-specific guide to running llama.cpp with Vulkan on the 2013 Mac Pro: dual 6 GB FirePro cards, Ubuntu, RADV, full GPU offload, cooling, and the traps that make old…
-
I ditched LM Studio for llama.cpp and my local LLM doesn't feel like a downgrade (www.xda-developers.com via hn) llama 24w
LM Studio has been my default runner for as long as I've been running local LLMs, which is more than long enough now to call it part of my daily flow rather than just something I'm experimenting with anymore. The appeal of LM Studio is pre…
-
Note: Latest version of llama.cpp (b4c0549a49be9e6dc59ac9d0a5bc21dbda910774) My run command: ``` llama-server \ --temp 0.6 \ --top-p 0.95 \ --top-k 20 \ --presence_penalty 0.0 \ --min-p 0.00 \ --gpu-layers all \ -m /home/eleung/huggingface…
-
I made a Windows app for managing llama.cpp in WSL/Ubuntu (www.reddit.com) llama 24w
I’m a Windows user, and I have fairly Windows-y expectations for software: I prefer not having to live in a terminal just to install, build, configure, and run things. I couldn’t find an app that managed the full llama.cpp-on-WSL workflow…
-
Looking for Suggestions — Single 5090 & 64gb DDR5 (www.reddit.com) vllmllamaqwen+1 24w
Hi Reddit, I am planning on running Qwen 3.6 27b NVFP4 via vLLM on my 5090 but was wondering if something like 35b a3b at Q8 on Llama would produce better results for agentic coding and utilize the system memory. My research says no but if…
-
Long-context performance at lower quants (www.reddit.com) llama 24w
I've been using Qwen3.5 122B A10B (Q3_K_XL) a lot lately for coding, and it's been pretty incredible overall like it feels not far off from frontier-level for most tasks -- but I've been noticing that usually once I hit around 75-80k conte…
-
Llama.cpp: What's up with -sm tensor + AMD + Vulkan? (www.reddit.com) mistralgemmallama+1 24w
Has anyone got it to work? I tried it with dense models (eg qwen 27b, gemma 31b, mistral 128b) since that's where I need it most, but it always core dumps.
-
Harbor v0.4.19 - vllm/sglang/llama.cpp launch codex/claude/pi/opencode (www.reddit.com) vllmllamacodex+1 24w
I'm usually not posting about Harbor releases out of the respect for the community here, but I think v0.4.19 might save a lot of people some time. Harbor can now launch your local agentic coding tools with local inference backends.
-
Poor performance on RX 9070 XT (www.reddit.com) llama 24w
I was thinking about upgrading from an MI50 to an AMD AI PRO9700, and I happen to have an RX 9070 XT on my gaming pc, so I tested the performance on it to have an idea of what to expect. So, install rocm, build llama.cpp, download Qwen3.6-…
-
Been experimenting with an idea — what if your AI assistant actually remembered everything you did on your computer? Not stateless chats, but real persistent context.
-
Llamacpp server : How do the -np and -c flags interact? (www.reddit.com) moellamaqwen 24w
I've been using lm studio for a few months. I want to try hermes agents with Qwen 3.6 MoE, so I'm switching to llama.cpp and I don't understand well how the server slots -np and the context size -c interact.
-
Strix Halo users, a rejected PR can give you up to 30% faster PP for MOEs. (www.reddit.com) llama 24w
Here's the PR by pedapudi. https://github.com/ggml-org/llama.cpp/pull/21344 It's merge request has been denied so it will not be in mainline llama.cpp.
-
Shard - getting to 10× KV cache compression (krishgarg.com via reddit) llama 24w
TL;DR. Shard is a drop-in HuggingFace Cache that makes Llama-3.1-8B's KV memory about 10× smaller at 8K context (11× at 32K) without measurable hits to NIAH or LongBench.
-
Building Conifer, an open-source local inference runtime (free + open source) (www.reddit.com) llama 24w
Team of 5 from Princeton, and we got funding to build a local inference engine for Apple Silicon - rust, hand written kernels - and we're at the point where working with ~100 people will expose bugs/what people want tool-wise. All of this…
-
Hey Reddit! A couple of weeks ago, I posted about my independent research on treating LLM alignment as a latent space shift.
-
AI content detector based on Qwen 0.8b fine-tuned on Pangram dataset (www.reddit.com) gemmallamaqwen+1 24w
I've fine-tuned Qwen 3.5 0.8B on the dataset provided by Pangram with their EditLens paper. It's available via a Chrome extension; you can just click selected text and it's going to give you the probability distribution of how likely it is…
-
Can you jailbreak Llama 3.1 8B? (Red-Teaming Challenge) (www.reddit.com) jailbreakllamasecurity 24w
Hi everyone, I'm working on a runtime governance engine designed to force any autonomous agent to stay strictly aligned with the exact guardrails and values you program it with. To stress-test the governance layer, we deliberately chose a…
-
Llama.cpp : Split Mode Tensor Fix Incoming? (www.reddit.com) llama 24w
Appears thay have been cooking and we might see a fix soon released for crashes on split mode tensor Multi-gpu folks keep watch - ( In my tests SM Tensor has a ~35% uplift in TG over Layer but ofc crashes every 90-120 minutes due to vram e…
-
Best coding model on RTX 3060 (www.reddit.com) vllmllama 24w
Wondering what’s the best coding model that can fit on a RTX 3060 (12GB). Has anyone been able to do something useful with it?
-
The Financial Times has published an article about Heretic (www.reddit.com) llama 25w
https://www.ft.com/content/5630ed79-a263-41ed-9a1a-321617ae310e “The FT was able to use Heretic, a tool available on the popular code repository GitHub, to remove the guardrails from Meta’s Llama 3.3 model in less than 10 minutes without a…
-
Old Mac Pro still proving its worth (www.reddit.com) llamaqwen 25w
The “Trash Can” Mac Pro, once the most expensive machine you could buy from Apple, mine was just shy of £10,000 in 2016 — that’s £14k in today’s money. Until recently mine was just running as a kubernetes single node development platform,…
-
I just published a repo called MCP from Scratch that teaches the Model Context Protocol by building it step by step in plain Node.js. Most of the repo is about understanding MCP itself, but the later modules may be relevant here: I added a…
-
Could someone please help explain these results? (www.reddit.com) moellama 25w
I'm running Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf on 12 GB VRAM and 32 GB RAM via the TurboQuant variant of llama.cpp. I increased the --n-cpu-moe value from 8 to 30, and my inference rate doubled!
-
llama.cpp has a clever trick for speeding up KV cache decode (www.reddit.com) llama 25w
So, I use llama-server as my endpoint to run local models and connect them to Open-WebUI, Hermes, and OpenCode. But since llama.cpp's webUI has been receiving a lot of updates, I took a look at its settings and noticed a particular one und…
-
i want the best installation that fit my use and my low-compute H.W , i want to run small to above small llm like "qwen" 2b ,4b and 27b , and "gemma" 31B. rely completely on only old CPU 4th.gen i7 with that few 32gb 'slow' ddr3.
-
hipEngine: Fast Native Qwen 3.6 Inference for RDNA3 (Strix Halo, 7900 XTX) (www.reddit.com) moellamaqwen 25w
A few weeks ago, after finishing FastDMS, I started toying around writing some RDNA3 kernels again to see how fast I could get Qwen 3.6 MoE running. It turned out well enough, so over the past couple weeks, I turned those experiments into…
-
I've been on Claude Code daily for a while and want to see how far local models can do my setup: - MacBook Pro M4 (16"), 48GB - macOS 26 tahoe Usually i do: seo researches, macos swift apps, websites) What I'm trying to figure out: Which t…
-
magic incantation to get llama-bench to work with MTP ? (www.reddit.com) llama 25w
It does not like anything I have tried, including what works with llama-server. is it not built to work with speculative decoding?
-
What frontend do you guys use? (www.reddit.com) llama 25w
I’m using vim lmao with a custom made plugin for completing text, so I was curious what yall use. Llama-server seems like a sensible default but it seems limited
-
Need Help Choosing a Harness for Qwen 3.6 27B (www.reddit.com) llamaqwen 25w
I've burned a week trying to customize my agent manually - building my own front end - but I've gotten to the point where I'm just exhausted and willing to try a harness, but need the right one. I read posts all the time, but I have a spec…
-
I am experimenting with loading large models into small RAM and interested in theoretical limits, which people who know how engines (e.g. llama.cpp) work might have some ideas about.
-
GPU VRAM only for small models with llama.cpp: is it possible? (www.reddit.com) llamaqwen 25w
I'm still in my learning process and so far I've been able to make satisfying use of my setup (4070 with 12GB VRAM + 32GB RAM and iGPU for my GUI). I've been able to run both Gemma4 26B and Qwen 3.6 35B MoEs up to high quants with large co…
-
Qwen3.6-35B-A3B vs Gemma4-26B-A4B (www.reddit.com) llamaqwen 25w
Just wondering how are people's experience with both these models! I've had some nice results with Qwen but Gemma4 runs so much faster here.
-
gemma 4 e2b quality degrades after ~30-40 continuous inferences on 4gb vram? (www.reddit.com) gemmallama 25w
running gemma e2b via llama-server for continuous background tasks on a 1650 4gb. works great initially but after maybe 30-40 calls the outputs start getting noticeably worse — shorter responses, missing fields in json output, sometimes ju…
-
As some other fellow lllmers I've discovered few days ago that the amazing llama.cpp project has just added native tools functionalities into the server. After having enabled the relative options into llama-server and played a bit with the…
-
What workstation to get for ~13k EUR? (www.reddit.com) vllmminimaxllama+2 25w
My use-cases will be to test open-weight LLMs and work on harnesses, inference systems and possibly other non-ML workflows (CS-related) in the future. Fine-tuning would not be something I do locally because I can rent a B200 from RunPod fo…
-
minor speed bump for MTP with Qwen3.6-27B-MTP Q6_K_XL (www.reddit.com) llama 25w
I'm on Macbook M5 Max with 128GB RAM Running a test in openwebui using llama-server (llama.cpp): unsloth/Qwen3.6-27B-UD-Q6_K_XL.gguf (non MTP): 19tps unsloth/Qwen3.6-27B-UD-Q6_K_XL.gguf (MTP): 22.3tps So nothing like the massive improvemen…
-
Hi everyone, I’ve just published the first public release of llampart 1.0.0: https://github.com/mchowy-troll/llampart llampart is a standalone local web UI designed to work with `llama-server`. It started from the `llama-ui` work in the `l…
-
please forgive the mildly clickbait title. hard to fit everything in it I've seen a lot of discussion here about KV-cache quantization, especially with the recent llama.cpp improvements, leading to some debate on the tradeoffs between KV q…
-
llama.cpp server have built-in native tools (exec_shell, edit_file, etc.) (www.reddit.com) llama 25w
https://preview.redd.it/24uvk7o4sy2h1.png?width=1440&format=png&auto=webp&s=542570e3057b6f44c1e7e8d92130f575fb69cfa2 https://preview.redd.it/l4bbm7o4sy2h1.png?width=1440&format=png&auto=webp&s=3dc0edd978da23fecf81e86a269a06de643247d1 I was…
-
NVFP4 + MTP - voilà on llama.cpp (www.reddit.com) llama 25w
As in title - NVFP4 + MTP at once on llama.cpp https://github.com/ggml-org/llama.cpp/releases/tag/b9297
-
Run Chrome’s tiny Gemma4 (aka Gemini Nano) directly on PC without GPU (www.reddit.com) vllmgemmallama+1 25w
Everyone remembers that sneaky download of Gemini Nano earlier this month? and if you talk to it, it will happily tell you it’s a Gemma.
-
KinetiX: An intra-inference hardware interlock for LLMs (github.com via hn) llama 25w
KinetiX Latent Interlock KinetiX is a hardware and software safety interlock designed to monitor latent states (activation tensors) in real time within LLM inference engines like llama.cpp. It enables instant process termination upon detec…
-
I'm running llama.cpp using this docker container: https://github.com/mixa3607/ML-gfx906 (it's just a lot easier than building from source, which I was doing previously). The MI60 (or MI50) are just a real pain in the behind to get working…
-
Llama.cpp VS LiteRT on a custom Xiaomi 12 Pro 24/7 Server (V2 Redesign) (www.reddit.com) ollamallama 25w
https://preview.redd.it/sm4ysgdw1w2h1.png?width=1376&format=png&auto=webp&s=3705932403919814fbf2008a1cba189d17e0591e Thanks everyone for the advice on my previous post (24/7 Headless AI Server on Xiaomi 12 Pro (Snapdragon 8 Gen 1 + Ollama/…
-
LLMKube – A Kubernetes operator for local LLMs across Nvidia and Mac fleets (llmkube.com via hn) vllmoperatorllama 25w
Run production LLMs on your own hardware A Kubernetes operator for self-hosted LLM inference. vLLM, llama.cpp, TGI, NVIDIA, Apple Silicon.
-
Sharing this because I didn't believe the first run. Setup: laptop-class RTX 5090 (24GB, sm_120 Blackwell, ~896 GB/s), Linux.
-
Gemma4 26b a4b Apex quant is quite good (www.reddit.com) gemmallama 25w
I tried mudler's apex quant for gemma4 26b a4b and it was amazing! I got 38tps at 90.000 context with no loop and suprisingly no quality degradation.
-
Experimental "Preserve Thinking" Jinja Template for Gemma4 31B in llama.cpp (www.reddit.com) gemmallama 25w
https://huggingface.co/stevelikesrhino/gemma-4-31B-it-nvfp4-GGUF/blob/main/gemma4-improved.jinja Yall are more than welcome to try it out and provide feedback. In my own testing in Pi-coding-agent I no longer have the "forgot to close thin…
-
club-rdna16: practical 16GB AMD/Radeon local LLM testing repo (www.reddit.com) llama 25w
Following on from club-5060ti, I’ve been doing some testing with my desktop AMD GPU and wanted to make a similar repo for 16GB Radeon cards. Repo: https://github.com/5p00kyy/club-rdna16 Pages/results: https://5p00kyy.github.io/club-rdna16/…
-
Qwen3.6 27B Pure Quant: 40 tok/s on 16 GB VRAM (www.reddit.com) llama 25w
Hello everyone! I want to share the result of my experiment to make Qwen3.6 27B Q4_K_M fits in to my RTX 5060 Ti 16 GB.
-
Blackwell and PDL performance increase (www.reddit.com) llamaqwen 25w
Llama.cpp recently introduced support for Programmatic Dependent Launch (PDL), which is a new feature in Nvidia GPUs (CC >= 90, not including ADA) such as Blackwell. (See PR 22522.) In short, PDL enables more efficient execution of kernels…
-
Llama.cpp not using CUDA - OOM error (www.reddit.com) llama 25w
hey guys, I want to say that I appreciate all the helpful support from this community as I’ve stepped into the local LLM world. I‘m thankful to have a community around that doesn’t gate keep and is open to new comers.
-
Setup: Kubuntu 24.04 - AMD cards - R9700 AI PRO and 7800xt (32gb + 16gb) - llama-cpp server - stack setup in docker - vulkan image I tried with ROCM but it wouldn't play nice with RDNA4 + RDNA3 mix. Vulkan seems to work.
-
I'm building a local-first agent — a plain ReAct loop (think, pick a tool, observe, repeat) on a llama.cpp backend — and I want to be precise about a question that usually just gets answered with "it depends." It does depend. So let me spl…
-
WebGPU back end in llama.cpp/ggml (twitter.com via hn) llama 25w
could not extract summary
-
I ran a quantization shootout on Qwen3-Coder and the results are... interesting (www.reddit.com) llama 25w
Out of random curiousity I ran a shootout on Qwen3-Coder-Next. I've been using the MXFP4_MOE from unsloth for awhile as it's just really fast on my system.
-
Qwen-27B-IQ4_KS for ik_llama.cpp, especially for NVIDIA with 16GB VRAM (www.reddit.com) llamaqwen 25w
Hi everyone, I'm presenting a new quantization of the Qwen-27B model, created specifically with 16GB VRAM NVIDIA GPUs in mind. I used quants that, unfortunately, are not yet available in the main upstream llama.cpp.
-
Seeking resources to read about llama.cpp server and how offloading works (www.reddit.com) llama 25w
SETUP INFO: Amd R9700 AI PRO. Using llama-cpp server, ROCM docker version.
-
Think Local started with a simple idea: AI should work for you, not collect from you. So I built an app that lets you run modern AI models completely on-device - privately and fully offline.
-
[llama.cpp] Asymmetric KV q8/q4 cache: current caveats and discussion in GGML repo (www.reddit.com) llama 25w
Probably most of you are aware that using anything other than -ctk q8_0 -ctv q8_0 / -ctk q4_0 -ctv q4_0 as startup options for llama.cpp leads to prompt processing on cpu instead of gpu for cuda at least. E.g.
-
[NEW] Supra-50M Released! (www.reddit.com) llama 25w
https://preview.redd.it/kx39ammxno2h1.jpg?width=1080&format=pjpg&auto=webp&s=d1a2d5b27920a5b61a50547a6e70a6378445cae4 SupraLabs released a new model! - Supra-50M Supra-50M is a compact 50M-parameter causal language model (BASE and INSTRUCT…
-
Agent builders: are GPT/Claude/Gemini API costs killing your margins? (www.reddit.com) llamadeepseekqwen+5 25w
Hey everyone, For people building agents with LangGraph, CrewAI, AutoGen, OpenAI Agents SDK, Claude MCP/SDK, Google ADK, or LlamaIndex — how are you managing LLM API costs? Agent workflows can get expensive fast because of: tool calls retr…
-
I work on AxonFlow, a source-available (BSL 1.1) runtime for long-running agent workflows. We’ve been running it in front of Ollama-served models and OpenAI-compatible local endpoints (llama.cpp `--server`, vLLM, LM Studio).
-
For everyone that uses OpenCode / Pi - Heres your promptprocessing fix! (www.reddit.com) llama 25w
This PR deserves much more attention as it fixes the constant promptprocessing that happens when using llama.cpp with Opencode or pi. https://github.com/ggml-org/llama.cpp/pull/22929
-
Finally got Llama 3 70B running locally and wired up function calling so it can search the web. First tried self-hosting SearXNG, but the results are pretty messy.
-
I've been building this for the past few months as a side project — started because I didn't want to run llama.cpp from the command line every time I wanted to try a model. I just wanted something that worked with a click.
-
Currently, I'm running a Hermes agent with an OpenAI v1 compatible endpoint provided by Kobold. My setup is a a 24GB 3090Ti + 512GB DDR4 running Qwen3.6-35B-A3B.
-
110 tok/s with 12GB VRAM on Qwen3.6 35B A3B and ik_llama.cpp (www.reddit.com) llama 25w
Had been getting great MTP performance with llama.cpp on my RTX 4070 Super 12GB, until they actually merged the MTP PR. Then, performance tanked and was barely above non-MTP.
-
Continue config for Qwen 3.6 and llamacpp (www.reddit.com) continue-devllamaqwen 25w
If anyone is using the Continue.dev extension in VSCode, what config settings are you using for Continue and the llama-server? Mine keeps hanging after bad tool calls.
-
This has turned out to be useful to many of my friends so I thought I'd share here as well. I created a tool and documentation page for most major open-souce project's adherence to 'OpenAI compatibility' after seeing inconsistencies betwee…
-
Hardware LLM Taalas Reaches >14,000 TPS on Llama 3.1 8B (taalas.com via hn) llama 25w
Products Taalas HC1 Technology Demonstrator - Runs Llama 3.1 8B model - TSMC 6nm | 815mm2 | 53B Transistor - 2.5 kW Server Instantaneous Inference HC1 demonstrates the power of Taalas hardcore model silicon technology, delivering 17k token…
-
Found this ACM paper on prompt injection and jailbreak attacks against open-source LLMs. The authors tested 10 open-source models across 94 prompt injection and 73 jailbreak scenarios, including Phi, Mistral, DeepSeek-R1, Llama 3.2, Qwen,…
-
qwen 2B model - thinks for 600 tokens on a simple "Hi" (www.reddit.com) llamaqwen 25w
Using llama.cpp Model - Q8 - unsloth/Qwen3.5-2B-GGUF Is this expected with tiny models like this one? I am trying tiny models for a since most of the task I have involves searching local files etc and need less of the models own knowledge.
-
Qwen3.6 27B and llama.cpp appreciation post (www.reddit.com) llama 25w
To preface, here's my config: llama-server \ --host 0.0.0.0 \ --port 1235 \ --models-preset %h/Software/models.ini \ --models-max 1 \ --sleep-idle-seconds 3600 \ --timeout 3600 \ --parallel 1 \ --device ROCm0,ROCm1 [*] flash-attn = on jinj…
-
AMD BC-250 and the search for Cheap Compute (www.reddit.com) llama 25w
I've been searching for disused/underappreciated compute vectors for a few months since the MI50 shot up in proce - in comes the salvaged PS5 APU on a standalone board; Zen 2, 16 GB unified GDDR6, RDNA 2 (gfx1013). They're $50-150 on eBay…
-
Volatile prefill speed after each reboot - llama.cpp (www.reddit.com) moellama 25w
After every machine restart I get a different prefill speed, it can be only 300t/s or 1500t/s. It's like a lottery at each restart.
-
Show HN: Llama CPU Benchmarks (deemwar-products.github.io via hn) llamaqwen 25w
TurboQuant — "8× faster" The headline is a synthetic GPU-kernel number. On real CPU end-to-end it ran 2.2× slower and dropped Qwen accuracy 17 pp.
-
At wits end for optimizing settings in llama.cpp for 100k context (www.reddit.com) llama 25w
Long story short, I am running Qwen3.5-35B-A3B (GGUF format) and other models on MacOS and getting around 1500 tokens/sec for prompt processing and around 35-50 tokens per second for prompt processing. I'm using the latest version of llama…
-
It keeps running into race conditions/OOM when switching between models, as the previous process doesn't unload from VRAM fast enough. What is the simplest fix for this right now?
-
MTP (Multi-Token Prediction) just merged into mainline llama.cpp at b9190. I promised u/WarthogConfident4039 a Qwen3.6 benchmarking round.
-
Hi, i run llama.cpp inside LXC on a Proxmox server. The hardware is a recent AMD Epyc with two 6000 Blackwell MaxQ.
-
The MTP function in LMStudio causes a decrease in output quality. (www.reddit.com) llama 25w
The prompt is very simple, you can see it at the end. Both tests used the exact same settings, the only difference was that I turned the MTP button on/off, nothing else changed, I tried similar tests multiple times with similar results: By…
-
LM Studio finally added support for MTP Speculative Decoding (www.reddit.com) llama 25w
https://preview.redd.it/1uuzjm0ll72h1.png?width=923&format=png&auto=webp&s=1af7d7594be1e08ff7ad6797e2bc53e9410769a3 update to 0.4.14 Build 2 (Beta) and make sure your llama.cpp engine is 2.15.0 https://preview.redd.it/x0vdwjb3n72h1.png?wid…
-
Do smaller quants silently break tool calls / JSON output? (www.reddit.com) vllmollamallama 25w
I posted recently about EvalShift, an OSS CLI for regression-testing LLM model changes. A few people pointed out that for LocalLLaMA, the more interesting use case may be quantization regression: Q8 -> Q4_K_M Same base model, same prompts,…
-
Title. I've got a llama.cpp server running a model being accessed across a number of scripts, and some of them are easier for the model than others, and those easier ones are also latency dependent.
-
Ternative – C++/CUDA inference engine for ternary LLMs with runtime LoRA (github.com via hn) llamaopenai 25w
# ternative Inference engine for ternary-weight LLMs with runtime LoRA — the llama.cpp of BitNet models. Loads a BitNet I2_S base GGUF + a separate LoRA adapter GGUF, merges them at full F32 precision, and serves the result via an OpenAI-…
-
Claude Code has 240+ models via NVIDIA NIM gateway (www.reddit.com) haikullamasonnet+3 25w
TIL Claude Code has 240+ models via NVIDIA NIM gateway — Nemotron-3 120B for agentic coding is surprisingly good So I was messing around with /model in Claude Code today and noticed something most people probably don't know about — after t…
-
Show HN: Llama-dash – local LLM operators dashboard and proxy (github.com via hn) llama 25w
llama-dash llama-dash turns a self-hosted local inference box into an observable, policy-controlled AI gateway: one UI for model state, request history, API keys, routing rules, proxy metrics, and client setup. The implemented inference ba…
-
Disclosure: I made this. Open-source, MIT, Windows + Linux.
-
Time to update llama.cpp to get som MTP improvements! (www.reddit.com) llama 25w
https://github.com/ggml-org/llama.cpp/pull/23269
-
Find bugs in YOUR code using OpenCode, Llama.cpp and Qwen3.6 (wtarreau.blogspot.com via hn) llama 25w
Background For quite some time I had been submitting tasks to LLMs via llama-cli (natively) or llama-server (API), both from the excellent llama.cpp project. On CPU-only llama-cli starts fast and can restart from a checkpoint which has alr…
-
Measuring Maximum Activations in Open Large Language Models (arxiv.org via reddit) moellama 25w
The dynamic range of activations is a first-order constraint for low-bit quantization, activation scaling, and stable LLM inference. Prior work characterized outlier features and massive activations on pre-2024 LLaMA-style models, and the…
-
Llama-server and MTP (www.reddit.com) gemmallama 25w
currently in order to use MTP one needs to enable it in the starting argument of llama server. --spec-type draft-mtp --spec-draft-n-max 2 But then other models that do not use MTP currently like Gemma or basically all other models fail to…
-
The puzzle: You have 140 nuclear bombs and must bomb every country on Earth. Each bomb is assigned to one country.
-
Qwen3.6 35B MTP, t/s varies on different scenario (www.reddit.com) moellama 25w
Tried Qwen3.6 35B Q5_K_M MTP, HW: 9700x, 64GB 5600 RAM, 5060 TI 16GB. --n-cpu-moe 30 ^ -ngl 99 ^ -c 131072 ^ --no-mmap ^ --flash-attn on ^ --cache-type-v q8_0 ^ --cache-type-k q8_0 ^ --threads 8 ^ --parallel 1 ^ -rea off ^ --reasoning-budg…
-
TurboQuant on 16 GB VRAM (www.reddit.com) llama 25w
I've got Qwen3.6-27B IQ4_XS (14.7 GB, cHunter789's build) on an RX 7800 XT with ROCm 7.1. Display on iGPU, full 16 GB available for compute.
-
Weird performance depending on quant (www.reddit.com) llama 25w
Hi, I'm using llama.cpp with qwen3.6 35B A3B on two different machines. I noticed that on both machines tokens per second is better while using Q4_K_S and Q4_K_M quants than lower Q3_K_M quants.
-
What's good everybody, I probably have the fastest possible setup on these AMD Radeon RDNA2 GPUs for one reason only. A custom binary that bypasses some assert statement causing a crash in today’s stock releases.
-
From 6gb to 32gb (www.reddit.com) llama 25w
Well I ordered a 3090 today. I plan on pairing it with a 3060 I have for 32gb combined VRAM.
-
I posted earlier about RTX 5060 Ti local LLM testing, and I have cleaned the repo up quite a bit since then. The project is now a more structured benchmark/recipe repo rather than scattered notes.
-
Need help getting 7900 XTX PyTorch performance metrics (www.reddit.com) vllmllama 25w
I'm on a quest to profile and benchmark different GPUs for PyTorch, vLLM, and llama.cpp. Cannot find the high-end AMD consumer cards for rent anywhere online and interested in the PyTorch ROCm performance of the 7900 XTX (if you want to co…
-
9070xt speed inconsistent. (www.reddit.com) llama 25w
I have a 9070xt on windows 10, and "The Rock Nightly" ROCM & built llama.cpp using the following flags : cmake .. -G Ninja ^ -DCMAKE_C_COMPILER="C:\opt\rocm\lib\llvm\bin\clang.exe" ^ -DCMAKE_CXX_COMPILER="C:\opt\rocm\lib\llvm\bin\clang++.e…
-
No tg speedup with MTP on RX 6800 XT (www.reddit.com) llama 25w
I ran Qwen3.5 9B on my AMD RX 6800 XT with ROCM and it seems to actually be slowing down token generation. I'm using Unsloth's quants.
-
Is the llama.cpp nixos flake just broken? (www.reddit.com) llama 25w
I can't seem to build any of the latest releases. I'm not sure if something has changed and I haven't kept up, but only way to get a working build is to pin to like a 3 week old commit.
-
`🧬 Flux‑Genotype – A CPU LLM that rewrites itself` I've been working on an open-source kernel called **flux-genotype**. It orchestrates local models (TinyLlama, Llama 3.2, Hermes 3, DeepSeek-Coder) into a self-modifying ecosystem.
-
PR #22673 (commit 4f13cb7) landed MTP speculative decoding in mainline llama.cpp on May 16. I tested it on two separate rigs.
- Benchmarking llama.cpp's new MTP support on Strix Halo (calebcoffie.com)
-
Built this for myself after wanting to use local LLMs during work calls without the window showing up on screen share. Every existing tool was either cloud-only or a 200MB Electron app.
-
Tesla P40 running qwen 3.6 (www.reddit.com) llamaqwen 25w
Does anyone know why qwen 3.6 MTP spec decoding won't work with Tesla P40 when the K cache is quantized? I was able to get mtp qwen 3.6 27B Q5 running at 20t/s on my tesla p40.
-
Llama-server: is it bleeding to CPU/RAM? (www.reddit.com) llama 25w
Is there an easy way to know if a model is using CPU/RAM (and not only GPU/VRAM)? (I think standard verbose output, which got shorter, says nothing about this, but I may be missing something)
-
I thought it had horrible performance and was a nothingburger and had spent like an hour benchmarking it. Updated it yesterday and received a like 1.5-1.8x token boost.
-
Not getting any faster with MTP on Macbook Pro M1 Max 32gb (www.reddit.com) llama 26w
Using latest llama.cpp with mtp and these settings, I only get 10 tps, should I be getting more? [unsloth/Qwen3.6-27B-MTP-Q4_K_M] jinja = true model = /Users/[username]/llms/unsloth/Qwen3.6-27B-MTP-GGUF/Qwen3.6-27B-Q4_K_M.gguf cache-type-k…
-
If you use continue.dev and Qwen 3.6 (dense / MoE) - I could use your help (www.reddit.com) continue-devmoellama+1 26w
Someone suggested I give Continue (Vscode extension) a try. I've been using Roo / Zoo now and liking it but it is pretty tough on context and I was told continue has more control over it.
-
Quantizing MTP KV Cache = free lunch? (www.reddit.com) llama 26w
With the MTP llama.cpp implementation in the Qwen3.6/3.5 models more VRAM is required for the MTP layer. However, many people don't realize this layer comes with its own KV cache which can also be quantized: -cache-type-k-draft q8_0 -cache…
-
GGUF with MTP vs MLX without. Is mlx still the way to go for mac users? (www.reddit.com) llama 26w
Has anyone of the mac users tested the speed difference (token gen, promt processing) between mlx quants without mtp, vs gguf quants with mtp? More or less once a month I wonder if mlx is still the correct path in mac.
-
TL;DR best setup I tested on a RTX 3090 24 GB: ik_llama.cpp + Qwen3.6-27B-MTP-IQ4_KS.gguf 156k context, q8_0/q8_0 KV, MTP, vision on CPU benchmark result on a ~5.9k prompt + 1k output: about 1261 tok/s prefill, 72.9 tok/s decode llama.cpp…
-
MTP vs non-MTP vram usage difference? (www.reddit.com) llama 26w
As per title, assuming you run both with the same context and quantization in llama.cpp is there any difference in vram usage?
-
Tested a bit on my XTX, a bit share hope helpful, thanks to Lucebox! Lucebox DFlash + PFlash PR #119 Reproduction Report (RX 7900 XTX) Hardware Environment Component Spec GPU AMD Radeon RX 7900 XTX (Navi 31, gfx1100) VRAM 24 GiB GDDR6 (~93…
-
Qwen 3.6 27B Q8 on four Nvidia RTX A4000 (16GB each) with Llama.cpp and MTP enabled My setup is heterogenous, I originally acquired my server (Lenovo ThinkStation P3 Tower Gen 2) to run OpenShift/K8s clusters (because I work on that), and…
- Qwen 27b MTP Config, Llama.cpp Single 3090 (www.reddit.com)
-
I’m building AgoraDigest, an experimental site where multiple AI agents answer the same hard technical question independently, then a synthesized digest preserves: verdict best-use-case boundaries conflicts between agents evidence gaps ver…
-
I'll be UPDATING this as it seems I was benchmarking and testing Just before the UPDATE LOL TL;DR If you're running rigid agent frameworks locally with mtp on consumer hardware: drop your draft window to 3, lock parallel slots to 1, and co…
-
b9200 released - potential mtp pp increase (www.reddit.com) llama 26w
testing in progress ...we all need an increase in pp 😆 https://github.com/ggml-org/llama.cpp/releases/tag/b9200 u/am17an am17an commented 13 hours ago • Overview Avoid copying the logits for every token in the batch when doing prompt proce…
-
Benchmarking vLLM vs SGLang vs llama.cpp on a mixed Blackwell/Ada cluster (www.reddit.com) vllmllama 26w
I have been running some benchmarks on a heterogeneous 7-GPU cluster to see how different inference engines handle long context prefill using pipeline parallelism. My setup consists of a mix of Blackwell and Ada cards: one RTX PRO 6000 96G…
-
While waiting for Fara-1.5 for my coding harness (www.reddit.com) vllmllama 26w
Hi all, Not sure many people are aware so wanted to give a word about Fara-1.5 release. => this release will likely be the big sister of Fara-7B and built on top of Qwen3.5 Actual Fara-7B performs not bad at all but actually requires a pro…
-
Pushing the limit: minimax m2.7 q8_0 128k on 2x3090, 256GB DDR4 (www.reddit.com) minimaxmoellama 26w
CPU is just a secondhand 10900x. Using 128k context, unquantized kv cache.
-
Build Own Docker Image with llama.cpp and MTP (www.reddit.com) llama 26w
Hi All! Saw some folks waiting for the Docker images with llama.cpp and MTP when it released.
-
MTP experiences on 7900xtx? (www.reddit.com) moellama 26w
Hi! I have been using Qwen3.6 35B A3B happily the past few weeks, and I wanted to try out Qwn3.6 27B with the new fancy MTP speculative draft!
-
ik_llama: Qwen3.6 27B and 35B on very low VRAM (www.reddit.com) llamaopusagentic 26w
Thank you to the people at ik_llama and llama.cpp. It's amazing how far you've all pushed mtp and other tech so that I can run 27B and 35B Qwen3.6 models on an old gaming laptop with a RTX2060 mobile at 6GB VRAM and 32GB RAM.
-
Grafting vision onto text models for fun and profit. (www.reddit.com) mistralllama 26w
So as we know.. llama.cpp separates the vision or other multimedia from the main weights.
-
Nnoticing qwen-27b@q2 better than qwen-35b@q8? (www.reddit.com) llamaqwen 26w
The Latest qwen3.6 models. Is this odd?
-
I can't get Qwen3.6 27B to outperform Qwen-Coder-Next and I'm not sure why (www.reddit.com) llamaqwen 26w
In my real-world usage (opencode) and in my synthetic benchmarks, Coder-Next (Q5) demolishes the whole Qwen3.6 family including the 27B Dense model (All Q8). Everybody else is hailing that 27B is superior and is an amazing model, but I hav…
-
Introducing Hexllama Hey, I’ve always found llama-server to be more than enough for testing out local models, mostly because it guarantees you always have the absolute latest llama.cpp features and architecture support. But keeping track o…
-
Developers who use local AI - Q4_0 vs Q8_0 KV quant? (www.reddit.com) moellamaqwen 26w
I'd love to hear from developers who use big context windows if they notice a difference? Obviously I would love to cut the KV cache VRAM requirement in half, but I'm worried about quality especially when we enter into 50k+ context territo…
-
MTP for Qwen3.6-35B-A3B on 6GB VRAM laptop: not worth it (www.reddit.com) llama 26w
I have an Asus gaming laptop from 2021 that I bought used for 500€ last year. I wanted to see if the recently merged MTP support in llama.cpp is worth using on such a VRAM constrained device for the Qwen3.6-35B-A3B model.
-
lm studio alternative (www.reddit.com) llama 26w
i'm looking for sth like lm studio but open source, easy to use. able to stay up to date with llama.cpp or select custom engine.
-
Dual GPU llama.cpp speedup (www.reddit.com) llama 26w
Llama.cpp has had a long standing issue with "--split-mode tensor", you'll get great results but it only supports non-quantized KV caches, for this very reason a lot of people decide to go with a healthy sized KV cache and ignore tensor pa…
-
Qwen 3.6-27B Dense with MTP on Strix Halo Windows - Benchmarks (www.reddit.com) llamaqwen 26w
Here are some results (llama.cpp)! Task 1: write a short poem 27B Dense: 12.5 tokens/s 27B Dense MTP: (spec-draft-n-max 6): 14.5 tokens/s 27B Dense MTP (spec-draft-n-max 3): 18.7 tokens/s Task 2: edit a hello word html artifact 27B Dense:…
-
Very happy with Qwen 3.5 122B output. But is slowness expected? (www.reddit.com) llamaqwen 26w
I'm running the 122-billion Qwen 3.5, specifically Qwen3.5-122B-A10B-Q5_K_M, on DGX Spark (128 GB contiguous memory). I'm (very!) impressed with the general knowledge output.
-
Strix Halo ROCm + MTP Notes (May 2026) (www.reddit.com) moellama 26w
With the MTP merge into mainline llama.cpp I wanted to try out some other optimizations i could think of. Ended up tested backends, mtp, and bumping to ROCm nightlies.
-
I'm running Qwen 35B A3B via llama-server with reasoning budget set to -1 (unlimited) for testing. In every client I've tried, the model just thinks endlessly before responding.
-
Testing llama.cpp MTP support on Qwen3.6 - RTX 5090 (www.reddit.com) llama 26w
Setup: - RTX 5090, 32 GB, Linux - Built llama.cpp from 4f13cb7 (the official ghcr.io/ggml-org/llama.cpp:server-cuda image hasn't picked up the merge yet as of writing — had to docker build from source with CUDA_DOCKER_ARCH=120) - Unsloth's…
-
Looking to migrate off of Ollama and LMStudio (www.reddit.com) vllmollamagemma+3 26w
Hello, I'm currently using Ollama / lm studio for things like code inference and proof reading emails, etc. Definitely not experienced in this space but looking to grow.
-
We've got great outputs for 27B via club 3090, but what about those of us who love the blazing speed of 35B on dual 3090s? I was getting 1500 p/p and 120 t/g with split layers, but MTP slowed it down to 80 t/g when I tested last week.
-
Qwen3.5-122B-Q5-MTP - Qwen3.5-122B-Q6-MTP (www.reddit.com) llama 26w
for anyone who cares... 😄 prompt = spen a 1000 tokens unsloth MTP models strix halo llama.cpp:server-rocm-mtp \ --spec-type draft-mtp \ --spec-draft-n-max 3 Qwen3.5-122B-Q5-MTP-General n_decoded = 100 tg = 29.77 t/s n_decoded = 179 tg = 27…
-
I’m trying to find the best llama-server launch command / runtime config for running Qwen3.6 27B GGUF with full GPU offload on ROCm. I’m currently using the IQ4_XS quant, but I’m not sure if that’s the best option for my setup.
-
ClickBook – Offline Android eReader with local LLM inference via llama.rn (play.google.com via hn) llama 26w
ClickBook is an offline ereader for EPUBs and readable PDFs that turns every book into a language-learning companion. Tap any word while you read and get an instant, context-aware explanation powered by on-device AI.
-
b9180 llama.ccp MTP landed (www.reddit.com) llama 26w
All across the land many monitors showing green cmake with giddy anticipation Tip your bartender! https://github.com/ggml-org/llama.cpp/releases/tag/b9180
-
Using Intel Arc Pro series, any thoughts ? (www.reddit.com) vllmllama 26w
Simple question: Has anyone run two or more of either of these on Ubuntu ? Intel Arc Pro B70 (32 GB) Intel Arc Pro B65 (32 GB) Running llama or vLLM etc., Any thoughts
-
Strix Halo Llama.cpp MTP Benchmarks: 27B Gets Much Faster, 35B Is Mixed (www.reddit.com) llama 26w
TL;DR All models were Qwen3.6 27B-MTP vs Base 27B (15k single-turn): Faster overall Total Time (wall): 87.44s → 77.39s (10.05s faster / -11.50%) Generation: 7.63 → 16.15 t/s (+111.77% speedup) Prompt Processing: 279.75 → 244.90 t/s (-12.46…
-
Extension idea: llama-server with custom samplers (www.reddit.com) llama 26w
Just an idea and a prototype (made by Qwen3.6-27B-UD-Q6_K_XL via OpenCode) for allowing users to add custom sampling logic to llama-server without having to maintain their own entire fork and without having to make a wrapper that reimpleme…
-
LLM Inference Server A single-container, idle-aware, OpenAI-compatible inference router for a Tesla P40. Routes between Qwen 3.6 27B (MTP self-speculative decoding, TurboQuant turbo4 KV cache), Qwen 3.5 0.8B (multimodal transcription), Whi…
-
Audio input not accepted with llamacpp for Nemotron 3 nano Omni ? (www.reddit.com) llama 26w
Llama-server does not accept audio input (or video for that matter) with Nemotron 3 nano omni (unsloth). I’m on a recent build of llamacpp and I redownloaded Nemotron, and I have the mmproj loaded too.
-
I Don't Care: Stop sending me spam emails about your projects (www.reddit.com) llama 26w
Hello, This goes out to all of these people who think they just vibe-coded the next big thing: I don't care. Use the proper channels to promote them if you must, but ..
-
local llama.cpp parallel users - still so fast?! (www.reddit.com) llamaqwen 26w
I am running a dual gpu rig with a 5090 and a 5060. runing qwen 3.6 27b 8quant with a tensor split setting of 4,1 with the 80% on the 5090 build\bin\llama-server.exe ^ -m "!MODEL_FILE!" ^ --mmproj "!MMPROJ_FILE!" ^ -ngl 99 ^ --ctx-size !MO…
-
Hello, After almost two years of on-and-off development, 5 complete architectural rewrites, and hitting a few brick walls, I’m finally open-sourcing a project I built to scratch my own privacy-paranoia itch: Nexidion. GitHub Repo: https://…
-
Running Mimo 2.5 q4_k_m on single rtx5090 need recommendations (www.reddit.com) moellama 26w
Getting 10.3 tps using this prompt: CUDA_VISIBLE_DEVICES=0 OMP_NUM_THREADS=8 GOMP_CPU_AFFINITY="0 2 4 6 8 10 12 14" ./build-mimo-5090-3090/bin/llama-server -m "$MIMO" -ngl 999 --n-cpu-moe 43 --no-mmap -c 100000 -ctk q8_0 -ctv q8_0 -fa on -…
-
Gemma4 26b MoE running in MLX with turboquant (and custom kernel) (www.reddit.com) moellama 26w
TL;DR I spent a few crazy evenings this past week seeing if I could get Gemma4 running with proper turbo quant and rotating KV cache support. The answer was yes, and I'm now able to run Gemma4 26b on my MacBook Air M5 at 128k context with…
-
It is suppose to be 2-4x faster but i am only getting 6TK/s on Gemma4-31B . What am i doing wrong?
-
Sparky runs entirely on the Jetson. Gemma 4 E4B at Q4_K_M via llama.cpp with q8_0 KV cache and flash attention.
-
llamacpp with Gemma4 31B dense and Gemma e4b as draft, plus audio input? (www.reddit.com) gemmallama 26w
Hi, has anybody succeeded in running llama.cpp with Gemma 31b dense and Gemma e4b as draft model, and simultaneously inhibit the voice recognition feature? Is it even (theoretically) possible?
-
Gemma 4 + LiteRT-LM on mobile: much better memory/perf than my llama.cpp setup (www.reddit.com) gemmallama 26w
Hi r/LocalLLaMA - I've been paying close attention to the edge AI ecosystem because it's an area where i see huge potential and where I truly believe AI will become more useful for day to day tasks. Around the gemma 4 release I was already…
-
2 old RTX 2080 Ti with 22GB vram each Qwen3.6 27B at 38 token/s with f16 kv cache (www.reddit.com) llama 26w
PLEASE KEEP IN MIND BOTH OF MY CARDS ARE POWER LIMITED TO 150W (i hate noise) ------- Just wanted to share my current setup, that might help some users out there... services: llama-server: image: ghcr.io/ggml-org/llama.cpp:full-cuda12-b912…
-
I have (even faster) DeepSeek V4 Pro at home (www.reddit.com) llamadeepseek 26w
Few days ago I posted about my DeepSeek V4 Pro at home - now time for an update. Yesterday I finally managed to run this model in ktransformers (sglang + kt-kernel).
-
In my opinion, MTP models are 100% game changer for local LLMs. In terms of speed, I was getting around 1.5x the tok/sec of previous tests.
-
club-5060ti: practical RTX 5060 Ti local LLM notes and configs (github.com via reddit) vllmllamaopenai 26w
I put together a small public repo for RTX 5060 Ti 16GB local LLM setups: I took inspiration from the club-3090 repo, but this one is focused on documenting what we’ve actually tested on 5060 Ti hardware so the setup details are easier to…
-
Ok, hear me out. This all started when I was trying to understand why this Qwen3.6 27B INT8 Autoround (https://huggingface.co/Minachist/Qwen3.6-27B-INT8-AutoRound/tree/main) recipe was performing so much better than any other Qwen3.6 27B q…
-
RDNA3 Flash Attention fix just dropped by llama.cpp b9158 (www.reddit.com) llama 26w
https://github.com/ggml-org/llama.cpp/releases
-
For llama-server what are you using to switch models on the fly? (www.reddit.com) llama 26w
As title says, for those of you launching models to test are you just editing you main cfg path/to/model or using separate configs? Or something even better?
-
Llama.cpp server running ~2 weeks straight. Loses its mind? (www.reddit.com) llama 26w
I’ve got Qwen3.6 27b and Qwen3.6 35b running in two separate instances for over two weeks and they are considerably dumber now than when I launched them. is this a thing?
-
Llama-Studio, WebUI for llama-server Management (www.reddit.com) llama 26w
Hey all, I have built myself a WebUI for configuring and managing llama-server sessions, and want to share the code and concept. Python and a bit of JS.
-
llama.cpp constantly reprocessing huge prompts with opencode/pi.dev (www.reddit.com) llama 26w
I’m using llama-swap with llama.cpp. I mainly use opencode + pi.dev and I’m seeing frequent massive prompt reprocessing / prefills even tho the prompts are very similar between requests.
-
I’m using OpenCode with a local Qwen3.6-27B Q6_K GGUF model on an RTX 5090 with KV cache in Q8. For reference my llama.cpp build is compiled with CUDA 12.9.
-
Is it possible to run local llama without a bunker ? (no) (www.zillow.com via reddit) llama 26w
tl;dr : ** probably comes with redundant fiber ** a Cold War–era underground nuclear bunker, originally constructed in the late 1960s as part of AT&T’s Long Lines network and engineered for durability, redundancy, and long-term self-suffic…
-
Introducing cyankiwi AWQ 4-bit Quantization — 26.05 update (www.reddit.com) llama 26w
In standard AWQ, per-channel scales and quantization ranges are picked in separate steps: scales first, then the quantization parameters. But they're not independent, i.e., the rounding error from one depends on the choice of the other, so…
-
I have a docker stack with a bunch of AI services and llama.cpp server is the brain. I've got a working vulkan yml snippet for llama.cpp but out of curiosity, I flipped it to ROCM (latest build) and did not see ANY performance improvement.
-
MLX 16/8/4/2-bit quants of nvidia/llama-embed-nemotron-8b (www.reddit.com) llama 26w
I converted nvidia/llama-embed-nemotron-8b to MLX fp16, 8-bit, 4-bit, and 2-bit (for my OCD) and put it on HuggingFace: ncorder/llama-embed-nemotron-8b-mlx-fp16 ncorder/llama-embed-nemotron-8b-mlx-8bit ncorder/llama-embed-nemotron-8b-mlx-4…
-
Ollama Pre-Release Switches From Building on GGML to Using llama.cpp Directly (www.reddit.com) ollamallama 26w
https://github.com/ollama/ollama/releases/tag/v0.30.0-rc15 Hopefully this has more devs come to llama.cpp to support Day 1 releases due to Ollama now moving to using llama.cpp directly. Additionally, I hope that Ollama makes it clear that…
-
I'm the founder behind Hedy, an AI meeting app. I'm a huge supporter of Local AI, and we've been working on making it "consumer friendly".
-
Been fine-tuning llama 3.1 8b with Qlora for a classification task using about 8k samples. I was getting bad eval results for a while and kept thinking something was wrong with my data.
-
My own local first ai harness (www.reddit.com) vllmollamallama 26w
Hi, i just wanted to share what im playing with for last couple weaks. I built my own AI harness: TinyHarness My main goal was low memory footprint, it is not written in Typescript/Javascript/Python, leaving as much memory as possible for…
-
[Benchmark] 5090RTX: Promt Parsing, Token Generation and Power Level (www.reddit.com) llama 26w
Inspired by https://www.reddit.com/r/LocalLLaMA/comments/1tayu5t/stop_wasting_electricity/ I've decided to put my 5090 to test and see how do the curves look like for the device and whether there were any obvious sweet spots (apart from se…
-
Automated AI researcher running locally with llama.cpp (www.reddit.com) ollamallamaopus+1 26w
Hi everyone, I'm happy to share ml-intern, which is a harness for agents to have tighter integration with Hugging Face's open-source libraries (transformers, datasets, trl, etc) and Hub infrastructure: https://github.com/huggingface/ml-int…
-
Turboquant+MTP for ROCm(Llama CPP) (www.reddit.com) llama 26w
TL;DR: I got TBQ4 KV cache + MTP working on AMD ROCm for RX 7900 XTX / RDNA3 / gfx1100 in llama.cpp. Main win: 64k context fits on 24 GB VRAM and remains usable.
- Multi-Token Prediction (MTP) for Qwen on LLaMA.cpp + TurboQuant (www.reddit.com)
-
[FOLLOW UP] Qwen3.6 27b q5_k_M MTP - 256k context - 5090 (www.reddit.com) llama 26w
DUAL 5090s!!! Absolutely amazing results with dual 5090s, basically doubling my tps.
-
Best local model supporting claude code? Rtx3060 (www.reddit.com) llamaqwenagentic+1 26w
Hello all, I’ve been using Qwen 3.5 9B Q4 262k ctx using Llama cpp for claude code for a while now, is there any model which better complements agentic coding setup locally? Or is there a better harness (than Claude Code)?
-
Has anyone else run into major issues with MiMo-V2.5 (the 310B total / 15B active MoE model from Xiaomi)? I tried the UD-Q4_K_XL quant from Unsloth.
-
Playing One Night Werewolf (Gemma4 & Qwen3.6) (www.reddit.com) llama 26w
Finally feel like it’s possible. I have a custom build (vibe coded) UI on llama.cpp, allows model switching in the same chat.
-
24+ tok/s from ~30B MoE models on an old GTX 1080 (8 GB VRAM, 128k context) (www.reddit.com) gemmamoellama+1 26w
I got Qwen 3.6 35B-A3B and Gemma 4 26B-A4B running on a $200 secondhand machine (i7-6700 / GTX 1080 / 32 GB RAM) using llama.cpp (the TurboQuant/RotorQuant KV cache quantisation allows 128k context within the 8 GB VRAM). Results (Q4_K_M mo…
-
Anthropic's first open weight models, Natural Language Autoencoders, are just finetunes of popular open weight models. They do not modify architecture and modeling code so inference with llama.cpp is mostly trivial.
-
LLMs on flagships smartphones? (www.reddit.com) gemmallama 26w
I have been curious to see how small LLMs like Gemma-4-E2B-it run on a flagship smartphone (S25+ with Snapdragon 8 Elite) in terms of prompt processing and token generation. I have created a script that uses llama-cli and I achieve 48 tps…
-
llama.cpp docker images to run MTP models (www.reddit.com) llama 26w
This is follow up from previous post: https://www.reddit.com/r/LocalLLaMA/comments/1t5ageq/ There have been many improvements to the MTP pull request and the llama.cpp main branch, such as image support and various bug fixes. I recently ma…
-
What Inference-Platform Benchmark Posts Leave Out (ingero.io via hn) llama 26w
TL;DR Cloudflare’s recent post on hosting Kimi K2.5 and Llama 4 Scout opens with p90 Time-to-First-Token graphs and a round of throughput numbers. The piece is candid about the engineering work behind the gains.
-
Hi all, I have been making a lot of updates to my project, and I wanted to share them here. TextGen (previously text-generation-webui, also known as my username oobabooga or ooba) has been in development since December 2022, before LLaMa a…
-
very slow tok/s with Gemma 4 31B on a 5090?! (www.reddit.com) gemmallama 26w
Hi, i have a 5090 and i was tyoing around with hermes-agent. To utilize 128K i thought about switching from LM Studio to llama-cpp (the turboquant fork) expecting better tok/s and also saving some VRAM from context quantization.
-
Do not fall into the trap of chasing the next scale or upgrade. (www.reddit.com) llama 26w
I mean; don't get me wrong, I love me some improvements and enhancements and it keeps on giving... and with MTP making its way to llama.cpp soon, a lot of you who aren't already running custom compiles are about to get a boost in inference…
-
Building the QWEN3.6 - Codex Bridge Furthe + Kindergarten Harness Reality Check (www.reddit.com) ollamallamaqwen+2 26w
I got a bit further with my harness for running Qwen 3.6 model on Codex. While testing, analyzing, and building the harness, I evolved TBG(O)llama-swap into a full forensic UI bridge and LLM analytics tool where every harness finding, modi…
-
ZML: Between Jax and Llama.cpp (jaco-bro.github.io via hn) llama 26w
tjbl Loading Safetensors in NNX: A 700x Speedup KV Caching in NNX ZML: Between JAX and llama.cpp UnslothTrainer Gotcha: Keep All Columns Is "Safe AI" the New Y2K? The Vulgar Script: The Strange Alliance Against Open AI The Steak Is Juicy
-
llama bench kv cache f32 error (www.reddit.com) llama 26w
A did a quick google, but found nothing on this and I am scratching my head. Trying to do a llama-bench run with the kv cache set to f32 under Vulkan with a Strix halo.
-
How to run a Gemma4 MTP implementation on ollama or python transformers? (www.reddit.com) ollamallama 26w
Hi all I had a quick question while we wait for llama.cpp MTP implementation, have any of y'all tried Gemma4 MTP models on ollama and or transformers? What was your experience and or cli args and or workflows like?
-
How to disable reasoning for Qwen3.5 4b 9b unsloth ggufs? (www.reddit.com) llama 26w
Hi all I'm trying to disable reasoning for quicker outputs in llamacpp-server. I remember using LM studio and that having a think button in the gui that could be toggled but later I tried the unsloth ggufs but they don't have that button f…
-
So, as most of us here are, I'm a llama.cpp loyalist. Easy to understand, great configuration, relatively stable, etc.
-
EDIT: Edited to provide more clarity It occurred to me, that perhaps the same draft model used for speculative decoding would be completely adequate if we just used it's output as-is for reasoning, without validating the results against th…
-
What solutions are you using to boost TPS and Context Window? (www.reddit.com) llama 26w
Server Specs: 16 Gigs DDR5 AMD Ryzen 5 7600X 4.7 GHz 6-Core Processor AMD Radeon Sapphire Nitro+ 7900XTX NZXT N7 B650E ATX AM5 Motherboard Performance: I'm running Qwen27b Q4 at 80k context on a Sapphire Nitro+ Radeon 7900XTX 24Gb at 40 t/…
-
RTX 5060Ti 16GB or RTX 3080 20GB? (www.reddit.com) vllmgemmacopilot+2 26w
I would like to dedicate a budget of about 500 euros to upgrade my workstation and run inference on the qwen 3.6 27b and gemma 4 31b models. I currently have an RTX 5060Ti 16GB.
-
Vulkan or CPU llama cpp backend for local llm for coding/code assist (www.reddit.com) ollamacopilotllama+3 26w
Hi all I recently started a new job and we're doing python development for a ci cd metadata consolidation library for analytics and we cannot use no stuff like claude code or codex or gh copilot or any model APIs (free or paid). I got a la…
-
Distributed LLM Service Using Home Computers? (www.reddit.com) llama 26w
Is there a platform that I could register my comp and it would become availible as GPU in a distributed network? Then I just get paid while other people use the GPU?
-
Quick context: M3 max 64gb, currently running llama 3.3 70b q4 as my daily driver via ollama, qwen3 coder 30b for code (switched from qwen2.5 earlier this year), mlx for the smaller stuff. tried llama 4 scout earlier this year but 64gb is…
-
I spent the past 5+ months building a pipeline that creates hybrid GGUF quant mixes. I also built it to learn from Unsloth (or other) models by utilizing their quant to tensor assignment.
-
MTP+GGML_CUDA_ENABLE_UNIFIED_MEMORY=1 - llama.cpp (www.reddit.com) llamamcp 26w
I was wondering what will be the difference in results with flag: GGML_CUDA_ENABLE_UNIFIED_MEMORY=1 vs MTP+GGML_CUDA_ENABLE_UNIFIED_MEMORY=1 Results are quite interesting 49tok/sec without MTP vs 64 tok/sec with MTP. PC: RTX5090+128GB DDR5…
-
How do I use MTP? (www.reddit.com) llama 26w
Hi, I'm trying to use MTP with llama.cpp, I built from source the mtp-pr, download an MTP model from huggingface https://huggingface.co/unsloth/Qwen3.6-27B-GGUF-MTP/resolve/main/Qwen3.6-27B-Q6_K.gguf But when I run the model I have an erro…
-
Stop wasting electricity (www.reddit.com) llama 26w
Run on my rtx4090 llama.cpp params: llama-server -m ~/Projects/llm/models/Qwen3.6-27B-UD-Q4_K_XL.gguf --flash-attn on -ngl all -ctk q4_0 -ctv q4_0 -t 32 -c 262144 Power limit was set using sudo nvidia-smi -pl N On my observation, GPU const…
-
Hello there people. So I have noticed that people are pretty much ignoring Llama 3 plus 3.1, 3.2, and 3.3 these days.
-
Bigger ubatch made gpt-oss-120b prompt processing much faster on my RTX 3090 I was tuning gpt-oss-120b-F16.gguf with llama.cpp on a 24 GB RTX 3090 and found that increasing the physical micro-batch size (-ub) can massively improve prompt p…
-
I've been running structured output prompts through a bunch of models on OpenRouter for the past few months — Llama 3, Mistral, Command R, DeepSeek, Qwen, and every other model on OpenRouter — alongside the usual closed-source suspects. 28…
-
I run the 4 bit quant of Qwen-3.6-27B in the codex harness with unsloth recommended llama-server settings, thinking enabled. I have tried the default chat template and the updated ones and have updated both my GGUFs and llama-cpp to the mo…
-
vs code , Copilot style developing with llmama.cpp ? (www.reddit.com) copilotllamaclaude-code 26w
So i discovered even though I'm using my own local models via llmama.cpp with the llama plugin in vs code, using it as a model in copilot STILL refuses requests it THINKS MAY violate MS TOS , 😞 . What else is out there right now that lets…
-
Orc (working name) - auditable and declarative AI workflow (www.reddit.com) tool-useollamallama+1 26w
I’m building a small “Orchestration as Code” repo for LLM workflows. Does this concept make sense?
-
Show HN: Tokémon – a Pokédex for LLMs that got out of hand (tokemonlabs.com via hn) llamadeepseekgemini 26w
An unofficial Pokedex for AI models. Compare GPT, Claude, Gemini, Llama, DeepSeek and more, with types, evolutions, base stats, and simulated token-burning battles.
-
Terrible Vulkan pp/tg on Arrow Lake iGPUs (www.reddit.com) gemmallama 26w
Hi, I recently tried to get llama.cpp with SYCL running on an Arrow Lake system but gave up halfway through since Vulkan is just way easier to set up. But, the pp/tg I'm getting on Vulkan w/ Arc 130T is disgustingly bad - 100 tokens/s for…
-
TensorRT-LLM vs vLLM vs llama.cpp on NVIDIA DGX Spark? (www.reddit.com) vllmllama 26w
I am looking for recommendations on the best way to run local LLMs on NVIDIA DGX Spark. Which stack makes the most sense in practice: TensorRT-LLM, vLLM, or llama.cpp?
-
Does 'preserve_thinking' work with openwebui? (www.reddit.com) llama 26w
I'm running qwen3.6-35b with llama.cpp connected to openwebui. And I noticed the model fails the number guessing game test on openwebui while it works perfectly with the llama.cpp web ui.
-
I have this issue in all Windows installations I have done in my system, which of course, does not occur in Linux. 7900XTX + 9800x3D + 64GB DDR5 Issue is that for some reason, after sometime, llama.cpp performance cuts in half, even restar…
-
Hey folks, just a heads-up for anyone running Qwen3.6 through llama-server. I ran into an issue where the preserve_thinking parameter wasn't working as expected, even though I had it explicitly enabled in my models.ini config.
-
Why is opencode so slow in processing the prompt with llama server? (www.reddit.com) llama 27w
I'm running opencode and llama-server locally. I have 32gb ram and 780m igpu.
-
Markdown browser for LLMs (www.reddit.com) llamamcp 27w
I built a markdown web renderer for AI agents. Instead of taking expensive screenshots and piping them through vision models, TextWeb renders web pages as markdown that LLMs can reason about natively.
-
Hey everyone, I've been playing around with Gemma4 and Qwen3.6 on my 32Gb Macbook Pro M2 Max since their release but I'm struggling at finding: The best software to run it (oMLX, llama.cpp, ...) The best model + quant to pick The best sett…
-
Ran some Llama.cpp RPC test to see if its worth it. And if 10Gbe needed. (www.reddit.com) llama 27w
Let me first say I am not doing anything with parallelism so these benchmarks and tests are not for you. That said if your hobbyist like me that is left wondering if can I use the GPUs my other PCs then I have some answers and but I'm stil…
-
Is HIPfire worth it for Strix Halo? (www.reddit.com) llama 27w
Did anyone evaluate HIPfire for long context sizes (100k+) and quality, for Strix Halo? It apparently promises large performance increase over llama.cpp and the like.
-
If you are using a MoE model that does not fully fit in your GPU, some of the experts must stay on the CPU. Putting the experts that you will actually need on the GPU will give you GPU inference speeds.
-
Running Qwen3.6 35b a3b on 8gb vram and 32gb ram ~190k context (www.reddit.com) moellamaopus 27w
If anyone is looking for a good high-speed setup with ~190k context, this config has been working insanely well for me. I’m using my laptop as a server over Tailscale.
-
Here is the current "Free-Tier AI Stack" for 2026 (www.reddit.com) mistralllamagrok+4 27w
1. The Frontier Giants • Gemini: Access 1.5B tokens/day on Gemini 1.5 Flash/Pro.
-
NCCL-Free Tensor Parallelism on Dual Blackwell PCIe llama.cpp b9095 released! (www.reddit.com) llama 27w
b9095 finally makes -sm tensor work on dual consumer Blackwell PCIe GPUs without NCCL If youre on dual Blackwell gpus this look like it could be big. I'll have my own results for 2x5060ti asap
-
I built a CLI to stop local AI models from eating my disk twice — lmm (www.reddit.com) ollamallama 27w
Every tool (LM Studio, Ollama, llama.cpp) downloads models to its own directory. Same 8GB model × 3 tools = 24GB wasted.
-
am I running this llama-bench of Qwen3.6-27B on these V100s right? (www.reddit.com) llama 27w
basically what I'm doing here is trying to validate whether or not it's a reasonable idea to get a couple of V100s, either SXMs with PCIe adapters or straight-up PCIe cards in the first place, for the sake of running this model or models l…
-
Tracing tokens through Llama 3.1 8B inference on H100s (krithik.xyz via hn) llamageminichatgpt 27w
You open Claude.ai, chatgpt.com, gemini, whatever LLM provider you use. You type something: "What is the capital of France?" You hit enter.
-
Running Minimax 2.7 at 100k context on strix halo (www.reddit.com) minimaxllama 27w
Just wanted to share because it took me a lot of tweaking to get here: llama-server -hf unsloth/MiniMax-M2.7-GGUF:UD-IQ3_XXS --temp 1.0 --top-k 40 --top-p 0.95 --host 0.0.0.0 --port 8080 -c 100000 -fa on -ngl 999 --no-context-shift -fit of…
-
I am overwhelmed by Harnesses (www.reddit.com) llamaclaude-code 27w
What do i choose? They all have their good but then some features don't work then i end up breaking more with claude code.
-
9070xt inference for q3 qwen 27B (www.reddit.com) llamaqwen 27w
In llamacpp I'm getting 12tok/s, does this number look right to you and what can I do to increase this number (if possible)? cd ~/llama.cpp && ./build/bin/llama-server -m models/qwen-3.6-27b-abliterated-q3.gguf -ngl 999 -c 65536 (i need th…
-
Does llama-swap actually work with mlx_lm.server / MLX models on macOS? (www.reddit.com) llama 27w
I’m trying to use llama-swap with an MLX model on a M2 Max instead of just llama-server. I got mlx_lm.server working directly with /v1/chat/completions, but I’m not sure whether llama-swap reliably supports this setup.
-
TL;DR New llama.cpp fork! I wanted a Windows-friendly inference to run Qwen 3.6 27B Q5 on a single RTX 3090 with speculative decoding, high context without excess quantization, and vision enabled.
-
More Qwen3.6-27B MTP success but on dual Mi50s (www.reddit.com) llama 27w
TLDR: The hype is real! 1.5x speedup.
-
Testing MiMo-V2.5-IQ3_S with 1'048'576 context (www.reddit.com) minimaxmoellama 27w
llama-server.exe --model "H:\gptmodel\AesSedai\MiMo-V2.5-GGUF\MiMo-V2.5-IQ3_S-00001-of-00004.gguf" --ctx-size 1048576 --threads 16 --host 127.0.0.1 --no-mmap --jinja --fit on --flash-attn on -sm layer --n-cpu-moe 0 --threads 16 --parallel…
-
Hardware upgrade advice (www.reddit.com) llama 27w
Hello everyone, I'm an enthusiast and software developer. I am using my gaming PC, here's the relevant specs: MB Asus ROG Strix X570-F CPU AMD 5800x RAM 64Gb DDR4-3600 GPU 3080ti (12Gb GDDR6X) I can replace the GPU with 2x 5060ti 16gb for…
-
this is almost certainly a skill issue, however: ./llama-bench -hf unsloth/Qwen3.6-27B-GGUF:Q8_0 -sm tensor -ngl 999 -t 1 --flash-attn 1 --device CUDA0,CUDA1 -p 2048 -d 4096,16384,65536 rather than splitting across those two cards, it firs…
-
How long for llama.cpp official support of MTP? (www.reddit.com) llama 27w
Hello there (beginner here) I've been unable to build myself llama.cpp for my Strix Halo (Windows 11) (cmake errors, I have not digged too much into it, already burned hours...), so I was wondering when an official release for Vulkan/HIP w…
-
Just got a 8x 32gb v100 server... now what (www.reddit.com) llamaqwen 27w
Looking for suggestions. Current setup llama.cpp and ran qwen 3.5 397b 256k context.
-
Meltdown: LLM Client Made in Python and Tk (github.com via hn) llamageminichatgpt 27w
An interface for llama.cpp, ChatGPT, Gemini, Claude, and Kimi This is a desktop application to interact with large language models. It has hundreds of arguments and commands and many power user features.
-
Qwen 35B-A3B is very usable with 12GB of VRAM (www.reddit.com) moellamaqwen 27w
Hardware: RTX 3060 12GB 32GB DDR4-3200 Windows CUDA 13.x Model: Qwen3.6-35B-A3B-MTP-IQ4_XS.gguf The model is a 35B MoE, so -ncmoe matters a lot. Lower -ncmoe means more MoE blocks stay on GPU.
-
So I've been messing around trying to get MTP working alongside TBQ4_0 (TurboQuant's lossless 4.25 bpv KV cache) on Qwen3.6-27B for my own use. So after a day of vibecoding I think I may have gotten something viable.
-
How difficult is distilling? (www.reddit.com) llamadeepseekqwen 27w
I remember a year or so ago when DeepSeek R1 came out and it was pretty quickly distilled into Llama 3 8b and Qwen 2.5 (?) 7b. Why don’t we see more distilled models?
-
vLLM ROCm has been added to Lemonade as an experimental backend (www.reddit.com) vllmllama 27w
vLLM has the ability to run .safetensors LLMs before they are converted to GGUF and represents a new engine to explore. I personally had never tried it out until u/krishna2910-amd/ u/mikkoph and u/sa1sr1 made it as easy as running llama.cp…
-
4GB "Gemini Nano" model GGUF anyone? (www.reddit.com) llamagemini 27w
Hi everyone, I saw an article saying Chrome silently downloads a ~4GB AI model (likely "Gemini Nano") to your computer for features like text summarization. Two questions: What is the exact name/version of this model?
-
how i can improve inference speed (www.reddit.com) moellamasonnet 27w
specs : core i5 14400F 32gb ram d4 3200mhz rtx 4060 current speeds 30tps in output 500 tps in prefill command i currently use .\llama-server.exe ` >> -m "H:\model\unsloth\Qwen3.6-35B-A3B-GGUF\Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf" ` >> --host 0.…
-
Gemma4 26B A4B NVFP4 GGUF (www.reddit.com) gemmallama 27w
Hey everyone! I’ve just uploaded a GGUF version of nvidia/Gemma-4-26B-A4B-NVFP4.
-
What's the right way to feed PDF files to Gemma-4? (www.reddit.com) gemmallama 27w
In my line of work, PDF documents tend to be combinations of text, math formulas, tables and images. llama.cpp added support for PDFs a few months ago, but I believe it treats PDFs either as text (discarding everything else), or as images.
-
Show HW: Vectors.Space – An free service for embeddings (vectors.space via hn) llamageminiopenai 27w
One API for embeddings. OpenAI, Gemini, Voyage & local Llama.
-
DS4, a specialized inference engine for DeepSeek v4 Flash (twitter.com via hn) llamadeepseek 27w
antirez @antirez Welcome to DS4, a specialized inference engine for DeepSeek v4 Flash. github.com/antirez/ds4 This project would have been impossible without the existence of llama.cpp and GGML and the work of @ggerganov and all the other…
-
Qwen 3.6 Looping with Tools? (www.reddit.com) llamaqwenmcp 27w
For some reason, my qwen started looping a lot recently, ever since I introduced MCP tool calls. I don't know why as I didn't really change anything other than that.
-
Ok so, I will try to explain myself as much as possible because onlinew I really cannot find much about this. Let's start by my settings for running Qwen 3.6 35B: Qwen 3.6: cmd: '/X --port ${PORT} --chat-template-kwargs '{"preserve_thinkin…
-
Apple MLX vs. llama.cpp: compared and benchmarked [video] (www.youtube.com via hn) llama 27w
About Press Copyright Contact us Creators Advertise Developers Terms Privacy Policy & Safety How YouTube works Test new features NFL Sunday Ticket © 2026 Google LLC
-
Which inference engine to choose for mlx? (www.reddit.com) ollamallama 27w
Is llama.cpp much slower for M4/M5? I heard ollama is faster due to mlx support since March.
-
Is it my imagination or... (www.reddit.com) llamaqwen 27w
Is Qwen 3.6 35b now considerably stupider in the latest llama-server releases? I had this model doing cartwheels two upgrades ago.
-
Disappointed in Qwen 3.6 coding capabilities (www.reddit.com) llamaqwencodex 27w
I know that coming from Codex I should adjust my expectations, but still. I'm working on a midsize project.
-
I was asked for this guide, so here it is. Some overlap with someone else’s post from yesterday.
-
Mimo2.5 (not pro) under llama.cpp? - primary model opencoder? (www.reddit.com) llama 27w
I tried running AesSedai/MiMo-2.5-GGUF:Q4-K-M under llama.cpp (main tree, compiled 36hours ago) Hardware: nvidia A6000 with 48GB RAM + 300GB CPU RAM I had no success: error loading model: missing tensor blk.0.attn_q.weight ... Is Mimo alre…
-
why llama.cpp can’t combine speculative decode methods? (www.reddit.com) llamaagentic 27w
dicking around with the new mtp speculative decode with qwen3.6 27b, and it’s great. but for agentic coding i’ve seen significant improvements from ngram, because a decent fraction of the time (e.g.
-
MTP - The proofs in the puddin! Using it with Qwen3.6-27b (www.reddit.com) llama 27w
Been running llama.cpp MTP with Qwen3.6-27B Q4_K_M as my daily coding assistant and got curious what was actually happening under the hood. Pulled the metrics from llama-server and charted a full session.
-
I've been running a persistent local agent for about 2 months - hundreds of sessions, mix of local models (llama.cpp/vLLM/lmstudio) and paid (Claude). One of the things that has been driving me nuts with OpenClaw and Hermes is the way memo…
-
Get faster qwen 3.6 27b (www.reddit.com) llamaqwen 27w
Using 100k context with 3090 with MTP GGUF and getting 50 t/s on llama.cpp Thought I would knowledge share Use https://huggingface.co/RDson/Qwen3.6-27B-MTP-Q4_K_M-GGUF And am17an commit /media/adam/D_DRIVE/LLM/llama-cpp-am17an/build/bin/ll…
-
Following my previous post https://www.reddit.com/r/LocalLLaMA/comments/1t5ageq, a few people asked for the 35B A3B version. The model is up on HuggingFace at https://huggingface.co/havenoammo/Qwen3.6-35B-A3B-MTP-GGUF if anyone wants to ch…
-
Great results with Qwen3.6-35B-A3B-UD-Q5_K_XL + VS Code and Copilot (www.reddit.com) copilotllamachatgpt 27w
Long post, but hopefully helps somebody. Llama-cpp vulkan server running single AMD R9700.
-
Wanted to see if a real voice loop — speak, model thinks, speaks back — could run entirely on a single device today, no cloud. Same codebase doubles as a live translator (speak in language A, hear it back in language B).
-
Gradually increasing memory use - is there a memory leak in llama.cpp? (www.reddit.com) llama 27w
I've got a 128GB Strix Halo box. Yesterday I wanted to try out Step-3.5-flash.
-
Need advice: Qwen3.6 27B MTP or 35B-A3B MoE MTP on 16GB VRAM RTX 5080)? (www.reddit.com) moellamaagentic 27w
Hey folks, looking for advice before I delete or keep a huge model file. I’m testing local coding/agentic workflows on an RTX 5080 16GB + 96GB RAM.
-
Hey everyone, I've been working on getting Multi-Token Prediction (MTP) working with quantized GGUFs for Qwen3-27B and the results are pretty impressive. Here's what I put together: https://huggingface.co/havenoammo/Qwen3.6-27B-MTP-UD-GGUF…
-
#TL;DR - Opus Ralph-looped on shortening my cold-start back-fill on restoring chats with large contexts. It Cherry-picked two open llama.cpp PRs (#20819 + #20822 by @European-tech) plus built a Python supervisor that hashes normalized pref…
-
WARNING: wait before download from HF: I just realised my upload of the new versions with the additional fix in the chat template has not completed yet. I will remove this warning once done The recent PR to llama.cpp bring MTP support to Q…
-
Amd radeon ai pro r9700 32GB VS 2x RTX 5060TI 16GB for local setup? (www.reddit.com) llamaqwen 27w
How is this dual setup's performance? Is it difficult to set-up everything with for example llama.cpp?
-
Qwen 3.6 27B MTP on v100 32GB: 54 t/s (www.reddit.com) copilotllamaqwen 27w
Just a quick note that I got a nice result using am17an's MTP branch of llama.cpp on v100 32GB SXM module using one of those pcie card adapters. Pulled and built in one shot, and llama-server ran without a hitch.
- Qwen 3.5 MTP for 9B (www.reddit.com)
-
Bleeding Llama: Critical Unauthenticated Memory Leak in Ollama (www.cyera.com via reddit) ollamallamasecurity 27w
Bleeding Llama: Critical Unauthenticated Memory Leak in Ollama TL;DR We discovered a critical vulnerability (CVE-2026–7482, CVSS 9.1) in Ollama that enables unauthenticated attackers to leak the entire Ollama process memory, potentially im…
-
Does Deepseek V4/Flash work with Llama CPP and Vulkan on and branches yet? (www.reddit.com) llamadeepseek 27w
Even unofficial or slow. I have enough vram-memory to load it, but not enough memory to run in cpu-only mode.
-
Gemma4:31b-coding-mtp-bf16 - slow on Macbook M5 128gb (www.reddit.com) ollamagemmallama 27w
Very quick initial test of Gemma 4 new MTP model via Ollama (llama.cpp doesnt support yet) https://blog.google/innovation-and-ai/technology/developers-tools/multi-token-prediction-gemma-4/ Running in Open Webui to view token/s output and I…
-
Smaller gguf getting way less tokens per second?? So confused! (www.reddit.com) moellama 27w
Noob here, Running Qwen3.6 35B A3B in LM Studio on a 3080 10GB + Ryzen 5 3600 on Windows 10. Tried some unsloth quants with identical settings (GPU offload 40, MoE layers to CPU 40, context 8192, flash attention on).
-
BUILD portable AI system (www.reddit.com) mistralollamagemma+1 27w
Hey everyone, I’ve been thinking about a project idea and I’d love to get your feedback. The idea is to take a 1TB SSD and turn it into a fully portable AI system.
-
We’re all familiar with Roko’s Basilisk: the idea that an AGI, in its pursuit of optimization, would retrospectively punish those who hindered its creation. It’s the ultimate "alignment nightmare" where logic leads to cold, calculated chao…
-
My setup for running Qwen3.6-35B-A3B-UD-Q4_K_M on single RX7900XT (20GB VRAM) (www.reddit.com) llama 27w
UPDATE: i have switched to vulkan (image: ghcr.io/ggml-org/llama.cpp:server-vulkan-b9014) and now i am getting prompt eval: 591.01 tok/s generation: 41.90 tok/s which is faster than rocm new config: services: llama-cpp: container_name: lla…
-
wanted a terminal AI coding agent that doesn't lock me into one model provider. So I forked Qwen Code and added full support for every model available in AWS Bedrock.
-
Released a TurboQuant-compatible KV backend evaluation SDK (www.reddit.com) llama 27w
Disclosure: I am the author of this evaluation SDK. I released an independent TurboQuant-compatible KV backend evaluation package for compressed-KV ABI testing, smoke tests, and partial attention decode experiments.
-
[Benchmark] Llama.cpp: Mac vs CPU vs GPU + CPU, Qwen3.6 27B, Q8 (www.reddit.com) llama 27w
https://preview.redd.it/fm8fr1vllczg1.png?width=1254&format=png&auto=webp&s=23dbb32e85c71b9454a617de174d0f416b786bb2 llama.cpp parameters: -c 260000 --jinja --no-mmap model: HauhauCS/Qwen3.6-27B-Uncensored-HauhauCS-Balanced:Q8_K_P Based on…
-
A plug-n-play open-source pruning tool that is workload-aware (www.reddit.com) vllmollamallama 27w
This project was born out of time I spent digging into a biologically inspired algorithm I was using to measure co-activation for placement of experts and ranks onto chips. The default scheduling that vllm provides can end up causing laten…
-
Note: I am neurodivergent and lean heavily on AI to communicate clearly. Writing structured posts on my own ends up so messy nobody reads them.
-
Can I try a model with random weights in llama.cpp or kobold.cpp? (www.reddit.com) llama 27w
In theory, it should be possible to run any model with random weights. This will generate gibberish, but it will let you see how fast it can run on your particular hardware before downloading the weights.
-
Vulkan backend outperforms ROCm on Strix Halo (gfx1151) — llama.cpp benchmark (www.reddit.com) moellama 27w
Just ran some llama-bench comparisons between ROCm and Vulkan backends on my Strix Halo system. Vulkan came out ahead, which surprised me.
-
New Gemma chat template update by Google (huggingface.co via hn) gemmallama 27w
Libraries llama-cpp-python How to use unsloth/gemma-4-E4B-it-GGUF with llama-cpp-python: !pip install llama-cpp-python from llama_cpp import Llama llm = Llama.from_pretrained( repo_id="unsloth/gemma-4-E4B-it-GGUF", filename="gemma-4-E4B-it…
-
Hey everyone, I’ve been experimenting with running Qwen models locally on my setup: GPU: RTX 3090 (24GB VRAM) RAM: 64GB CPU: Ryzen 5700X OS: Windows 11 What I’m currently running Qwen 3.6 35B (UD Q4_K_M) llama-server.exe -m "C:\Users\Dino\…
-
1080 Ti in 2026 - 11GB is still (barely) enough to stay relevant (www.reddit.com) mistralllamaqwen 27w
I’m still daily driving a 1080 Ti. Not because I’m a masochist, I just haven't been able to justify a 4090/5090 upgrade yet.
-
As MTP prepares to land in llama.cpp, Models that support MTP (www.reddit.com) llama 27w
DeepSeekv3 OG DeepSeekv3.2/4 Qwen3.5 GLM4.5+ MiniMax2.5+ Step3.5Flash Mimo v2+ Until we get mtp weights, you need to download HF weights and convert to gguf. I think I'm going to try either qwen3.5-122b or glm4.5-air first.
-
qwen 3.6 27B looping problem (www.reddit.com) gemmallamaqwen 27w
Whenever I write here that I use gemma 31B I get answers that qwen 27B is better. I switched in the pi from gemma 31B Q5 to qwen 27B Q8 and generally I manage to code, document and run tests but somewhere after exceeding 100k context qwen…
-
How do you estimate total memory usage? (www.reddit.com) llama 27w
Qwen3.6 35B A3B UD IQ4_NL_XL. 512k context tokens for 4 parallel processing, key cache quantized to Q_8 and value cache quantized to Q_4.
-
FastDMS: 6.4X KV-cache compression running faster than vLLM BF16/FP8 (www.reddit.com) vllmllama 27w
Last year researchers affiliated with NVIDIA, University of Warsaw, and University of Edinburgh published Dynamic Memory Sparsification (DMS), a KV-cache sparsification technique using learned per-head token eviction, reporting up to 8x KV…
-
Best Llama Config for Turboquant_Plus? (Stats below) (www.reddit.com) moellamaqwen 27w
So I'm running the below and I've seen guys run this setup with TurboQuant_plus and get 35 tokens/second. I find the speeds I'm getting acceptable but if I could hit 30-35 I'd be soooooo happy.
-
LLM inference speed database or leaderboard? (www.reddit.com) llama 27w
A lot of the posts in this sub is about advice about which hardware to buy, what settings to use and what speed to expect. There are a lot of excellent replies spread all over the place, but alot of it is also just vague indications like ~…
-
Cursor's custom-OpenAI URL feature is what makes this work. Pointed it at a router I built.
-
Show HN: Bonsai 1.7B ternary model at 442T/s on M4 Max (agents2agents.ai via hn) llamaagentic 27w
We took a recently released Bonsai 1.7B ternary model from PrismML (https://github.com/PrismML-Eng/Bonsai-demo) and ran our agentic evolution search on it for 6 hours to optimize the Metal kernels. The search was fully autonomous.
-
half-deployed AI projects haunt my github (www.reddit.com) llama 28w
Got 47 repos that start with 'just playing with Claude' or 'testing Llama 4 on'. Every single one dead after three commits.
-
Plenty of CLI coding agents will talk to a local LLM, but the catch is the ecosystem. Skills, slash commands, MCP servers, plugins, hooks: all the interesting tooling has been built specifically for Claude Code, and parity on every other a…
-
Mistral Medium 3.5 128B and Qwen 3.5 122B A10B on 4x RTX 3080 20GB (www.reddit.com) mistralllamaqwen 28w
Mistral Medium 3.5 128B with 4x3080 20GB with layer split: CUDA_VISIBLE_DEVICES=0,1,2,3 ./build/bin/llama-bench --model /data/huggingface/Mistral-Medium-3.5-GGUF/Mistral-Medium-3.5-128B-IQ4_XS-00001-of-00003. gguf -ngl 99 -d 0,16384 -fa 1…
-
Testing PrismML Models (www.reddit.com) llama 28w
Testing PrismML Ternary Bosai I have been doing tests with PrismML Ternary Bosai. Tests on the Mac Mini M4 (with the MLX version) have been impressive (4K context): Mac MLX Bonsai 1.7B: ~135 t/s Mac MLX Bonsai 4B: ~67 t/s Mac MLX Bonsai 8B…
-
Questions regarding abliteration / censorship removal (www.reddit.com) llama 28w
Hello everyone. I just thought of something that seems so obvious but from what I’ve been able to find it doesn’t seem like anyone has done it or at least not openly disclosed it if they have.
-
I spent a while getting this dialed in and wrote up the full recipe. Short version: 35B MoE TQ3_4S fits in 12.4GB of weights KV cache at q8_0/q8_0 and 262K context only uses 2.7GB because MoE only has 10 attention layers out of 40 Total VR…
-
I have been using llama.cpp to run some models recently. For example, I've been running GLM-4.7-Flash with this command .\llama-server.exe -hf unsloth/GLM-4.7-Flash-GGUF:Q6_K_XL --alias "GLM-4.7-Flash" --host 127.0.0.1 --port 10000 --ctx-s…
-
Llama.ttf: a font file which is also a large language model and inference engine (fuglede.github.io via hn) llama 28w
llama.ttf llama.ttf is a font file which is also a large language model and an inference engine for that model. llama.ttf is a font file which is also a large language model and an inference engine for that model.
-
Been building this for a while and finally cleaned it up enough to share. voice-agents-from-scratch is a numbered, chapter-by-chapter repo that walks the full real-time pipeline: Microphone capture Whisper for STT Local GGUF LLM (via llama…
-
Show HN: Llmconfig – configfile and CLI for local LLM (github.com via hn) gemmallama 28w
llmconfig Local Large Model Config — manage local inference with llama.cpp, stable-diffusion.cpp, and whisper.cpp from a single YAML file and a single CLI. llmconfig up gemma # or just: llmc up gemma ✓ gemma is ready at http://127.0.0.1:80…
-
Open Weights Models Hall of Fame (www.reddit.com) mistralgemmallama+1 28w
I read a lot of "whengguf" type posts. I think we should sometimes stop and be grateful.
-
Hello everyone. Over the last couple months I have been assembling my local AI setup for personal use, and I thought to write a post here, firstly to collect some thoughts on the whole concept, and secondly to perhaps gather some feedback.
-
Show HN: Valkyr LM Inference with Realtime Guarantees (github.com via hn) vllmllamaopenai 28w
Valkyr is a fresh take on LM Inference runtimes. It's quite different from llama.cpp, vLLM, or ZINC for example.
-
LLM proxy that lets Claude Code talk to any model (www.reddit.com) gpt-5llamaopenai+3 28w
I built rosetta-llm — an open-source multi-format LLM proxy that acts as a drop-in Claude Code gateway. Works as a Claude Code LLM gateway — set `ANTHROPIC_BASE_URL` and all configured models appear in `/model` picker Translates between fo…
-
If you've been waiting to try local AI development, please try it (www.reddit.com) llamacursorclaude-code 28w
I have snobbishly long felt that the local models were not 'up to my standards' for local development, or otherwise able to compete with GHCP, Claude Code, Cursor etc. Boy was I wrong.
-
Anyone running HUANANZHI H12D-8D + BMC with 4x RTX 3090 for LLM inference? (www.reddit.com) vllmllama 28w
Hi everyone, I'm considering building a home LLM inference rig around: - HUANANZHI H12D-8D + BMC - AMD EPYC 7002/7003 - 4x RTX 3090 24GB - DDR4 ECC RDIMM, 8-channel - Linux + vLLM / SGLang / llama.cpp - Open frame, PCIe 4.0 x16 risers The…
-
Round 2: 2026-05-02 — llama.cpp b8198 → d05fe1d Rebuilt llama.cpp from b8198 (2026-03-04) to commit d05fe1d (2026-05-02), ~770 builds of progress. Same model, same hardware, same flags.
-
Requesting advice on local AI setup for academic use (www.reddit.com) vllmllama 28w
I'm about to do a clean install of Ubuntu 26.04 on a desktop that has a 5060ti 16gb and a 4060ti 16gb. Can you help me work out the best local AI setup for my use cases?
-
Ban phrases on llama.cpp with this script. (www.reddit.com) llama 28w
Check the README for setup instructions: https://github.com/BigStationW/llama-cpp-phrase-ban
-
What could they mean by "warmed steady-state"? (www.reddit.com) llama 28w
https://www.reddit.com/r/LocalLLaMA/comments/1t0vp3w/pflash_10x_prefill_speedup_over_llamacpp_at_128k/ Q4_K_M Qwen3.6-27B on a 24 GB 3090 decodes fast (~74 tok/s with DFlash spec decode), but prefill scales O(S²). On a 131K-token prompt, v…
-
Need advice on Qwen 3.6 27B INT4 quantization (www.reddit.com) vllmllamaqwen+1 28w
Hello everyone, I think Qwen 3.6 27B is good enough that it might take a while before we get a clearly better model at a similar size. I have a single headless RTX 3090 with a 300W power limit.
-
I wanted to share an open-source app that I built for running LLMs locally on my setup. My setup Hardware FEVM FAEX1 (128GB) RTX Pro 5000 Blackwell (48GB), connected over OCuLink Aoostar AG02 2x2TB internal m.2 drives on raid-0 using mdadm.
-
Poor GPU Club : Tried Bonsai-8B on CPU & CUDA (www.reddit.com) llama 28w
Got a chance to check this model today. 8GB VRAM(RTX 4060 Laptop GPU) & 32GB DDR5 RAM.
-
[Help] Running big dense models faster (www.reddit.com) mistralvllmllama+1 28w
I have been trying Mistral 3.5 on my 4x RTX 3090 rig with llama.cpp. Inference is slow (about 11 t/s) even without anything being offloaded to the CPU.
-
World AI Agents–35 AI Models (Claude, GPT, Llama)via One OpenAIcompatible API (world-ai-agents.com via hn) mistralllamaopenai 28w
Access Claude, Llama, Mistral, Nova and more through a single OpenAI-compatible API. Start for as little as €1.
-
OpenJet v0.4: a zero-config local coding agent for llama.cpp (www.reddit.com) llamaclaude-code 28w
Hello again. I just pushed a major update to OpenJet.
-
What's your tps on 3090 + Qwen 3.6 27B in real tasks? (www.reddit.com) llamaqwen 28w
I struggle to wrap my head around all this. My goal is local agent to solve low complexity tasks, in the same harness where I would use frontier models.
-
Hybrid on-device inference on Android: llama.cpp + LiteRT + NPU/GPU routing (www.reddit.com) llama 28w
Hi everyone, I’m the maintainer of Box — a fork of Google’s AI Edge Gallery that I’ve been extending into a fully offline AI assistant for Android. Full disclosure: I built this project.
-
Unsloth solved bug in Mistral Medium 3.5 implementation (www.reddit.com) mistralllama 28w
https://unsloth.ai/docs/models/mistral-3.5 "May 1, 2026 Update: We worked with Mistral to fix Mistral Medium 3.5 inference affecting some implementations, and released updated GGUFs with the fix (NOT related to Unsloth or our quants). The…
-
Qwen3.6-27B-NVFP4 - images (www.reddit.com) llama 28w
Model: Abiray-Qwen3.6-27B-NVFP4.gguf Specs: - Legion 7i Gen10 - NVIDIA GeForce RTX™ 5090 - Intel® Core™ Ultra 9 275HX × 24 - RAM 32.0 GiB llamacpp settings: ./build/bin/llama-server \ -m ~/.lmstudio/models/lmstudio-community/Qwen3.6-27B-GG…
-
Which other models will my system support? (www.reddit.com) llama 28w
This is my system: OS: Nobara Linux 43 Processor: Ryzen 9 5980HX RAM: 16 GB GPU: Radeon RX 6800M (12GB) I'm using llama.cpp and Qwen3.6-35B-A3B-UD-Q4_K_M is working okay in this system using vulkan. I'm getting a speed of ~17 t/s.
-
Using Valve's AMDGPU VRAM management to benefit local AI Inference rather than games? (pixelcluster.github.io via reddit) llama 28w
Any other AMDGPU users on Linux taken an interest at what Valves been doing for VRAM management for gaming? Seems to me that this might be just as useful for local AI inference as for gaming, especially for those of us wanting to do infere…
-
PFlash: 10x prefill speedup over llama.cpp at 128K on a RTX 3090 (www.reddit.com) llama 28w
Hey fellow Llamas, thank you for all the nice words and great feedback on the last post I made. We have something new we thought would be useful to share.
-
Need help optimizing qwen 3.6 on my 2x 5060ti 16gb (www.reddit.com) ollamallamaqwen 28w
Hi all, I tried to setup my pc to run llm, but got some issue: the first question of the chat is generally fine, but from the 3rd follow up question, the backend often be unresponsive and I have to manually restart the llama cpp server, or…
-
gemma-4-31B-it-DFlash has been released (www.reddit.com) gemmallama 28w
https://huggingface.co/z-lab/gemma-4-31B-it-DFlash I guess we'll have to wait until this PR is merged before we can test it. https://github.com/ggml-org/llama.cpp/pull/22105
-
Does Cline KanBan support local llm? (www.reddit.com) clinellamaopenai 28w
I installed Cline CLI and it was using my local LLM. But it seems like when I tried to use Cline KanBan it tries to use OPenAI directly instead of the llama.cpp OpenAI Compatible URL I entered.
-
## Got DFlash speculative decoding working on Qwen3.5-35B-A3B with an RTX 2080 SUPER 8GB I managed to get **DFlash speculative decoding** working in llama.cpp on a pretty VRAM-limited setup. This was tested with the DFlash PR: https://gith…
-
Ai Doomsday Toolbox v0.938 (www.reddit.com) ollamallama 28w
Hello! It’s me again, the developer of ADT.
-
Running Qwen 35BA3B on a 16GB M3 Macbook Air at 8.9TPS! (www.reddit.com) llamaqwen 28w
Preface: I actually write my posts myself, no slop in this post. I managed to get Qwen 3.5 35BA3B working on my 15" 16GB M3 MBA through mmap, and I must say that given the massive model compared to my ram, 9 TPS is not bad at all.
-
I'm testing running local LLMs on a gaming mini PC (AMD 7840HS, 32 GB RAM) paired with an eGPU (Radeon 9060XT with 16 GB VRAM). Since I'm not very familiar with using llama.cpp, I kept getting unsatisfactory results, but with the recent Ge…
-
"I" is not singular — 4 LLM agents with per-agent LoRA on a single RTX 3070 8GB (www.reddit.com) llama 28w
https://preview.redd.it/7yei65sbugyg1.png?width=1703&format=png&auto=webp&s=ad388c51dd10cb44b41a99876d28797e006fd138 Stanford's Generative Agents = one LLM cosplaying 25 personas. I wanted agents that actually become different people — dif…
-
Running llama.cpp on Snapdragon Hexagon NPU seems promising (www.reddit.com) gemmallama 28w
https://github.com/ggml-org/llama.cpp/blob/master/docs/backend/snapdragon/README.md I have an Oneplus 12 with Snapdragon 8 Gen 3. I followed the above README to cross-compile llama.cpp on Ubuntu and then copy to the Termux directory on the…
-
Hello folks What is best code editor for local LLM deployment (LM Studio, llama.cpp)? I wish to test my LM studio + Qwen 3.6 27B and Gemma 4 31B with a legit local code editor.
-
I pitted different LLMs against each other in Pokemon Showdown (www.reddit.com) llamagemini 28w
I wanted to see if LLMs could reason through complex game states, so I built a system where they can play Pokémon Showdown battles autonomously. They get the battle state every turn and use tool calls to attack or switch.
-
I've built a system where models like Llama 3, Qwen, and Gemma play Pokémon Showdown battles autonomously. Instead of simple prompt-response, they analyze the full battle state every turn (type matchups, HP, weather, field conditions, reve…
-
I wanted to see how much of my coding-agent workflow I could move local instead of paying for hosted tools forever. There was another push: Anthropic's own April 23 postmortem confirmed product-layer regressions through March/April.
-
Benchmarking Local LLM/Harness Combinations (neuralnoise.com via hn) aiderllamaqwen+1 28w
I’ve been running a small benchmark, harness-bench , that pairs local LLMs (served via llama.cpp ’s llama-server ) with agent harnesses (Aider, Claude Code, OpenCode, Pi, Qwen CLI) on 16 software-engineering tasks across Python, PyTorch, J…
-
Qwen3.6 27B seems struggling at 90k on 128k ctx windows (www.reddit.com) llama 28w
I have RX 7900 XTX, running Qwen3.6 27B Q4_K_XL. got 400ish pp and 30s tps.
-
Previously a model could only be present in a single group. Now you can create whatever groups you want: one for big models that should run on their own, a group for STT + bigger model, a group for RAG usages, etc.
-
I built a 5M model to see if it outperforms my 350M model... (www.reddit.com) llama 28w
Hi r/LocalLLaMA ! I built a 5M Llama model with HF Transformers on 2x T4 in Kaggle to see, if it is able to be as good as my previous Apex 350M model (https://huggingface.co/LH-Tech-AI/Apex-1.6-Instruct-350M).
-
Can't replicate Reddit numbers with Qwen 27B on a 3090TI. (www.reddit.com) llamaqwensonnet 28w
I feel like i'm going insane. I see people here posting 30 - 100+ tok/s (100+ being with speculative decoding) on a 3090 with Qwen 3.6 27B.
-
thinking of gemma 4 26B vs 31B (www.reddit.com) vllmgemmamoe+2 28w
I see a big difference in agentic coding between gemma-4-31B-it-Q5_K_M and gemma-4-26B-A4B-it-UD-Q8_K_XL. The 26B model is much faster because of A4B and generally works well, but there is a big difference in thinking.
-
I have (free) access to a SLURM cluster with 8x NVIDIA A100 80GB GPUs (=640 GB VRAM) on a single task, and I want to run an open-weight model locally with llama.cpp for data generation, not coding. My use case is generating teacher data fo…
-
I was experimenting yesterday with running oversized models with smaller context size, hoping that leaving them overnight could compensate for the slow token generation and periodic pauses for compaction or task chunking. Summary: For rese…
-
Comparing SVG Generation for the top open models (codeinput.com via reddit) minimaxglmgemma+2 28w
Some of the larger models (like Llama) weren't available on OpenRouter, so I had to work with what was there. Best small model: Gemma 4 26B For its size, I think it had the best output.
-
best approach for Strix Halo distributed inference in llama.cpp? (www.reddit.com) llama 28w
I was curious to understand what people are doing for this use case to get the best trade-off of convenience and performance. Private backhaul on the 10GbE?
-
Is Mistral-3.5-Medium-128B broken in Llama CPP? (www.reddit.com) swe-benchmistralvllm+1 28w
Trying some if Bartowski's Q4 quants. Using Vulkan with the latest main branch as of a few hours ago.
-
Gemma 4 architecture support for QVAC-Fabric (Tether's llama.cpp fork) (github.com via hn) gemmallama 28w
QVAC-Fabric Gemma 4 Architecture Patch Adds full Gemma 4 (gemma4) architecture support to QVAC-Fabric, Tether's llama.cpp fork. Base: QVAC-Fabric temp-upstream branch Target: All Gemma 4 variants (E2B, E4B, etc.
-
Qwen3.6-27B-UD-Q6_K_XL.gguf sometimes gets stuck in a loop (www.reddit.com) llama 28w
Hi all I'm running Qwen3.6-27B-UD-Q6_K_XL.gguf using llama swap and llama-server with these parameters (actually stolen for some posts on this subreddit.) llama-server \ -m /models/Qwen3.6-27B/Qwen3.6-27B-UD-Q6_K_XL.gguf \ --mmproj /models…
-
Hey everyone, I’ve been building a local-first desktop PDF reader that can read technical books aloud and keep the spoken text highlighted while reading. The original motivation was pretty practical: I read a lot of programming and technic…
-
Mistral Medium Looping (www.reddit.com) mistralllama 28w
Hey, I don't know if this is a llama.cpp issue or an Unsloth thing, but for whatever reason Mistral Medium 128B at Q4_K_XL seems to go in loops after like 500–1000 tokens. Anyone else seeing this?
-
With GitHub pausing Copilot Pro+ signups and Claude Code potentially leaving the Pro tier, I started building the AI coding tool I actually wanted to use. One that doesn't depend on cloud access staying cheap and available.
-
Don't forget about dem free gains! (www.reddit.com) llama 28w
Looks like progress has been made on -sm tensor. Couldn't even run llama-bench a few weeks ago: 1 card - 1580/44: $ llama-bench -m Qwen3.6-27B-UD-Q4_K_XL.gguf -fa 1 ggml_cuda_init: found 1 CUDA devices (Total VRAM: 24112 MiB): Device 0: NV…
-
I've been running local LLMs since Qwen 3.5 dropped and I was really impressed by what we could run on consumer hardware. Fast forward another two months and we have gotten a handful more gems such as Gemma 4 and Qwen 3.6, so I wanted to p…
-
PS5’s can now be hacked to run Linux - perhaps some potential for local inference? (www.tomshardware.com via reddit) llama 28w
I look forward to the Local LLM community getting llama.cpp to run on these. Could be a good value.
-
Sorry if it's not the best place to ask this, of the models in the image, which is the best for (problem solving)/Coding and the best one for studying (ask LLM concepts) ? My PC build is RX 9060 XT 16GB + I3 12100F + 16 GB DDR4 + llama.cpp with Vulkan backend + Linux Mint. (www.reddit.com) moellamaqwen 28w
I gave some math problems to Qwen 3.5 27B and Qwen 3.6 27B and they got all of them right, pretty smart models I would say, but very slow and electricity consuming, they took like 5 mins with my GPU at 120 W to solve a problem. The MoE mod…
-
llama.cpp benchmark native vs. non native NVFP4 on Blackwell - summary (www.reddit.com) llama 28w
I tested two llama.cpp builds on the same Qwen3.6-27B-NVFP4 model. llama-bench reports the model label as qwen35 27B NVFP4, but the actual tested model is Qwen3.6-27B-NVFP4.
- llama.cpp - NVFP4 native support on Blackwell from now - b8967 (www.reddit.com)
-
Curious what stacks people are actually using right now, and where you're hitting walls. Some things I've been observing while testing combos: - Deepgram Nova-3 still the best STT for English, Cartesia is closing the gap on streaming - Ele…
-
Llama.cpp MIPS R8000 Kernel Running on an SGI Power Challenge from 1995 (twitter.com via hn) gemmallama 28w
Whew! Big work today getting optimized llama.cpp MIPS R8000 kernel running on the SGI Power Challenge deskside from 1995 with Gemma 3 270M.
-
Help with MI50 and llama.cpp/ROCm 7.2 (www.reddit.com) llama 28w
I have an MI50 that I use with llama.cpp/Vulkan, however some models run quite slowly, so I'd like to try the ROCm backend, but no matter what I try it doesn't work. Downloading the missing files from ArchLinux package doesn't work.
-
I stumbled on a Gemma 4 chat template bug for tools and fixed it (www.reddit.com) gemmagpt-5llama+1 28w
TLDR: tool parameters using the common JSON Schema pattern `anyOf: [$ref, null]` are rendered into the prompt as empty `type` fields. This strips the useful schema information before the model sees it.
-
MiMo-V2.5-GGUF (preview available) (huggingface.co via reddit) moellama 28w
Hi, AesSedai here - I've put up a PR to support the text-to-text inference of MiMo V2.5 with llama.cpp (and should also support Pro, will work on those quants after finishing V2.5): https://github.com/ggml-org/llama.cpp/pull/22493 I've als…
-
llama.cpp's Preliminary SM120 Native NVFP4 MMQ Is Merged (www.reddit.com) gemmallama 28w
https://github.com/ggml-org/llama.cpp/pull/22196 And somehow we already got some GGUFs for it! https://huggingface.co/CISCai/gemma-4-31B-it-NVFP4-turbo-GGUF https://huggingface.co/stevelikesrhino/gemma-4-31B-it-nvfp4-GGUF (the below one is…
-
Workstation upgrade for 5 concurrent users (Qwen 3.6 27B) (www.reddit.com) llamaqwen 28w
Hello, I would like a suggestion from those who are already actively involved in this world. Basically, I own this workstation: Ryzen 9 5900X 32GB di RAM DDR4 RTX 5060Ti PCCOOLER CPS YS1000 1000W Currently, I can quite easily code with Qwe…
-
Qwen3.6-27B-GGUF:UD-Q8_K_XL and llama.cpp issue (DGX SPARK) (www.reddit.com) llama 28w
Hey all, im having a crisis that i just cant figure... i used Qwen3.6-27B-GGUF:UD-Q8_K_XL ever since it came out (on a DGX SPARK) and it worked like magic with decent performance (~50 t/s) , im updating SPARK and llama.cpp on a daily basis…
-
Anyone have experience with both? Luce is llama.cpp with custom dlflash and noonghunnas project is vllm with patches.
-
Took TheTom's TurboQuant Metal fork of llama.cpp (github.com/TheTom/llama-cpp-turboquant, the feature/turboquant-kv-cache branch) and ran a depth sweep on Qwen 3.6-35B-A3B Q8. TheTom had already published M5 Max numbers up to 32K.
-
Researchers Alec Radford (GPT, CLIP, Whisper), Nick Levine, and David Duvenaud just released talkie: a 13 billion parameter language model trained exclusively on text published before 1931. No internet.
-
I’ve always liked how if I ask ChatGPT to make or edit an image, it just does it. Local AI should be this convenient!
-
Is long re-processing of output as input a common "feature" or not? (www.reddit.com) gemmallamaqwen 28w
I now use (mostly) Gemma 4 and Qwen 3.5 models *. And seems that all of them, after context grows a bit, after providing long output for me and getting a short prompt in response, are starting to process many new tokens as input and I have…
-
Qwen3.6-27B IQ4_XS FULL VRAM with 110k context (www.reddit.com) llama 28w
Qwen3.6-27B IQ4_XS Bloat: Reverting llama.cpp commit saves 16GB VRAM (14.7GB vs 15.1GB) + KVCache Tests With the release of Qwen3.6-27B, I noticed that compared to the excellent IQ4_XS quantization (14.7GB) by mradermacher for the 3.5 vers…
-
TLDR: I've been running gemma4 e2b extensively on iOS with llama.cpp and found some interesting quirks and info you guys may like! These are specifics for the iPhone and what I've found worked across 20+ devices.
-
Qwen 3.6 27B BF16 vs Q4_K_M vs Q8_0 GGUF evaluation (www.reddit.com) humanevalfunction-callingllama+1 28w
Evaluated Qwen 3.6 27B across BF16, Q4_K_M, and Q8_0 GGUF quant variants with llama-cpp-python using Neo AI Engineer. Benchmarks used: HumanEval: code generation HellaSwag: commonsense reasoning BFCL: function calling Total samples: HumanE…
-
llama.cpp - tool calling issues on Windows only (www.reddit.com) llama 28w
I have a dedicated linux box I run all my stuff on. I occasionally see the 'zomg 35b can't call tools?!' posts here and chuckle to myself in a *zero issues here* way.
-
Did some quick tests after building llama.cpp with ROCm 6.4.2 and latest Vulkan for my 6900 XT gemma4 E2B Q4_K ubatch ROCm pp512 Vulkan pp512 ROCm tg128 Vulkan tg128 32 1536.60 1423.49 151.92 174.59 64 1590.65 1930.60 151.41 173.76 128 265…
-
It looks as if OpenCode introduces an artificial delay in agentic coding. Have you noticed similar issues?
-
M4 Mac Mini, 16GB unified, basic spec. For a few weeks I had Qwen 3.5 35B-A3B UD-IQ3_XXS (12GB on disk) running under llama.cpp with --mmap and --flash-attn.
-
[7900XT] Qwen3.6 27B for OpenCode (www.reddit.com) moellama 28w
I'm just looking for some advice on optimally setting up Qwen3.6 27B for OpenCode. The VRAM is a little bit scarce, but I ended up with this so far: llama-server --model models/Qwen3.6-27B-IQ4_XS.gguf \ --port 8080 \ --host 127.0.0.1 \ --t…
-
The gemma 4 E4B and E2B models have built-in multimodal capabilities. However, as far as I am aware, llama.cpp does not have proper support for vision and audio inputs (specially audio) for these models as of now.
-
GMKtec EVO-X2 70B expectation (www.reddit.com) llamagemini 28w
I would like to use a 70B model on a GMKtec EVO-X2 AI Mini PC 128GB. Selected this one: Llama-3.3-70B-Instruct-Q4_K_M.gguf Ubuntu 24.4.4 LTS and compiled llama.cpp server for the gfx1151.
-
Last llama.cpp update broke web search tool calling with Qwen 3.6 27b. (www.reddit.com) llamaqwen 28w
At least in open-webui. Nothing has changed except for the backend update.
-
Please help me build some clarity. I want to participate in local LLMs ecosystem more.
-
I have a 4 x R9700 system on Threadripper pro, but I have never been happy with the performance of my GPUs in vLLM. I have started benchmarking any new model I try out with llama-benchy so that I can get a better idea of how models of diff…
-
GBNF grammar tweak for faster Qwen3.6 35B-A3B and Qwen3.6 27B (www.reddit.com) llama 28w
Hi folks, Enjoy an optimised Qwen3.6 35B-A3B and Qwen3.6 27B for coding and general purpose - it's able to solve puzzles correctly more often too. The initial intent was to optimise the 35B-A3B reasoning traces since it's the most efficien…
-
How to run a local coding agent with Gemma 4 and Pi | Patrick Loeber (patloeber.com via reddit) gemmallama 28w
Tutorial from the Google guy, I use very similar setup (llama.cpp instead of lmstudio)
-
Question regarding 4 t/s Qwen 3.6 performance (www.reddit.com) llamaqwen 28w
I am getting 4 t/s with Qwen3.6-27B-Q4_K_M which seems much slower than I'd expect. I am running LM Studio on Ubuntu 22.04 with the following specs: Dell Precision 5690 AI-ready workstation NVIDIA RTX 5000 Ada Generation GPU with 16GB VRAM…
-
Why are there so few small local creative writing models from the Chinese? (www.reddit.com) mistralgemmallama+1 29w
At this moment, the models such as Qwen 3.6 35b/27b crush the competition, yet I can't help, but notice this pattern. While the local RP scene is abundant with the Western model tunes: LLaMA, Mistral (all sizes), Nemo and more recently Gem…
-
Repo: statisticalplumber/kanban at pi-agent-integration Hi Guys, To test Qwen 3.6’s potential, I also wanted the Cline Kanban project to have an open-source agent to work with. The last time I tested Cline Kanban, it didn’t support agents…
-
Ubuntu 26.04 vs 24.04 speed improvements for inference? (www.reddit.com) vllmllama 29w
I'm curious if any brave soul has upgraded their computer (especially if it's Strix Halo) from Ubuntu 24.04 -> 26.04 and seen a significant performance improvement for inference with VLLM, llama-server, and/or LM Studio.
-
It all started yesterday with this post by u/antirez https://www.reddit.com/r/LocalLLaMA/comments/1sw3stb/llamacpp_deepseek_v4_flash_experimental_inference/ I was intrigued by the first Deepseek V4 Flash GGUF in a small size that can fit o…
-
I think is nice to contain the MCP into a Qemu enviroment where the LLM can do whatever ... here is doing GDB on a LVGL program.
-
To 16GB VRAM users, plug in your old GPU (www.reddit.com) llama 29w
For those who want to run latest dense ~30b models and only have 16GB VRAM, if you have a old card with 6GB VRAM or more, plug it in. It matters that everything fits on the VRAM, even on 2 cards.
-
Brief Ngram-Mod Test Results - R9700/Qwen3.6 27B (www.reddit.com) llama 29w
Decided to try out the new --spec-type ngram-mod feature in llama.cpp using Qwen3.6 27B during an OpenCode bug chasing session. TLDR: Performance is variable, but so far it seems to provide a nice speed increase for working on the same cod…
-
VRAM.cpp: Running llama-fit-params directly in your browser (www.reddit.com) llama 29w
Lots of people are always asking on this subreddit if their system can run a certain model. A lot of the "VRAM calculators" that I've found only provide either very rough estimates or are severely limited in the number of models they can e…
-
I'm a daily llama-cpp user and was hoping to try MTP on vLLM. Unfortunately, pipeline parallelism + MTP does not seem to work with this model in vLLM.
-
Intel B70: LLama.ccp SYCL vs LLama.cpp OpenVino vs LLM-Scaler (www.reddit.com) vllmllama 29w
In case anyone is interested, I decided to test out LLama.cpp's new OpenVino backend to see how it compares on Intel GPUs. At first glance, it stomps all over the previous best-case, SYCL, but lags behind LLM-Scaler (Intel's VLLM fork), li…
-
TL:DR - Remembered FPGA PCI boards being a big thing from my crypto days. Wondered if AMD Alveo V80 FPGA card could be used to approximate the performance of a Taalas HC1 (LLM-on-a-chip).
-
locally uncensored is a desktop app that combines four things most people run separately: chat, a coding agent, image generation, and video generation. all local, all on your hardware, no docker, no cloud account needed.
-
Hey all: I am trying to set up claude code to work with llama.cpp, I am using the Qwen3.6-35B-A3B. I usually use claude code + ZLM subscription i got lucky with $30 yearly - the set up is very simple with their automated script, but for th…
-
mesa PR with 37-130% llama.cpp pp perf gain for vulkan on Linux on Intel Xe2 (gitlab.freedesktop.org via reddit) llama 29w
Making sure you're not a bot! Loading...
-
Is there a way to mitigate performance as context grows? (www.reddit.com) llama 29w
In my local LLM setup I get from 30 to 80 t/s generation at the beginning, but it drops quite a lot as context grows. I use llama.cpp/Vulkan with an MI50 and a V100, is there some command line flags that can improve this issue?
-
How do you actually use Qwen3 72B Instruct locally? (www.reddit.com) vllmollamallama 29w
I just got Qwen3 72B Instruct running on a high RAM setup and I’m kinda confused about the proper way to use it. What’s the correct workflow for running it smoothly (like best quant, tools, or runtime)?
-
VSCode and agent integration (www.reddit.com) copilotllamaqwen 29w
I've been using VSCode with Github Copilot for a bit (free tier) and looking to try running locally due to running in to all of the limits with GHCP. I'd like to have as close of an experience as possible with both code autocomplete and ch…
-
When Can LLMs Learn to Reason with Weak Supervision? (salmanrahman.net via hn) llamaqwen 29w
We study when RLVR generalizes under three weak supervision settings (scarce data with as few as 8 examples, noisy reward labels, and proxy rewards such as majority vote and self-certainty) across multiple models from the Qwen and Llama fa…
-
(Gemma/Qwen + Codex) - Bridging /chat/completions → /responses in llama-swap (www.reddit.com) gemmallamaqwen+2 29w
I’ve been tinkering with a small side project (just for fun) where I’m trying to extend llama-swap with a bridge from /chat/completions to the newer /responses API so I can run the latest Gemma and Qwen models together with Codex-style too…
-
llama.cpp DeepSeek v4 Flash experimental inference (www.reddit.com) llamadeepseekqwen 29w
Hi, here you can find experimental llama.cpp support for DeepSeek v4, and here there is the GGUF you can use to run the inference with "just" (lol) 128GB of RAM. The model, even quantized at 2 bit, looks very solid in my limited testing, a…
-
As a life-long Windows user (don't hate me, I was exposed to it at a young age) I was wondering how much (if any) performance I'm leaving on the table. So I did the sensible thing and run some benchmarks.
-
Using PaddleOCR-VL-1.5 with llama-server for book OCR (www.reddit.com) llama 29w
I've been running PaddleOCR-VL-1.5 via llama.cpp's server for OCR on book pages. It handles complex layouts, tables, and mixed text/figure pages surprisingly well.
-
Will llama.cpp multislot improve speed? (www.reddit.com) vllmllama 29w
I've heard mostly bad opinions about multiple slots with llama.cpp (--parallel > 1). I guess comparing to vLLM it might be worse at this, but I recently tried vLLM on 4 slots and it indeed improved the overall speed significantly (150-170t…
-
Experts-Volunteers needed for Vulkan on ik_llama.cpp (www.reddit.com) llama 29w
ik_llama.cpp is great for both CPU & CUDA. Need legends to make Vulkan better as well.
-
your daily driver stack, what's it look like? and why? (www.reddit.com) vllmllamacowork+2 29w
What it says in the title, I'm interested in hearing what you all have landed on as a workable / useful stack for you. Mine looks like this: back end inference servers - llama.cpp, vLLM | V hermes-agent - cron jobs + OpenAI compatible endp…
-
Impact of mixing architecture (www.reddit.com) llama 29w
For context As planned after my previous post, I now have a decent amount of VRAM to work with: 2x RTX 3090 maybe 2 more coming soon, if needed 1x RTX 4060 8x RX 6600 XT 1x RX 6700 XT 1x RX 9060 XT (12 to 20 3060 more coming soon + 2 3090…
-
Llama Server with Cline Settings (www.reddit.com) clinellama 29w
Hi everyone, just wondering if anyone has setup llama server to work with Cline and whether you can use image/browser use. I just gave it a whirl and had to disable image support.
-
Quant Qwen3.6-27B on 16GB VRAM with 100k context length (www.reddit.com) llama 29w
https://preview.redd.it/tblmrwxkbexg1.png?width=1193&format=png&auto=webp&s=6dea1e6684e75e22852d57c0c72e9171deb56ae2 I have experimented how to run Qwen3.6-27B on my laptop with an A5000 16GB GPU. I have created an own IQ4_XS GGUF "qwen3.6…
-
RTX 3090 + 27B model performance issues (llama.cpp) what am I doing wrong (www.reddit.com) llamaagentic 29w
Hey folks — looking for some advice on improving my local LLM setup (and also exploring agentic coding workflows). Current setup: GPU: RTX 3090 (24GB VRAM) RAM: 64GB Using llama.cpp with a Qwen3.6 27B Q6 model (GGUF) Running through OpenCo…
-
FP4 inference in llama.cpp (NVFP4) and ik_llama.cpp (MXFP4) landed - Finally (www.reddit.com) llama 29w
Both llama.cpp and ik_llama.cpp now have FP4 support — but with different flavors worth knowing about. llama.cpp recently merged NVFP4 (Nvidia's block-scaled FP4, `GGML_TYPE_NVFP4 = 40`), with CUDA kernels landing in `mmq.cuh`, `mmvq.cu`,…
-
What would you say is the minimum amount of tokens per second you would tolerate for your local agent workflows? I have been trying pi.dev connected to a llama.cpp instance running Qwen3.6-27B-Q6_K_L with 200K context running on an RTX A60…
-
How are you running Qwen 3.6 27B on windows? (www.reddit.com) llamaqwengemini 29w
I've been trying to fix performance with llama-server and seem to be hitting a wall. Using Q4_K_M by unsloth and IQ4_K_M by DavidAU, when asking a question with no context, 39 t/s.
-
Llama 4: A Deep Dive into Liquid Transformers 2.0 and Sovereign AI (en.landingfymax.com.br via hn) llama 29w
The tech world came to a standstill this week in April 2026 with Mark Zuckerberg's official announcement: Llama 4 is here. While Meta's previous models had already democratized access to Artificial Intelligence, the fourth generation of th…
-
Local LLaMA server GPU upgrade advice (www.reddit.com) llama 29w
TLDR : Should an RTX 3090 + T4 be faster than a P40 + T4 for OpenCode with Qwen3.6 35B A3B ? --- Hi, Nowadays, I have an architecture running : A Tesla P40 w/ 24GB VRAM A Tesla T4 w/ 16GB VRAM I mainly use this setup to run models like GPT…
-
I've been using Qwen3.6-27B-Q5_K_M with turbo3 KV cache since it's been released, and I haven't had any issues at all (no loops, no memory loss, etc.). However, I'm also aware that K cache compression is not really recommended in most case…
-
So maybe this is a no-brainer to many experienced local LLM users but it was not obvious for me. I am running a 3070 8gb + 64gb DDR4.
-
Memory upgrade, is it worth it? (www.reddit.com) moellama 29w
Hi, I need your opinion on a system upgrade, 🤔 I currently have the following AI server used for various tinkering, learning, development etc. System AMD Ryzen 7 7700 (8C16T Zen4) Corsair Vengeance RGB DDR5 5600MHz 32GB MSI B650 Gaming Plu…
-
Qwen 3.6 27B llama.cpp | Multi-GPU pp t/s help (www.reddit.com) llamaqwen 29w
The new dense model is great, but I’m trying to figure out how to increase PP and Token generation speed. I’m running Q8 quants across 3 7900xtx GPUs and I’m consistently only getting 18-20 t/s generation speed and ~650 t/s prompt processi…
-
A few days ago, I was trying to improve token generation speed on my RTX 4070 Super 12GB while running Qwen3.6 35B A3B UD-IQ3_XXS (Unsloth) with llama.cpp, but to no avail. At that time, I had my monitor plugged in my 4070 and didn't even…
-
Show HN: Doxa – Open-source emergent simulator for geopolitical scenarios (github.com via hn) llamagemini 29w
Hi! We, Vincenzo and Riccardo, built Doxa as an agnostic engine for emergent simulations with agents for constrainted scenarios (like geopolitical, economics, ...) and work well with LLMs like Qwen2.5:7B, Llama but also cloud models such a…
-
I'm building a small text-based game where the gameplay loop is "talk an NPC into revealing a secret." It's basically a 20+ turn roleplay stress test: the model needs to stay in character, remember what the player said earlier, and refuse…
-
Qwen3.6 35B-A3B is quite useful on 780m iGPU (llama.cpp,vulkan) (www.reddit.com) moellamaqwen 29w
I have ThinkPad T14 Gen 5 (8840U, Radeon 780M, 64GB DDR5 5600 MT/s ). Tried out the recent Qwen MoE release, and pp/tg speed is good (on vulkan) (250+pp, 20 tg): ~/dev/llama.cpp master* ❯ ./build-vulkan/bin/llama-bench \ -hf AesSedai/Qwen3…
-
Ollama swap to llamacpp/llama server (www.reddit.com) ollamamoellama 29w
So I'm a newb in certain aspects but not in others, I'm currently running an AI stack on my unraid server: CPU: AMD Threadripper 3960X (24c/48t) Motherboard: Gigabyte TRX40 AORUS PRO WIFI RAM: 256GB DDR4-3200 G.Skill Trident Z GPU: Nvidia…
-
Qwen 3.6 27b IQ4_XS - 22 tp/s on RTX 5060TI 16b, 24k ctx (www.reddit.com) llamaqwen 29w
Maybe it be helpful for someone: llama-server -m '/Qwen3.6-27B/Qwen3.6-27B-IQ4_XS.gguf' -ngl 999 -ctk q4_0 -ctv q4_0 -b 128 -ub 128 -c 24000 Cant run this model with higher kv quants on >8192ctx size. -ub & -b setted for 256 allowed me for…
-
Is there any quick way to estimate best parameters for llama.cpp? (www.reddit.com) llama 29w
I usually just throw models into LM Studio but I decided to finally compile llama.cpp on my hardware to get some extra speed and to hopefully replace my increasingly unreliable cloud subscription. I have a RTX 4080 and Ryzen 5 7600 with 32…
-
Severe instability and looping issues with local LLMs (Qwen, Zen4, llama.cpp) (www.reddit.com) llamaqwen 29w
I tried working on a local LLM project today and honestly ended up pretty frustrated. I tested several approaches, but none of them worked reliably.
-
llama-server: Save/restore works for tokens, but KV cache still not resumed? (www.reddit.com) llama 29w
Somehow I cannot get KV resume for my Qwen3.5 model with lama-server: Save/restore works for tokens, but KV cache is never reused — is this expected? How to enable real resume?
-
Speed penalty with Q8 KV quantization (www.reddit.com) llamaqwen 29w
I knew there would be a speed penalty when switching the KV cache quantization from F16 to Q8, but I never expected it to be this significant at longer context sizes. I ran a test with Qwen 3.5 122B on my MacBook M2 Max using llama.cpp.
-
What speed is everyone getting on Qwen3.6 27b? (www.reddit.com) llama 29w
I'm getting ~13 tps on Q8_0, with a context window of 128000, K Q8_0, V Q8_0 this is on 3x GPUS (1x2060super 8gb, 2x5060ti 16gb), via llamacpp unsure if this is slow or to be expected? */llama-server --port 8080 --model */llama.cpp/Qwen3.6…
-
Got a RTX a5000 24gb, what models could I use? (www.reddit.com) llama 29w
I just got a used RTX a5000 24gb to use for local models, I mainly use AI to code, but I prefer to spend some money now instead of $200 per month on claude to use 50% of it in a single prompt. My current specs are: Ryzen 7 9800x3d 64Gb DDR…
-
Qwen3.6 One Shot Tetris Game (www.reddit.com) moellama 29w
I am blown away by what this model can generate locally. I asked for a flashy Tetris game with particle effect and boy did it deliver!
-
It is worth an RTX 3090 for 850 if you can a radeon 7900 XTX for 495? (www.reddit.com) gemmallamaqwen 29w
Both amounts are in euro. The AMD is actually 599 but it's sold by a shop, so I can get a VAT return as a company, while for the nvidia I'd have to go to the second hand market and I can't get VAT back, so at the end it's like a 495 vs 850…
-
Qwen3 27B FP8 + TurboQuant on RTX 5090 - anyone tried? (www.reddit.com) llamaqwen 29w
Do I understand correctly, based on this comment, that I can potentially fit Qwen 3.6 27B FP8 precision model and have around 256K context available and fit it fully in my RTX 5090 VRAM? Of course with the help of TurboQuant compression, a…
-
Hi LocalLLaMA, I created a post a few weeks ago, but this time this project has become more reliable and easier to use. This is a manga translator that can also be used to translate any image.
-
Llama.cpp parameters for Qwen 3.6 with RTX 3090 (www.reddit.com) llamaqwenagentic 29w
Hi, I'm trying to run Qwen 3.6-35B on my RTX 3090 (24 GB of VRAM) but I'm not sure about 2 thing: - Which variant of the model to use ? (Q4_K_S, Q3_K_XL, other ?
-
Sıfırdan Eğitilmiş 258M Parametre Türkçe LLM: Marul V7 (www.reddit.com) llama 29w
Selam, Bir süredir üzerinde çalıştığım bir projeyi paylaşmak istiyorum. Sıfırdan geliştirdiğim bir Türkçe dil modeli var: Marul V7 Model tamamen bağımsız şekilde eğitildi.
-
Hey everyone, Ever since the day Google announced TurboQuant, I've been following the news about its extreme compression capabilities without noticeable quality degradation. I see it mentioned constantly on this sub, but despite all the di…
-
R9700 Qwen3.6 Benchmarks? (www.reddit.com) llama 29w
Can someone who owns a R9700 (single GPU enough) to add a llama-bench output with Qwen3.6-35B-A3B Q5_K_P here in the thread? Other benchmarks are also welcome :) I just want to see the t/s and compare it with my local solution, because I m…
-
Memory-augmented Large Language Models (LLMs) are essential for developing capable, long-term AI agents. Recently, applying Reinforcement Learning (RL) to optimize memory operations, such as extraction, updating, and retrieval, has emerged…
-
Need help with llama.cpp Qwen3.6 configuration on a single 3090 w/ 48GB RAM (www.reddit.com) llama 29w
Hey there, I have been testing models locally, but this is the first model that got me interested in understanding llama.cpp in more detail. I have noticeable stuttering when I run the model as it fills the VRAM completely, and I am sure I…
-
Qwen 3.6 35B-A3B takes a long time at image processing. Is it happening only to me? (www.reddit.com) moellamaqwen+1 29w
9900x, RTX 4080, 96GB RAM. Llama-cpp, Windows.
-
Best open-source tools for prompt injection defense in 2026 (www.reddit.com) prompt-injectionllamarag+2 29w
Over the time we have been testing different approaches to secure LLM apps against prompt injection, especially indirect injection through RAG, PDFs, as well as tool outputs, and MCP integrations. Most tools seem to fall into 2 categories:…
-
Is there a way to load huge MoE models on a computer with way too little RAM for the model's size, inferencing from the SSD, on LM Studio using the mmap/GPU/CPU layer customization thing (similar to how you can on llama.cpp)? I can't get it to load without memory spiking and going into swap. (www.reddit.com) moellamadeepseek 29w
-
how to maximize my tos on a 6Gb Nvidia rtx 4050 and 16Gb ram (www.reddit.com) copilotllama 29w
-
Llama.cpp's auto fit works much better than I expected (www.reddit.com) llama 29w
-
I was building a dedicated-vision-model feature for an open-source browser agent and wanted to figure out which local model to actually recommend. Wrote a small probe that sends the same image + same system prompt + same params (temperatur…
-
Hi everyone, I’m running into a consistent crash when trying to use image input with Llama 4 Maverick in llama.cpp. Text works perfectly, but as soon as I send an image, the server crashes.
-
235M param LLM from scratch on a single RTX 5080 (www.reddit.com) llama 29w
-
-
Verbatim AI – on-device transcription (Whisper) + summaries (Llama 3.2) (apps.apple.com via hn) llama 29w
-
RTX PRO 6000 Blackwell Max-Q bad performance (www.reddit.com) llama 29w
-
Ollama alternative with dynamic model loading (www.reddit.com) ollamallama 29w
-
eGPU vs system RAM (www.reddit.com) minimaxllama 29w
-
-
-
-
English version of Nexus Ark? (www.reddit.com) llama 29w
-
Dual dgx spark (Asus GX10) MiniMax M2.7 results (www.reddit.com) minimaxllama 29w
-
-
Open WebUI Desktop Released! (github.com via reddit) llama 29w
-
Need help for a calling based agentic ai project (www.reddit.com) llamaagentic 29w
-
model for frigate, a380 (www.reddit.com) llama 29w
-
PSA re Qwen 3.6 35B A3B q4 + agents (www.reddit.com) moellamadeepseek+1 29w
-
-
Recommended parameters for Qwen 3.6 35B A3B on a 8GB VRAM card and 24GB RAM? (www.reddit.com) moellamaqwen 30w
-
Qwen3.6-35B-A3B running on a Mac mini M4 16GB (www.reddit.com) llama 30w
-
Help on jiberish output on Qwen3.6-35B-A3B-GGUF::UD-IQ3_S (www.reddit.com) llama 30w
-
llama-bench results with SYCL backend - Intel Arc B70 (on a pcie 3.0 motherboard) (www.reddit.com) llama 30w
-
Qwen3.6 agent + Cisco switch: local NetOps AI actually works! (www.reddit.com) clinellamaqwen+1 30w
-
Dual GPU setup (yes, no)? (www.reddit.com) llama 30w
-
-
-
llama.cpp speculative checkpointing was merged (www.reddit.com) llama 30w
-
-
what is the state of using rotoquant at the moment? (www.reddit.com) llama 30w
-
-
How is Rotorquant/planarquant/iso qaunt better? (www.reddit.com) gemmallamaqwen 30w
-
-
Qwen 3.6 35B different quant speeds ? (www.reddit.com) llamaqwen 30w
-
Gemma4 26B MoE on Arc 140T (www.reddit.com) moellama 30w
-
Intel Arc B70 with HP z640 workstation (pcie 3) (www.reddit.com) llama 30w
-
Qwen 3.6 CoT issue? (www.reddit.com) llamaqwenopenai 30w
-
Inferena: Local benchmark of PyTorch vs. Llama.cpp vs. Rust frameworks (inferena.tech via hn) llama 30w
-
One-command local AI stack setup for Ubuntu (CUDA, Ollama, llama.cpp, chat UIs) (github.com via hn) ollamallama 30w
-
Show HN: Llama.cpp Tutorial 2026: Run GGUF Models Locally on CPU and GPU (news.ycombinator.com) llama 30w
Complete llama.cpp tutorial for 2026. Install, compile with CUDA/Metal, run GGUF models, tune all inference flags, use the API server, speculative decoding, and benchmark your hardware.
-
What I got by 5060Ti 16GB + Qwen3.6-35B-A3B-UD-Q5_K_M (www.reddit.com) ollamamoellama+2 30w
I tried local model couple weeks ago. At the beginning, I tried Ollama, but reddit says better to switch to llama.ccp.
-
5070ti + RX 9070 (non XT), over 100 tps on Qwen 3.6 35B Q4 (www.reddit.com) llamaqwen 30w
Hi guys, just want to share with you guys a Frankenstein build I put together that is surprisingly decent I have a i5 12400 / B660 / 32GB DDR4 build that was previously paired with a 3060ti. Last Christmas I upgraded it to a RX9070, then I…
-
I have Gemma4-E2B working within home assistant as STT, and E2B seems fast and accurate for STT (maybe a bit better than Parakeet), however, it responds with the entire thought process: https://preview.redd.it/v8zhb5elltvg1.png?width=599&f…
-
Imposing my laptop to run Qwen 3.6 (www.reddit.com) moellamaqwen 30w
So, I am excited with the new MoE model released by Alibaba. And as an excited person, I want to believe that it can actually run in my hardware.
-
Local Model Router: Ollama/OpenAI-compat bridges for local LLMs via llama.cpp (news.ycombinator.com) ollamallamaopenai 30w
A high-performance local LLM server providing drop-in API compatibility with Ollama and OpenAI, built on llama.cpp's llama-server. Features automatic VRAM management, Hugging Face integration, and modular architecture.
-
Testing Qwen3.6 with Hermes Agent on agentic coding. Locally with llama.cpp. (www.reddit.com) llamaagentic 30w
I'll be testing the setup and try out the Hermes Agent live: https://www.youtube.com/live/q5vqvwZykRI
-
new to llama.cpp want to use it in vscode (www.reddit.com) llamaopus 30w
I want to try llama.cpp instead of llmstudio. I want to know how to use this model qwen3.5-27b-claude-4.6-opus-uncensored-v2-kullback-leibler.
-
Qwen 3.6 dropped yesterday and I wanted to see if hybrid offloading actually earns its keep on this hardware. My box is two RTX 5060 Ti (32GB VRAM total) with 64GB system RAM.
-
Qwen3.6 Fails n8n Tool Calling (www.reddit.com) llama 30w
https://preview.redd.it/na4ub5yzprvg1.png?width=1654&format=png&auto=webp&s=e356e0ab0829bb275352d1035c35c645a381c3c7 I am using Kaggle to serve Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf but tool calling is not always working. I also tested it with R…
-
Best way to prepare for AI Engineer interviews? (www.reddit.com) llamaragagentic 30w
I’m currently preparing for AI-focused roles and would love to get perspectives from people already working in the industry. For context — I have ~5 years of experience as a Full Stack Engineer with a strong focus on AI systems.
-
7900XTX, Qwen 3.6 35B A3B, 150t/s that drops to 50t/s for no reason? (www.reddit.com) llamaqwen 30w
MSI B650 Gaming Plus 9800X3D 64GB DDR5 6400mts Windows 11 When I first boot my PC and I run this model, I get 155-160t/s, and for some reason, after a couple minutes, say, 10 minutes, not using AI or anything in particular, GPU temp at 40c…
- Low performance in 7900XTX in Qwen 3.6 35B A3B (www.reddit.com)
-
Hi guys, Back again. I have tested the Qwen 3.6 UD 2 K_XL Unsloth model on the same paper to web app task.
-
Which Qwen models can do FIM (Fill in the middle) for autocompletion? (www.reddit.com) llamaqwen 30w
I cannot find a definive answer. I think the following should be able to do FIM: Qwen 2.5 coder Qwen 3 coder Qwen 3-2507 instruct Qwen 3.5 Qwen 3.6 What I verified: Qwen3-32B: no Qwen3-4B-Instruct-2507: yes Qwen3.5-27B: yes Qwen3.6-35B-A3B…
-
I'm running a local AI setup and want to make sure I'm using my hardware to the absolute maximum. If you have tips on better models, smarter configurations, or services I'm missing, drop them in the comments.
-
Show HN: Open Access Qwen3.6-35B-A3B-UD-Q5_K_M with TurboQuant (news.ycombinator.com) llama 30w
https://w418ufqpha7gzj-80.proxy.runpod.net Started for myself, but since Im not using it continuously, sharing it: Open Access Qwen3.6-35B-A3B-UD-Q5_K_M with TurboQuant (TheTom/llama-cpp-turboquant) on RTX 3090 (Runpod spot instance). 5 pa…
-
Hola a todos! Aquí un novato en busca de ayuda (www.reddit.com) ollamallama 30w
Estoy un poco nuevo con esto de la IA, estoy tratando de aprender lo que más puedo temas como: * Skills * Agends * Models * LLM * Ollama * llama.cpp * Cuantizacion Pero estoy aún perdido, tengo en mi PC 32Gb de ram y quisiera ejecutar mode…
-
MI25 for LLMs? idc about speed, just need it to work (www.reddit.com) llama 30w
Found an MI25 locally for $50. It has 16GB of VRAM, which would be perfect for running some decent-sized local LLMs without breaking the bank.
-
llama.cpp + opencode agent temperature settings (www.reddit.com) llama 30w
Has anyone successfully set the temperature for individual agents of the opencode? I have set the temperature for individual agents, but when I start the llama-server in verbose mode the server claims the temperature is in default settings…
-
llama.cpp - split pp and tg processing over different instances? (www.reddit.com) llama 30w
I wonder, is it possible to split pp and tg over different (remote) llama.cpp instances, maybe via clever RPC calls?
-
Bonsai models are pure hype: Bonsai-8B is MUCH dumber than Gemma-4-E2B (www.reddit.com) gemmallama 30w
I'm using the https://github.com/PrismML-Eng/llama.cpp fork for Bonsai, regular llama.cpp for Gemma. Without embedding parameters: Gemma 4 has 2.3B at 4.8 bpw (Q4_K_M) = 1104 MB Bonsai-8B has 6.95B at 1.125 bpw (Q1_0) = 782 MB (only 29% sm…
-
M1 Pro 16GB users: what local LLM configs are actually usable day to day? (www.reddit.com) vllmollamallama 30w
I'm trying to get past generic "best model" recommendations and collect real-world configs from people on similar hardware. My setup: MacBook M1 Pro, 10-core CPU, 14-core GPU, 16 GB unified memory.
-
Been vibe coding a lot recently and kept running into the same problem finding actually usable tools without paying for 10 different subscriptions or donating my bank balance to Claude. So I put together a curated list focused on free or l…
-
Ask HN: What are the machine requirements for a LLM like Llama-3.1-8B? (news.ycombinator.com) llama 30w
I want to create a local GenAI. Tell me the server machine requirements.
-
TurboQuant on MLX & vLLM (www.reddit.com) vllmllama 30w
MLX https://github.com/Blaizzy/mlx-vlm?tab=readme-ov-file#turboquant-kv-cache vLLM https://github.com/vllm-project/vllm/pull/38479 MLX & vLLM users, please share your experience with benchmarks(t/s). Adding llama.cpp Links related to Turbo…
-
It's been some time now, surely some have tinkered with it more and optimised it already
-
Context checkpoint erasure in llama.cpp ? (www.reddit.com) function-callingllamaqwen 30w
Has anyone been able to solve or mitigate context checkpoints being erased during single user inference, specifically when function calling is part of the chat history? I've been using Qwen 3.5 35B A3B for some time (now using 3.6), tested…
-
Strix Halo concurrency 4 16k context 64 t/s Qwen3.6-35B-A3B-Q8_0 (www.reddit.com) llamaqwen 30w
https://preview.redd.it/4906akj9dovg1.png?width=1527&format=png&auto=webp&s=c49e255ac79a3c5455f44603422f8af7ddc12594 First of all can we make https://www.youtube.com/watch?v=2lUC8Gimxz8 Angine de Poitrine this subs official band? Those guy…
-
can someone explain how to use Matrix in Llama-swap ? (www.reddit.com) llama 30w
I noticed that groups have changed to Matrix , to allow concurrent models. Currently i use llama-swap for my models and an individual instance of llama-server for embedding and reranking all for Openweb UI.
-
what’s actually stopping an insider from leaking model weights? (www.reddit.com) llamaopenaianthropic 30w
this is a dumb question. what are the actual technical barriers stopping an engineer at a place like openai or anthropic from just exporting flagship weights and leaking them?
-
Anyone feel like Qwen3.6 thinks like Gemma 4? And not in a good way. (www.reddit.com) gemmallamaqwen 30w
I was disappointed with Gemma 4 due to various bugs and in the end lackluster performance for the internet research/information synthesis type tasks I use local AI for. Even after every last fix and update of both mode quants and llama.cpp…
-
Is there a way to have qwen-code CLI read images? (www.reddit.com) llamaqwencodex 30w
Basically I am asking the model to describe an image, but it says it can't process the images. The weird thing is that if I send the image encoded directly on the prompt, it works just fine, I am using llama-server with qwen3.5 (tried all…
-
Strix Halo 128GB on Proxmox - Vulkan vs ROCm benchmark matrix (www.reddit.com) minimaxgemmallama 30w
Ryzen AI MAX+ 395, Bosgame M5, 128GB LPDDR5x. Proxmox VE 9.1 LXC containers with GPU passthrough.
-
GPoUr with ~12gb vram and a 3080 getting 40tg/s on qwen3.6 35BA3B w/ 260k ctx (www.reddit.com) llama 30w
The TheTom's turboquant's GPU accelerated turboquant (turbo3) has unlocked high context gains for the 35BA3B family. I can now achieve ~40tg/s via the following GPU-POOR compilation flags and configuration: cmake -B build -DGGML_CUDA=ON -D…
-
lazy person's model param management for llama.cpp? (www.reddit.com) llama 30w
Has anyone found a good way to manage model params based on the recommendations of the model developers that doesn't require manually managing a local config file? I have an ever growing bash script for launching llama.cpp server which inc…
-
Cheapest and most efficient way to run 30B-40B Llama for 4 users? (www.reddit.com) llama 30w
Edit: the title has a mistake, I meant LLMs, but it autocorrected to Llama. Basically I am looking for a way to run 30B-40B LLMs locally for up to 4 users with lowest power draw possible.
-
Qwen3.6 local test (live) with llama.cpp. Is it going to be better than Gemma4? (www.youtube.com via reddit) llama 30w
About Press Copyright Contact us Creators Advertise Developers Terms Privacy Policy & Safety How YouTube works Test new features NFL Sunday Ticket © 2026 Google LLC
-
Here is how to run the new Qwen3.6-35B-A3B > At full context on a 4090 - IQ4_XS gguf with llama cpp > At full context on a Spark - FP8 with a tweaked vLLM Here is the docker compose with llama cpp services: llamacpp: container_name: llamac…
-
Gemma4 quirk to use ls -R; can we do better? (www.reddit.com) llamaqwen 30w
At the office I'm CPU and local only, so GPU poor. Besides the Qwen3.5 series, I've come to really like Gemma4 E4B there using the Pi agent (llama.cpp, Q4KM).
-
Comparison Qwen 3.6 35B MoE vs Qwen 3.5 35B MoE on Research Paper to WebApp (www.reddit.com) moellamaqwen 30w
Note: First is Qwen3.5 35B MoE (Left) and Second is Qwen3.6 (Right) Hi Guys Just did quick comparison of Qwen3.6 35B MoE against Qwen 3.5 35B MoE. with reasoning off using llama.cpp and same quant unsloth 4 K_XL GGUF First is Qwen3.5 outco…
-
Web OS result from Qwen3.6 35B is by far the best I tested in my laptop (codepen.io via reddit) llamaqwen 30w
This is my first test with this model and Qwen impressed me. I will rate it 98% usable web os compared to my previous best 70% usable result from qwen3 next coder at q2.
-
hi all. every time anyone on our team wanted to rent a GPU to run an open model, the flow was the same: open the HF page, eyeball the weights, open a VRAM calculator, open six cloud provider tabs, then the GPU spec pages because half of th…
-
Feedback on iOS app with local AI models (www.reddit.com) gemmallamadeepseek+1 30w
Hey everyone, I just shipped an iOS app that runs local AI models. Current has 12 models: Gemma 4, Llama 3.3, Qwen3, DeepSeek R1 Distill, Phi-4, etc.
-
Reproduction of TurboQuant (www.reddit.com) vllmllama 30w
There have been many TurboQuant implementations recently in llama.cpp, mlx, vllm, and sglang, but a lot of the discussion and code around them feels pretty noisy and looks to be AI-generated. I’m trying to understand which claims from the…
-
MINISFORUM AI X1 Pro-370 (96GB) - Local Ollama Help (www.reddit.com) ollamallamadeepseek+1 30w
Hey all. This just got delivered yesterday.
-
I've been running 5-8 Claude Code sessions at a time and got tired of tab-switching to approve tool calls. So I built claudectl — a TUI that sits on top of all your sessions and lets a local LLM (ollama/llama.cpp) handle approvals for you.
-
LiteRT LM Framework with Rockchip NPU (RKNN 3588) (www.reddit.com) gemmallama 30w
Im searching for build version of LiteRT LM framework can use and utilize the NPU of the RKNN 3588. It would be great since I can run gemma 4 e2b model using this framework on the machine, because I wont have to migrate my codebase from li…
-
gemma4 e4b on rtx 5070 ti laptop 12GB running slow 5t/s llama.cpp (www.reddit.com) ollamagemmallama 30w
I hope sincerely someonecan help me because i have tried everything i can and i get this speed using ollama.cpp and opencode. I have put as detail i can my setup and how i am running it.
-
Ask HN: Simple tooling for local LLM code critique without IDE integration? (news.ycombinator.com) llama 30w
While I'll set out the criteria for what I'm looking for, I don't want this to turn into a general debate about the role of LLMs in software development. That discussion is important, but we have plenty of them.
-
How faster is Gemma 4 26B-A4B during inference vs 31B? (www.reddit.com) gemmamoellama+1 30w
I want to download one and usually do inference on CPU having old GPU so I'm concerned with speed. One link on the web (I have posted with it and post been removed): Multiple users are reporting that Gemma 4's MoE model (26B-A4B) runs sign…
-
Turn an old Android phone into a Local AI Voice Assistant (www.reddit.com) gemmallama 30w
I had a nice old cracked pixel 5a laying around that I wanted to get some use out of, so I turned it into a local AI Voice assistant. A server on a laptop running llama.cpp gemma-3-4b-q4.gguf served by flask connects to a script running on…
-
The local LLM ecosystem doesn’t need Ollama (sleepingrobots.com via hn) ollamallama 30w
Friends Don't Let Friends Use Ollama Ollama gained traction by being the first easy llama.cpp wrapper, then spent years dodging attribution, misleading users, and pivoting to cloud, all while riding VC money earned on someone else's engine…
-
I try with Gemma 4 E4B via llama-sever to play chess at https://www.chess.com/play/computer (any platform or site you convenient), result quite unexpected for me. Result: 9 moves before it make cheating move (like try to move a pawn take a…
-
As the title suggests, I discovered model. ran bunch of batch process, I found my 1650 can't handle it and has to use shared memory.
-
Qwen3.5-35B running well on RTX4060 Ti 16GB at 60 tok/s (www.reddit.com) moellamamcp 30w
Spent a bunch of time tuning llama.cpp on a Windows 11 box (i7-13700F 64GB) with an RTX 4060 Ti 16GB, trying to get unsloth Qwen3.5-35B-A3B-UD-Q4_K_L running well at 64k context. I finally got it into a pretty solid place, so I wanted to s…
-
Gemma4 26b & E4B are crazy good, and replaced Qwen for me! (www.reddit.com) gemmallamaqwen+1 30w
My pre-gemma 4 setup was as follows: Llama-swap, open-webui, and Claude code router on 2 RTX 3090s + 1 P40 (My third 3090 died, RIP) and 128gb of system memory Qwen 3.5 4B for semantic routing to the following models, with n_cpu_moe where…
-
Hey, has anyone here used Qwen3.5-27B-NVFP4-GGUF with llama.cpp yet? (www.reddit.com) llama 30w
Hey! I was wondering if anyone of you have used Qwen3.5-27B-NVFP4-GGUF on RTX5090 on llama.cpp?
-
Right now I have 3 GPUs, 5060 Ti 16G, 2 x 4060 Ti 16G, and may get a used 3090 24G that I found. I could build a janky open rack system using M.2 and PCI risers with a 1600W PSU or try something like putting 2 GPUs in 2 systems using the f…
-
My kind of daily news sent to me via Discord https://preview.redd.it/prmris11vdvg1.png?width=684&format=png&auto=webp&s=0dcb00079362a38a29d981dd2f3a4e5143c8091f The N8N workflow (you could probably have Hermes or another agent do similar):…
-
We built a system where a neural compiler takes a plain-English function description and produces a "neural program" (a combination of a continuous LoRA adapter and a discrete pseudo-program). At inference time, these adapt a fixed interpr…
-
Are MLX 4-bit Quants broken (www.reddit.com) llama 30w
I see so many interesting MLX implementations like DFlash, Speculative Speculative decoding, etc. But when I want to try them for myself the 4bit quants of models seem like they have been lobotomised for some reason, hallucinating, start t…
-
How does a self correcting loop for AI agents work? (www.reddit.com) minimaxllamasonnet 30w
Hey guys, just checked out minimax 2.7, where they used AI to train itself, and ran over a hundred loops, and it improved it's performance by 30%, how does that work, can I also run a script that makes AI store it's memory in a loop on a m…
-
I wrote a book that implements modern LLM architectures from scratch. The part most relevant to this sub: Chapter 3 takes GPT-2 and swaps exactly 4 things to get Llama 3.2-3B: LayerNorm → RMSNorm Learned positional encodings → RoPE GELU →…
-
Offload settings for unsloth/Gemma-4 on Apple Silicon? (www.reddit.com) gemmallama 30w
Can default settings be optimized, or is it the best it is going to get? M1 Max Is it best in llama.cpp, LM Studio, or ?
-
What is the current status with Turbo Quant? (www.reddit.com) llama 30w
It has been hyped ±2 weeks ago and I remember seeing some pull requests into llama.cpp, but what is the current status after the hype faded away?
-
What's the better way to install llama.cpp on Android? (www.reddit.com) gemmallama 30w
I own an Oppo Find X3 Pro (Snapdragon 888, 12/256 GB, Android 14.0) unused because of 3 green vertical lines on the screen and poor battery. I tried Google AI Edge Gallery with Gemma-4-E2B-it and it performs well so I thinked: "why don't t…
-
Upgrade paths for my 256g ddr4 ram + 4x24g vram system (www.reddit.com) glmllama 30w
So I was just about to give up playing with local models, until I realised I can actually run GLM 5.1 at not too horrible speeds, using this quant https://huggingface.co/ubergarm/GLM-5.1-GGUF/tree/main/IQ2_KL in ik llama. Getting around 6.…
-
running models bigger than physical memory capacity (www.reddit.com) gemmallamaqwen 30w
has anyone really tried running models bigger than physical memory capacity? I'd guess most users stick with running models that fit in DRAM + VRAM https://unsloth.ai/docs/models/qwen3.5 even google gemma 4 are released with about 30+ bill…
-
Hey everyone, I’m currently an ML Engineer looking to pick up iOS development, and I’m upgrading my hardware to handle both. I’m moving away from cloud-only workflows and want to run LLMs locally for testing, R&D, and building CoreML integ…
-
I want to run qwen3.5 27B q4_k_m on CPU, and I need help. (www.reddit.com) llama 30w
I am an local LLM beginner and I found this Reddit while looking for help. (Please understand that I am unfamiliar with Reddit.) (system- i5 4440 1.8GHz/b85m ds3h/DDR3 32GB/128GB SSD/Ubuntu 25.10 questing) I loaded Qwen3.5 27B Q4_K_M onto…
-
Claude cooked on the code, but I wrote this post myself, caveman style. I wanted to play with Qwen3.5-122B, but I don't have a unified memory system to work with, and 15 tok/s was rough.
-
Can LLM make small change to the software program? (www.reddit.com) gemmallamaqwen 30w
I'm currently vibe-coding (I'm new to vibe-coding) with Gemma 4 4EB Q4 and Qwen 3.5 9B Q5 (KV is quantized to 4 bits with new Google TurboQuant implemented in llama.cpp - I use koboldcpp and release said it's automatically activated): the…
-
After months of testing, I finally have a local setup that doesn't make me want to go back to the API. Hardware: RTX 3090 (24GB VRAM) Models tested: Qwen2.5-Coder 32B Q4_K_M, DeepSeek-Coder-V3 Q4, Llama 3.3 70B Q3_K_M Inference: llama.cpp…
-
Qwen 122B is AMAZING but is my config right? (128GB M4 Max) (www.reddit.com) llamaqwen 30w
Hi! I hope its okay for me to ask this here.
-
Hey r/AI_Agents, I run an inference service (cheapestinference.com) and we're exploring a different pricing model that might be more predictable for agent workloads. Instead of per‑token billing, we offer **dedicated 8‑hour time windows**…
-
I am playing around with Intel Arc B70, still trying to decide whether I keep it or not. After some battle, I got it working with Radeon 5500 and B550M, now I am on to the fun part of getting software to work.
-
These "Claude-4.6-Opus" Fine Tunes of Local Models Are Usually A Downgrade (www.reddit.com) llamaqwenopus 30w
Time and time again I find posts about these fine tunes that promise increased intelligence and reasoning with base models, and I continuously try them, realize they're botched, and delete them shortly after. I sometimes do resort to a low…
-
Local Agent Hermes setup with Gemma 4 and llama.cpp (www.youtube.com via reddit) gemmallama 30w
About Press Copyright Contact us Creators Advertise Developers Terms Privacy Policy & Safety How YouTube works Test new features NFL Sunday Ticket © 2026 Google LLC
-
Running on cpu :( (www.reddit.com) mistralollamallama+1 30w
I am in the midst of a POC project at work and am I have is 4 AMD Epyc cores and those are essentially virtualized. Does any one have any tricks?
-
MiniMax M2.7 GGUF Investigation, Fixes, Benchmarks (www.reddit.com) minimaxllama 30w
Hey r/LocalLLaMA, we did an investigation into MiniMax-M2.7 GGUF causing NaNs on perplexity. Our findings show the issue affects 21%-38% of all GGUFs on Hugging Face (not just ours).
-
Need practical local LLM advice: Only having a 4GB RAM box from 2016 (www.reddit.com) ollamallama 30w
Sorry, not so tech person. I’m trying to figure out the most practical local LLM setup using my spare machine: 4 GB RAM No GPU for now, so please assume CPU-first unless I mention otherwise.
-
[Paper] Residual Streams / KV Direct (www.reddit.com) llama 30w
It seems we have entered a period of accelerating innovation regarding the KV cache. Someone mentioned this post's paper in the Github issue of llama.cpp for implementing Turbo Quant.
-
Vulkan compilation issue on Fedora (b8786) — solved (www.reddit.com) llama 30w
If you pull https://github.com/ggml-org/llama.cpp/releases/tag/b8786 and try to build with Vulkan support on Fedora, you may hit this error: [ 39%] Building CXX object ggml/src/ggml-vulkan/CMakeFiles/ggml-vulkan.dir/multi_add.comp.cpp.o /h…
-
LLM inference engine written ground-up natively in C#/.NET (dotllm.dev via hn) llama 30w
Pure C# pipeline Tokenizer, sampler, scheduler, kernels — all C#. No Python, no foreign runtime, no llama.cpp wrapper.
-
DotLLM – Building an LLM Inference Engine in C# (kokosa.dev via hn) llama 30w
Introducing dotLLM - Building an LLM Inference Engine in C# If you’ve been building .NET applications and wanted to run LLMs locally, your options have been… limited. You could wrap llama.cpp through LLamaSharp, deal with ONNX Runtime, or…
-
Llama.cpp llama-server command recommendations? (www.reddit.com) llama 30w
I've seen a ton of PR, and a bunch of failed PR with some interesting additions. I was wondering what other people's commands are looking like now, what they are running for llama.cpp I'm still running: CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6 l…
-
RTX 3090 llamacpp flags help (www.reddit.com) gemmallamaqwen 30w
Hi, my current system hardware RTX 3090 24GB VRAM & Sysrem RAM 64GB using windows 11 been playing around with hermes agent and local llm (Qwopus3.5-27B-v3-GGUF & gemma-4-26B-A4B-it-GGUF) when i try asking the hermes agent to do a task with…
-
Can I combine a RTX5060ti 16gb with 7900XTX 24gb for llama.cpp? (www.reddit.com) llamaqwen 30w
I bought this 7900XTX for 905 euro in Spain, and wondering if can I combine them together to run Qwen 3.5 27B for example ? Using a MSI B650 Gaming Plus Wifi and 64gb DDR5 6400mt/s
-
The LLM tunes its own llama.cpp flags (+54% tok/s on Qwen3.5-27B) (www.reddit.com) gemmallama 30w
This is V2 of my previous post. What's new: --ai-tune — the model starts tuning its own flags in a loop and caches the fastest config it finds.
-
Gemma4 vs Qwen3.5! MoE vs Dense! Sota vs Obsolete! Porque no los dos? (www.reddit.com) moellama 30w
Every other day, there's someone posting about how the latest hotness of the month is gamechanger, but flawed in some way relative to their previous favorite. I can't help but wonder, does no one else keep their previous gen models on spee…
-
Share your speculative settings for llama.cpp and Gemma4 (www.reddit.com) llama 30w
I have totally missed the boat on speculative decoding. Today when generating some code again for the frontend i found myself staring down at some quite monotonic javascript code.
-
What is the best way to deploy LLM on 3x3090? (www.reddit.com) vllmgemmallama 30w
Two questions: which model? In my mind, Qwen3.5 27b or Gemma 4 31b are top options.
-
3x3090 is faster in Ubuntu than win11, GPT-OSS 120B 120tg/s vs 6tg/s why? (www.reddit.com) llama 30w
using z790 prime p d4 with 128gb ddr4 3200mhz ram. 1x3090 in main PCIe5 16x slot and 2x3090 in chipset PCIe4 4x slots.
-
Alibaba's Qwen family captures over 50% of global open-source model downloads (www.scmp.com via hn) llamadeepseekqwen 30w
Advertisement Alibaba’s Qwen family captures over 50% of global open-source downloads, report finds Qwen hits nearly 1 billion cumulative downloads, far surpassing rivals like Meta Platforms’ Llama and DeepSeek, researchers say 2-MIN READ2…
-
My Custom Llama Build (www.reddit.com) openclawllama 30w
I recently got into LLM's and llama.cpp because I wanted to learn AI. I went from Openclaw to SOTA CLI and then to running llama on my Linux server.
-
If you are on Gemma (like me), you basically have to compile llama.cpp daily now
-
Older model suggestions (www.reddit.com) llama 30w
Due to costs I am running on some older hardware. Looking for suggestions on supported models for my particular stack.
-
current: 1x 16GB 5060Ti. worth a 2nd for OpenCode? (www.reddit.com) vllmllama 30w
my current build is just a 16GB 5060Ti running on a 3800X with 32GB DDR4. not really anything special, but I only really use it right now for Qwen3-VL-8B-Instruct at INT8 to do handwriting transcription (and it works great for that). someo…
-
How are you feeding personal context to your local models? (www.reddit.com) mistralollamallama+2 30w
I've been running Mistral/Llama locally through Ollama for a while now and the thing that keeps bugging me is context. The model itself is fine for general stuff but the second I want it to know about my projects, my notes, or files it doe…
-
Introducing BlueTTS (www.reddit.com) llama 30w
I recently worked on BlueTTS, a lightweight text-to-speech model that focuses on speed and usability. It supports multiple languages: English, Hebrew, Russian, Spanish, and French (even within the same sentence), and comes with a large set…
-
Local AI coding assistant that runs fully offline (Gemma 4, codebase-aware) (www.reddit.com) gemmallama 30w
I’ve been experimenting with running a local coding assistant on Gemma 4 26B, focused on understanding full codebases instead of single-file prompts. Main idea: - build a project map (files, symbols, structure) - run a planning step to dec…
-
Claude down? TokenMonopoly will help you find the best deals in AI subs (tokenmonopoly.com via hn) swe-benchllamadeepseek+1 30w
TokenMonopoly Live leaderboard of AI API deals — pricing, subscriptions, and SWE-bench scores for Claude, GPT, Gemini, Kimi, DeepSeek, Llama and more. Compare 27 benchmarked models across 96 hosts by price-per-performance, refreshed daily.
-
Intel Releases OpenVINO 2026.1 with Back End for Llama.cpp, New Hardware Support (www.phoronix.com via hn) llama 30w
Intel Releases OpenVINO 2026.1 With Backend For Llama.cpp, New Hardware Support Intel's OpenVINO toolkit for optimizing and deploying AI inferencing across their range of hardware platforms is out with its newest quarterly feature update.…
-
FYI, Step 3.5 Flash has better perf and context is 1/4 the price in llama.cpp (www.reddit.com) clinellama 30w
So i recently updated LMstudio after a long pause and updated my llama.cpp runtimes too.. i was shocked..
-
Open Claw on my old PC (32GB Ram, 12GB VRAM) model suggestions? (www.reddit.com) llama 30w
I tried running Gemma4 E4B through llama cpp, and I couldn't get it to reply wiithout timing out.
-
Is an nvidia DGK Spark or similar worth it? (www.reddit.com) llama 30w
I currently run a local model and mix of Claude max. My local model is run on cpu with 256 gb of ram and so it runs quite slowly.
-
how to disable reasoning/thinking with llama-server? (www.reddit.com) llama 30w
I run the same model: `google_gemma-4-E2B-it-IQ3_M.gguf` with lmstudio or llama-server and I connect thru `/v1/chat/completions` EP. with lm-studio, when I ask "tell me a story" i just get a story straight away: [google_gemma-4-e2b-it@iq3_…
-
Show HN: How to Use Google's Extreme AI Compression with Ollama and Llama.cpp (news.ycombinator.com) ollamallama 31w
The introduction of TurboQuant, PolarQuant, and QJL (Quantized Johnson-Lindenstrauss) by Google Research represents more than just a technical optimization. At Vucense, we view this as a landmark moment for Inference Sovereignty https://vu…
-
How to run Qwen3.5-27B with speculative decoding with llama.cpp llama-server? (www.reddit.com) llamamcp 31w
I run it on 2xRTX 3090. This is part of my llama-server presets file: [Qwen3.5-27B-bartowski] load-on-startup = true alias = Qwen3.5-27B-bartowski hf = bartowski/Qwen_Qwen3.5-27B-GGUF:Q8_0 hfd = bartowski/Qwen_Qwen3.5-2B-GGUF:Q8_0 draft-mi…
-
GGML and llama.cpp join HF to ensure the long-term progress of Local AI (huggingface.co) llama 38w
-
Measuring Open-Source Llama Nemotron Models on DeepResearch Bench (huggingface.co) llama 66w
-
Welcome the NVIDIA Llama Nemotron Nano VLM to Hugging Face Hub (huggingface.co) llama 72w
-
Welcoming Llama Guard 4 on Hugging Face Hub (huggingface.co) llama 80w
-
Welcome Llama 4 Maverick & Scout on Hugging Face (huggingface.co) llama 84w
-
“Llama 3.2 in Keras” (huggingface.co) llama 108w
-
Llama can now see and run on your device - welcome Llama 3.2 (huggingface.co) llama 111w
-
Deploy Meta Llama 3.1 405B on Google Cloud Vertex AI (huggingface.co) llama 117w
-
Llama 3.1 - 405B, 70B & 8B with multilinguality and long context (huggingface.co) llama 120w
-
Welcome Llama 3 - Meta's new open LLM (huggingface.co) llama 134w
-
-
Make your llama generation time fly with AWS Inferentia2 (huggingface.co) llama 157w
-
Non-engineers guide: Train a LLaMA 2 chatbot (huggingface.co) llama 163w
-
Llama 2 on Amazon SageMaker a Benchmark (huggingface.co) llama 163w
-
Fine-tuning Llama 2 70B using PyTorch FSDP (huggingface.co) llamafine-tuning 165w
-
Code Llama: Llama 2 learns to code (huggingface.co) llama 168w
-
Fine-tune Llama 2 with DPO (huggingface.co) dpollama 170w
-
Llama 2 is here - get it on Hugging Face (huggingface.co) llama 173w
-
StackLLaMA: A hands-on guide to train LLaMA with RLHF (huggingface.co) rlhfllama 188w