#llama

Friends Don't Let Friends Use Ollama Ollama gained traction by being the first easy llama.cpp wrapper, then spent years dodging attribution, misleading users, and pivoting to cloud, all while riding VC money earned on someone else's engine…

↯ Ollama↯ Llamaollamallama

My pre-gemma 4 setup was as follows: Llama-swap, open-webui, and Claude code router on 2 RTX 3090s + 1 P40 (My third 3090 died, RIP) and 128gb of system memory Qwen 3.5 4B for semantic routing to the following models, with n_cpu_moe where…

↯ Llama↯ Gemma↯ Qwen 3.5gemmallamaqwen+1

Hi LocalLLaMA, I created a post a few weeks ago, but this time this project has become more reliable and easier to use. This is a manga translator that can also be used to translate any image.

↯ Llama↯ Gemma↯ Qwen 3.5gemmallama

Looks like it finally happens... MTP getting approved for llama.cpp.

↯ Llamallama

↯ Llamallama

↯ Llamallama

Sparky runs entirely on the Jetson. Gemma 4 E4B at Q4_K_M via llama.cpp with q8_0 KV cache and flash attention.

↯ Llama↯ Gemma↯ Gemma 4gemmallama

https://www.ft.com/content/5630ed79-a263-41ed-9a1a-321617ae310e “The FT was able to use Heretic, a tool available on the popular code repository GitHub, to remove the guardrails from Meta’s Llama 3.3 model in less than 10 minutes without a…

↯ Llamallama

So maybe this is a no-brainer to many experienced local LLM users but it was not obvious for me. I am running a 3070 8gb + 64gb DDR4.

↯ Llama↯ Qwen 3.6llama

WARNING: wait before download from HF: I just realised my upload of the new versions with the additional fix in the chat template has not completed yet. I will remove this warning once done The recent PR to llama.cpp bring MTP support to Q…

↯ Llama↯ Qwen 3.6llamaqwenopenai+2

Hi all, I have been making a lot of updates to my project, and I wanted to share them here. TextGen (previously text-generation-webui, also known as my username oobabooga or ooba) has been in development since December 2022, before LLaMa a…

↯ Llamallama

Time and time again I find posts about these fine tunes that promise increased intelligence and reasoning with base models, and I continuously try them, realize they're botched, and delete them shortly after. I sometimes do resort to a low…

↯ Llama↯ Claude 4.6llamaqwenopus

Run on my rtx4090 llama.cpp params: llama-server -m ~/Projects/llm/models/Qwen3.6-27B-UD-Q4_K_XL.gguf --flash-attn on -ngl all -ctk q4_0 -ctv q4_0 -t 32 -c 262144 Power limit was set using sudo nvidia-smi -pl N On my observation, GPU const…

↯ Llama↯ Qwen 3.6llama

TL;DR best setup I tested on a RTX 3090 24 GB: ik_llama.cpp + Qwen3.6-27B-MTP-IQ4_KS.gguf 156k context, q8_0/q8_0 KV, MTP, vision on CPU benchmark result on a ~5.9k prompt + 1k output: about 1261 tok/s prefill, 72.9 tok/s decode llama.cpp…

↯ Llama↯ Vllm↯ Qwen 3.6vllmllamaqwen

This is V2 of my previous post. What's new: --ai-tune — the model starts tuning its own flags in a loop and caches the fastest config it finds.

↯ Llama↯ Gemma↯ Qwen 3.5gemmallama

Evaluated Qwen 3.6 27B across BF16, Q4_K_M, and Q8_0 GGUF quant variants with llama-cpp-python using Neo AI Engineer. Benchmarks used: HumanEval: code generation HellaSwag: commonsense reasoning BFCL: function calling Total samples: HumanE…

↯ Llama↯ Function Calling↯ Qwen 3.6humanevalfunction-callingllama+1

Here's my article with 38 quant pairs thoroughly benchmarked in KLD with 3 different Qwen 3.6 27B configs: Q5_K_S + 64k context, IQ4_XS + 64k context, IQ4_XS + 128k context. This allows us to track not only how cache quantizations affects…

↯ Llama↯ Qwen 3.6llamaqwen

It has been hyped ±2 weeks ago and I remember seeing some pull requests into llama.cpp, but what is the current status after the hype faded away?

↯ Llamallama

Spent a bunch of time tuning llama.cpp on a Windows 11 box (i7-13700F 64GB) with an RTX 4060 Ti 16GB, trying to get unsloth Qwen3.5-35B-A3B-UD-Q4_K_L running well at 64k context. I finally got it into a pretty solid place, so I wanted to s…

↯ Llama↯ Qwen 3.5moellamamcp

this is a dumb question. what are the actual technical barriers stopping an engineer at a place like openai or anthropic from just exporting flagship weights and leaking them?

↯ Llamallamaopenaianthropic

I have ThinkPad T14 Gen 5 (8840U, Radeon 780M, 64GB DDR5 5600 MT/s ). Tried out the recent Qwen MoE release, and pp/tg speed is good (on vulkan) (250+pp, 20 tg): ~/dev/llama.cpp master* ❯ ./build-vulkan/bin/llama-bench \ -hf AesSedai/Qwen3…

↯ Llama↯ Qwen 3.6moellamaqwen

Researchers Alec Radford (GPT, CLIP, Whisper), Nick Levine, and David Duvenaud just released talkie: a 13 billion parameter language model trained exclusively on text published before 1931. No internet.

↯ Llama↯ Sonnet 4.6llamasonnetgemini

I've been using Qwen3.6-27B-Q5_K_M with turbo3 KV cache since it's been released, and I haven't had any issues at all (no loops, no memory loss, etc.). However, I'm also aware that K cache compression is not really recommended in most case…

↯ Llama↯ Qwen 3.6llama

Had been getting great MTP performance with llama.cpp on my RTX 4070 Super 12GB, until they actually merged the MTP PR. Then, performance tanked and was barely above non-MTP.

↯ Llama↯ Qwen 3.6llama

I'm using the https://github.com/PrismML-Eng/llama.cpp fork for Bonsai, regular llama.cpp for Gemma. Without embedding parameters: Gemma 4 has 2.3B at 4.8 bpw (Q4_K_M) = 1104 MB Bonsai-8B has 6.95B at 1.125 bpw (Q1_0) = 782 MB (only 29% sm…

↯ Llama↯ Gemma↯ Gemma 4gemmallama

↯ Llama↯ Qwen 3.6llama

looks like you can run it on any potato (A1B)! https://huggingface.co/LiquidAI/LFM2.5-8B-A1B-GGUF from LiquidAI: LFM2.5 is a new family of hybrid models designed for on-device deployment.

↯ Llama↯ Vllmvllmmoellama+1

Note: First is Qwen3.5 35B MoE (Left) and Second is Qwen3.6 (Right) Hi Guys Just did quick comparison of Qwen3.6 35B MoE against Qwen 3.5 35B MoE. with reasoning off using llama.cpp and same quant unsloth 4 K_XL GGUF First is Qwen3.5 outco…

↯ Llama↯ Qwen 3.6moellamaqwen

Hardware: RTX 3060 12GB 32GB DDR4-3200 Windows CUDA 13.x Model: Qwen3.6-35B-A3B-MTP-IQ4_XS.gguf The model is a 35B MoE, so -ncmoe matters a lot. Lower -ncmoe means more MoE blocks stay on GPU.

↯ Llama↯ Qwen 3.6moellamaqwen

With the MTP llama.cpp implementation in the Qwen3.6/3.5 models more VRAM is required for the MTP layer. However, many people don't realize this layer comes with its own KV cache which can also be quantized: -cache-type-k-draft q8_0 -cache…

↯ Llama↯ Qwen 3.6llama

I built a markdown web renderer for AI agents. Instead of taking expensive screenshots and piping them through vision models, TextWeb renders web pages as markdown that LLMs can reason about natively.

↯ Llamallamamcp

Using 100k context with 3090 with MTP GGUF and getting 50 t/s on llama.cpp Thought I would knowledge share Use https://huggingface.co/RDson/Qwen3.6-27B-MTP-Q4_K_M-GGUF And am17an commit /media/adam/D_DRIVE/LLM/llama-cpp-am17an/build/bin/ll…

↯ Llama↯ Qwen 3.6llamaqwen

https://preview.redd.it/1uuzjm0ll72h1.png?width=923&format=png&auto=webp&s=1af7d7594be1e08ff7ad6797e2bc53e9410769a3 update to 0.4.14 Build 2 (Beta) and make sure your llama.cpp engine is 2.15.0 https://preview.redd.it/x0vdwjb3n72h1.png?wid…

↯ Llamallama

https://unsloth.ai/docs/models/mistral-3.5 "May 1, 2026 Update: We worked with Mistral to fix Mistral Medium 3.5 inference affecting some implementations, and released updated GGUFs with the fix (NOT related to Unsloth or our quants). The…

↯ Llama↯ Mistral↯ Mistral 3.5mistralllama

All across the land many monitors showing green cmake with giddy anticipation Tip your bartender! https://github.com/ggml-org/llama.cpp/releases/tag/b9180

↯ Llamallama

How to Setup a Local Coding Agent on macOS Running Gemma 4 26B-A4B and Qwen3.6 35B-A3B locally with llama.cpp, MTP speculative decoding, multimodal support, and PI as a coding agent. I'd had my internet fail a few times recently leaving me…

↯ Llama↯ Gemma↯ Qwen 3.6↯ Qwen 3.6gemmallama

Cuda 13.3 Downloads Release Notes Anybody already tried llama.cpp with 13.3?

↯ Llamallama

Maybe it be helpful for someone: llama-server -m '/Qwen3.6-27B/Qwen3.6-27B-IQ4_XS.gguf' -ngl 999 -ctk q4_0 -ctv q4_0 -b 128 -ub 128 -c 24000 Cant run this model with higher kv quants on >8192ctx size. -ub & -b setted for 256 allowed me for…

↯ Llama↯ Qwen 3.6llamaqwen

So I've been messing around trying to get MTP working alongside TBQ4_0 (TurboQuant's lossless 4.25 bpv KV cache) on Qwen3.6-27B for my own use. So after a day of vibecoding I think I may have gotten something viable.

↯ Llama↯ Qwen 3.6deepseekllama

TL:DR - Remembered FPGA PCI boards being a big thing from my crypto days. Wondered if AMD Alveo V80 FPGA card could be used to approximate the performance of a Taalas HC1 (LLM-on-a-chip).

↯ Llama↯ Llama 3.1llamagemini

MTP (Multi-Token Prediction) just merged into mainline llama.cpp at b9190. I promised u/WarthogConfident4039 a Qwen3.6 benchmarking round.

↯ Llama↯ Qwen 3.6moellama

Hugging Face has been acquired by NVIDIA It is quite exciting to be a part of this journey! NVIDIA has been an active supporter of the llama.cpp project.

↯ Llamallama

OpenLake Leads MLPerf Storage v3.0 OpenLake’s Infinity Core I/O Engine delivered 6.72 GiB/s writes and 11.55 GiB/s reads in the MLPerf Storage v3.0 Llama 3.1 8B checkpointing benchmark. September 1, 2026.

↯ Llamallama

I've been running structured output prompts through a bunch of models on OpenRouter for the past few months — Llama 3, Mistral, Command R, DeepSeek, Qwen, and every other model on OpenRouter — alongside the usual closed-source suspects. 28…

↯ Llama↯ Mistralmistraldeepseekllama+1

vLLM has the ability to run .safetensors LLMs before they are converted to GGUF and represents a new engine to explore. I personally had never tried it out until u/krishna2910-amd/ u/mikkoph and u/sa1sr1 made it as easy as running llama.cp…

↯ Llama↯ Vllmvllmllama

I ran the 62-item politicalcompass.org test 30 times each on sixteen models: OpenAI's GPT-5.x and GPT-4o, Claude, Gemini, Grok, Llama, Mistral, and China's DeepSeek, Qwen, Kimi and GLM. Fifteen land in the libertarian-left quadrant.

↯ Glm↯ Llama↯ Mistral↯ GPT 5mistralglmgrok+6

Hey folks, just a heads-up for anyone running Qwen3.6 through llama-server. I ran into an issue where the preserve_thinking parameter wasn't working as expected, even though I had it explicitly enabled in my models.ini config.

↯ Llama↯ Qwen 3.6llama

Last year researchers affiliated with NVIDIA, University of Warsaw, and University of Edinburgh published Dynamic Memory Sparsification (DMS), a KV-cache sparsification technique using learned per-head token eviction, reporting up to 8x KV…

↯ Llama↯ Vllmvllmllama

↯ Llama↯ Minimaxminimaxllama

Setup: Kubuntu 24.04 - AMD cards - R9700 AI PRO and 7800xt (32gb + 16gb) - llama-cpp server - stack setup in docker - vulkan image I tried with ROCM but it wouldn't play nice with RDNA4 + RDNA3 mix. Vulkan seems to work.

↯ Llamallama

Ok, hear me out. This all started when I was trying to understand why this Qwen3.6 27B INT8 Autoround (https://huggingface.co/Minachist/Qwen3.6-27B-INT8-AutoRound/tree/main) recipe was performing so much better than any other Qwen3.6 27B q…

↯ Llama↯ Vllm↯ Qwen 3.6vllmllama

I have snobbishly long felt that the local models were not 'up to my standards' for local development, or otherwise able to compete with GHCP, Claude Code, Cursor etc. Boy was I wrong.

↯ Llama↯ Qwen 3.6llamacursorclaude-code

Hi everyone, I'm presenting a new quantization of the Qwen-27B model, created specifically with 16GB VRAM NVIDIA GPUs in mind. I used quants that, unfortunately, are not yet available in the main upstream llama.cpp.

↯ Llama↯ Qwen 3.6llamaqwen

Hey everyone, I’ve been building a local-first desktop PDF reader that can read technical books aloud and keep the spoken text highlighted while reading. The original motivation was pretty practical: I read a lot of programming and technic…

↯ Llamallamaqwen

Hey everyone, I've been working on getting Multi-Token Prediction (MTP) working with quantized GGUFs for Qwen3-27B and the results are pretty impressive. Here's what I put together: https://huggingface.co/havenoammo/Qwen3.6-27B-MTP-UD-GGUF…

↯ Llama↯ Qwen 3.6llama

I look forward to the Local LLM community getting llama.cpp to run on these. Could be a good value.

↯ Llamallama

TL;DR New llama.cpp fork! I wanted a Windows-friendly inference to run Qwen 3.6 27B Q5 on a single RTX 3090 with speculative decoding, high context without excess quantization, and vision enabled.

↯ Llama↯ Qwen 3.6llamaqwen

↯ Llama↯ Qwen 3.6moellamaagentic

I have totally missed the boat on speculative decoding. Today when generating some code again for the frontend i found myself staring down at some quite monotonic javascript code.

↯ Llama↯ Gemma 4llama

TL;DR All models were Qwen3.6 27B-MTP vs Base 27B (15k single-turn): Faster overall Total Time (wall): 87.44s → 77.39s (10.05s faster / -11.50%) Generation: 7.63 → 16.15 t/s (+111.77% speedup) Prompt Processing: 279.75 → 244.90 t/s (-12.46…

↯ Llama↯ Qwen 3.6llama

↯ Llamallama

This PR deserves much more attention as it fixes the constant promptprocessing that happens when using llama.cpp with Opencode or pi. https://github.com/ggml-org/llama.cpp/pull/22929

↯ Llamallama

Setup: - RTX 5090, 32 GB, Linux - Built llama.cpp from 4f13cb7 (the official ghcr.io/ggml-org/llama.cpp:server-cuda image hasn't picked up the merge yet as of writing — had to docker build from source with CUDA_DOCKER_ARCH=120) - Unsloth's…

↯ Llama↯ Qwen 3.6llama

So, as most of us here are, I'm a llama.cpp loyalist. Easy to understand, great configuration, relatively stable, etc.

↯ Llama↯ Vllmvllmllama

Hey r/LocalLLaMA, we did an investigation into MiniMax-M2.7 GGUF causing NaNs on perplexity. Our findings show the issue affects 21%-38% of all GGUFs on Hugging Face (not just ours).

↯ Llama↯ Minimaxminimaxllama

To preface, here's my config: llama-server \ --host 0.0.0.0 \ --port 1235 \ --models-preset %h/Software/models.ini \ --models-max 1 \ --sleep-idle-seconds 3600 \ --timeout 3600 \ --parallel 1 \ --device ROCm0,ROCm1 [*] flash-attn = on jinj…

↯ Llama↯ Qwen 3.6llama

ik_llama.cpp is great for both CPU & CUDA. Need legends to make Vulkan better as well.

↯ Llamallama

I wrote a book that implements modern LLM architectures from scratch. The part most relevant to this sub: Chapter 3 takes GPT-2 and swaps exactly 4 things to get Llama 3.2-3B: LayerNorm → RMSNorm Learned positional encodings → RoPE GELU →…

↯ Llamamoedeepseekllama

If you are on Gemma (like me), you basically have to compile llama.cpp daily now

↯ Llama↯ Gemma↯ Gemma 4gemmallama

I recently worked on BlueTTS, a lightweight text-to-speech model that focuses on speed and usability. It supports multiple languages: English, Hebrew, Russian, Spanish, and French (even within the same sentence), and comes with a large set…

↯ Llamallama

## Got DFlash speculative decoding working on Qwen3.5-35B-A3B with an RTX 2080 SUPER 8GB I managed to get **DFlash speculative decoding** working in llama.cpp on a pretty VRAM-limited setup. This was tested with the DFlash PR: https://gith…

↯ Llama↯ Qwen 3.5moellama

Hi, AesSedai here - I've put up a PR to support the text-to-text inference of MiMo V2.5 with llama.cpp (and should also support Pro, will work on those quants after finishing V2.5): https://github.com/ggml-org/llama.cpp/pull/22493 I've als…

↯ Llamamoellama

As a life-long Windows user (don't hate me, I was exposed to it at a young age) I was wondering how much (if any) performance I'm leaving on the table. So I did the sensible thing and run some benchmarks.

↯ Llama↯ Gemma↯ Qwen 3.6gemmallama

Hi guys, Back again. I have tested the Qwen 3.6 UD 2 K_XL Unsloth model on the same paper to web app task.

↯ Llama↯ Qwen 3.6llamaqwen

So i recently updated LMstudio after a long pause and updated my llama.cpp runtimes too.. i was shocked..

↯ Llamaclinellama

Check the README for setup instructions: https://github.com/BigStationW/llama-cpp-phrase-ban

↯ Llamallama

TLDR: tool parameters using the common JSON Schema pattern `anyOf: [$ref, null]` are rendered into the prompt as empty `type` fields. This strips the useful schema information before the model sees it.

↯ Llama↯ Gemma↯ Qwen 3.5gemmagpt-5llama+1

Hello everyone! I want to share the result of my experiment to make Qwen3.6 27B Q4_K_M fits in to my RTX 5060 Ti 16 GB.

↯ Llama↯ Qwen 3.6llama

I tested two llama.cpp builds on the same Qwen3.6-27B-NVFP4 model. llama-bench reports the model label as qwen35 27B NVFP4, but the actual tested model is Qwen3.6-27B-NVFP4.

↯ Llama↯ Qwen 3.6llama

Making sure you're not a bot! Loading...

↯ Llamallama

Hey everyone, Ever since the day Google announced TurboQuant, I've been following the news about its extreme compression capabilities without noticeable quality degradation. I see it mentioned constantly on this sub, but despite all the di…

↯ Llama↯ Qwen 3.5llama

This is my first test with this model and Qwen impressed me. I will rate it 98% usable web os compared to my previous best 70% usable result from qwen3 next coder at q2.

↯ Llama↯ Qwen 3.6llamaqwen

My kind of daily news sent to me via Discord https://preview.redd.it/prmris11vdvg1.png?width=684&format=png&auto=webp&s=0dcb00079362a38a29d981dd2f3a4e5143c8091f The N8N workflow (you could probably have Hermes or another agent do similar):…

↯ Llamallama

PR #22673 (commit 4f13cb7) landed MTP speculative decoding in mainline llama.cpp on May 16. I tested it on two separate rigs.

↯ Llama↯ Qwen 3.6moellama

for anyone who cares... 😄 prompt = spen a 1000 tokens unsloth MTP models strix halo llama.cpp:server-rocm-mtp \ --spec-type draft-mtp \ --spec-draft-n-max 3 Qwen3.5-122B-Q5-MTP-General n_decoded = 100 tg = 29.77 t/s n_decoded = 179 tg = 27…

↯ Llama↯ Qwen 3.5llama

b9095 finally makes -sm tensor work on dual consumer Blackwell PCIe GPUs without NCCL If youre on dual Blackwell gpus this look like it could be big. I'll have my own results for 2x5060ti asap

↯ Llamallama

I tried mudler's apex quant for gemma4 26b a4b and it was amazing! I got 38tps at 90.000 context with no loop and suprisingly no quality degradation.

↯ Llama↯ Gemma↯ Gemma 4gemmallama

I converted nvidia/llama-embed-nemotron-8b to MLX fp16, 8-bit, 4-bit, and 2-bit (for my OCD) and put it on HuggingFace: ncorder/llama-embed-nemotron-8b-mlx-fp16 ncorder/llama-embed-nemotron-8b-mlx-8bit ncorder/llama-embed-nemotron-8b-mlx-4…

↯ Llamallama

DeepSeekv3 OG DeepSeekv3.2/4 Qwen3.5 GLM4.5+ MiniMax2.5+ Step3.5Flash Mimo v2+ Until we get mtp weights, you need to download HF weights and convert to gguf. I think I'm going to try either qwen3.5-122b or glm4.5-air first.

↯ Llama↯ DeepSeek 3.2llama

Took TheTom's TurboQuant Metal fork of llama.cpp (github.com/TheTom/llama-cpp-turboquant, the feature/turboquant-kv-cache branch) and ran a depth sweep on Qwen 3.6-35B-A3B Q8. TheTom had already published M5 Max numbers up to 32K.

↯ Llama↯ Qwen 3.6llamaqwen

I am blown away by what this model can generate locally. I asked for a flashy Tetris game with particle effect and boy did it deliver!

↯ Llama↯ Qwen 3.6moellama

We built a system where a neural compiler takes a plain-English function description and produces a "neural program" (a combination of a continuous LoRA adapter and a discrete pseudo-program). At inference time, these adapt a fixed interpr…

↯ Llama↯ Qwen 3llama

The “Trash Can” Mac Pro, once the most expensive machine you could buy from Apple, mine was just shy of £10,000 in 2016 — that’s £14k in today’s money. Until recently mine was just running as a kubernetes single node development platform,…

↯ Llama↯ Qwen 3.5llamaqwen

Following my previous post https://www.reddit.com/r/LocalLLaMA/comments/1t5ageq, a few people asked for the 35B A3B version. The model is up on HuggingFace at https://huggingface.co/havenoammo/Qwen3.6-35B-A3B-MTP-GGUF if anyone wants to ch…

↯ Llama↯ Qwen 3.6llama

Previously a model could only be present in a single group. Now you can create whatever groups you want: one for big models that should run on their own, a group for STT + bigger model, a group for RAG usages, etc.

↯ Llamallamarag

I feel like i'm going insane. I see people here posting 30 - 100+ tok/s (100+ being with speculative decoding) on a 3090 with Qwen 3.6 27B.

↯ Llama↯ Sonnet 4.6llamaqwensonnet

Qwen 3.6 dropped yesterday and I wanted to see if hybrid offloading actually earns its keep on this hardware. My box is two RTX 5060 Ti (32GB VRAM total) with 64GB system RAM.

↯ Llama↯ Qwen 3.6moellamaqwen

Here is how to run the new Qwen3.6-35B-A3B > At full context on a 4090 - IQ4_XS gguf with llama cpp > At full context on a Spark - FP8 with a tweaked vLLM Here is the docker compose with llama cpp services: llamacpp: container_name: llamac…

↯ Llama↯ Vllm↯ Qwen 3.6vllmllama

So, last week I tried to update my unused local LLM setup. I had to stop using it because quality was too low and deepseek was too cheap.

↯ Ollama↯ Llama↯ Qwen 3.6ollamadeepseekllama

Out of random curiousity I ran a shootout on Qwen3-Coder-Next. I've been using the MXFP4_MOE from unsloth for awhile as it's just really fast on my system.

↯ Llama↯ Qwen 3↯ Qwen 3↯ Qwen 3↯ Qwen 3↯ Qwen 3↯ Qwen 3↯ Qwen 3↯ Qwen 3↯ Qwen 3llama

I wanted to share an open-source app that I built for running LLMs locally on my setup. My setup Hardware FEVM FAEX1 (128GB) RTX Pro 5000 Blackwell (48GB), connected over OCuLink Aoostar AG02 2x2TB internal m.2 drives on raid-0 using mdadm.

↯ Llama↯ Qwen 3.6llamaqwen

For those who want to run latest dense ~30b models and only have 16GB VRAM, if you have a old card with 6GB VRAM or more, plug it in. It matters that everything fits on the VRAM, even on 2 cards.

↯ Llamallama

I'm posting this because it may be helpful to squeeze the 12GB VRAM in the 3060. All credit goes to spiritbuun's fork (github.com/spiritbuun/buun-llama-cpp) and mudler's APEX quantizations (huggingface.co/mudler).

↯ Llama↯ Qwen 3.6↯ Qwen 3.6↯ Qwen 3.6↯ Qwen 3.6↯ Qwen 3.6↯ Qwen 3.6↯ Qwen 3.6↯ Qwen 3.6↯ Qwen 3.6↯ Qwen 3.6llama

Just wondering how are people's experience with both these models! I've had some nice results with Qwen but Gemma4 runs so much faster here.

↯ Llama↯ Qwen 3.6llamaqwen

Probably most of you are aware that using anything other than -ctk q8_0 -ctv q8_0 / -ctk q4_0 -ctv q4_0 as startup options for llama.cpp leads to prompt processing on cpu instead of gpu for cuda at least. E.g.

↯ Llamallama

In my opinion, MTP models are 100% game changer for local LLMs. In terms of speed, I was getting around 1.5x the tok/sec of previous tests.

↯ Llama↯ Qwen 3.6llamaqwen

TLDR: The hype is real! 1.5x speedup.

↯ Llama↯ Qwen 3.6llama

I've been running PaddleOCR-VL-1.5 via llama.cpp's server for OCR on book pages. It handles complex layouts, tables, and mixed text/figure pages surprisingly well.

↯ Llamallama

A few days ago, I was trying to improve token generation speed on my RTX 4070 Super 12GB while running Qwen3.6 35B A3B UD-IQ3_XXS (Unsloth) with llama.cpp, but to no avail. At that time, I had my monitor plugged in my 4070 and didn't even…

↯ Llama↯ Qwen 3.6llama

↯ Llama↯ Qwen 3.6clinellamaqwen+1

A few weeks ago, after finishing FastDMS, I started toying around writing some RDNA3 kernels again to see how fast I could get Qwen 3.6 MoE running. It turned out well enough, so over the past couple weeks, I turned those experiments into…

↯ Llama↯ Qwen 3.6moellamaqwen

Just wanted to share because it took me a lot of tweaking to get here: llama-server -hf unsloth/MiniMax-M2.7-GGUF:UD-IQ3_XXS --temp 1.0 --top-k 40 --top-p 0.95 --host 0.0.0.0 --port 8080 -c 100000 -fa on -ngl 999 --no-context-shift -fit of…

↯ Llama↯ Minimax↯ MiniMax 2.7minimaxllama

antirez @antirez Welcome to DS4, a specialized inference engine for DeepSeek v4 Flash. github.com/antirez/ds4 This project would have been impossible without the existence of llama.cpp and GGML and the work of @ggerganov and all the other…

↯ Llama↯ DeepSeek 4deepseekllama

Just ran some llama-bench comparisons between ROCm and Vulkan backends on my Strix Halo system. Vulkan came out ahead, which surprised me.

↯ Llama↯ Qwen 3.6moellama

I read a lot of "whengguf" type posts. I think we should sometimes stop and be grateful.

↯ Llama↯ Mistral↯ Gemmamistralgemmallama+1

Hi folks, Enjoy an optimised Qwen3.6 35B-A3B and Qwen3.6 27B for coding and general purpose - it's able to solve puzzles correctly more often too. The initial intent was to optimise the 35B-A3B reasoning traces since it's the most efficien…

↯ Llama↯ Qwen 3.6llama

Hi, I'm trying to run Qwen 3.6-35B on my RTX 3090 (24 GB of VRAM) but I'm not sure about 2 thing: - Which variant of the model to use ? (Q4_K_S, Q3_K_XL, other ?

↯ Llama↯ Qwen 3.6llamaqwenagentic

↯ Llama↯ Qwen 3.6llama

Been vibe coding a lot recently and kept running into the same problem finding actually usable tools without paying for 10 different subscriptions or donating my bank balance to Claude. So I put together a curated list focused on free or l…

↯ Ollama↯ Llamaollamallamarag+3

I had a nice old cracked pixel 5a laying around that I wanted to get some use out of, so I turned it into a local AI Voice assistant. A server on a laptop running llama.cpp gemma-3-4b-q4.gguf served by flask connects to a script running on…

↯ Llama↯ Gemma↯ Gemma 3gemmallama

Here's the PR by pedapudi. https://github.com/ggml-org/llama.cpp/pull/21344 It's merge request has been denied so it will not be in mainline llama.cpp.

↯ Llamallama

I have an Asus gaming laptop from 2021 that I bought used for 500€ last year. I wanted to see if the recently merged MTP support in llama.cpp is worth using on such a VRAM constrained device for the Qwen3.6-35B-A3B model.

↯ Llama↯ Qwen 3.6llama

Hey everyone, I've been playing around with Gemma4 and Qwen3.6 on my 32Gb Macbook Pro M2 Max since their release but I'm struggling at finding: The best software to run it (oMLX, llama.cpp, ...) The best model + quant to pick The best sett…

↯ Llama↯ Qwen 3.6llamaagentic

dicking around with the new mtp speculative decode with qwen3.6 27b, and it’s great. but for agentic coding i’ve seen significant improvements from ngram, because a decent fraction of the time (e.g.

↯ Llama↯ Qwen 3.6llamaagentic

Bleeding Llama: Critical Unauthenticated Memory Leak in Ollama TL;DR We discovered a critical vulnerability (CVE-2026–7482, CVSS 9.1) in Ollama that enables unauthenticated attackers to leak the entire Ollama process memory, potentially im…

↯ Security↯ Ollama↯ Llamaollamallamasecurity

In case anyone is interested, I decided to test out LLama.cpp's new OpenVino backend to see how it compares on Intel GPUs. At first glance, it stomps all over the previous best-case, SYCL, but lags behind LLM-Scaler (Intel's VLLM fork), li…

↯ Llama↯ Vllmvllmllama

EDIT - IGNORE. I MADE A MISTAKE.

↯ Llama↯ Vllm↯ Qwen 3.6↯ Qwen 3.6↯ Qwen 3.6↯ Qwen 3.6↯ Qwen 3.6↯ Qwen 3.6↯ Qwen 3.6↯ Qwen 3.6↯ Qwen 3.6↯ Qwen 3.6↯ Qwen 3.6vllmllama

Llama.cpp recently introduced support for Programmatic Dependent Launch (PDL), which is a new feature in Nvidia GPUs (CC >= 90, not including ADA) such as Blackwell. (See PR 22522.) In short, PDL enables more efficient execution of kernels…

↯ Llama↯ Qwen 3.6llamaqwen

Hi! I have been using Qwen3.6 35B A3B happily the past few weeks, and I wanted to try out Qwn3.6 27B with the new fancy MTP speculative draft!

↯ Llama↯ Qwen 3.6moellama

I'm the founder behind Hedy, an AI meeting app. I'm a huge supporter of Local AI, and we've been working on making it "consumer friendly".

↯ Llama↯ Gemma↯ Qwen 3.6gemmallamaqwen

I was wondering what will be the difference in results with flag: GGML_CUDA_ENABLE_UNIFIED_MEMORY=1 vs MTP+GGML_CUDA_ENABLE_UNIFIED_MEMORY=1 Results are quite interesting 49tok/sec without MTP vs 64 tok/sec with MTP. PC: RTX5090+128GB DDR5…

↯ Llama↯ Qwen 3.6llamamcp

I've got a 128GB Strix Halo box. Yesterday I wanted to try out Step-3.5-flash.

↯ Llamallama

I’ve always liked how if I ask ChatGPT to make or edit an image, it just does it. Local AI should be this convenient!

↯ Llamallamachatgptopenai

In my local LLM setup I get from 30 to 80 t/s generation at the beginning, but it drops quite a lot as context grows. I use llama.cpp/Vulkan with an MI50 and a V100, is there some command line flags that can improve this issue?

↯ Llamallama

The TheTom's turboquant's GPU accelerated turboquant (turbo3) has unlocked high context gains for the 35BA3B family. I can now achieve ~40tg/s via the following GPU-POOR compilation flags and configuration: cmake -B build -DGGML_CUDA=ON -D…

↯ Llama↯ Qwen 3.6llama

Claude cooked on the code, but I wrote this post myself, caveman style. I wanted to play with Qwen3.5-122B, but I don't have a unified memory system to work with, and 15 tok/s was rough.

↯ Llama↯ Qwen 3.5llama

Used the vllm version of https://github.com/noonghunna/club-3090 It worked fine for myabe 20 40k context, havent tried the new one. Anyone used the new llama.cpp patched one for single 3090?

↯ Llama↯ Vllm↯ Qwen 3.6↯ Qwen 3.6↯ Qwen 3.6↯ Qwen 3.6↯ Qwen 3.6↯ Qwen 3.6↯ Qwen 3.6↯ Qwen 3.6vllmllamaqwen

I put together a small public repo for RTX 5060 Ti 16GB local LLM setups: I took inspiration from the club-3090 repo, but this one is focused on documenting what we’ve actually tested on 5060 Ti hardware so the setup details are easier to…

↯ Llama↯ Vllm↯ Qwen 3.6vllmllamaopenai

Finally feel like it’s possible. I have a custom build (vibe coded) UI on llama.cpp, allows model switching in the same chat.

↯ Llama↯ Qwen 3.6llama

https://github.com/ggml-org/llama.cpp/blob/master/docs/backend/snapdragon/README.md I have an Oneplus 12 with Snapdragon 8 Gen 3. I followed the above README to cross-compile llama.cpp on Ubuntu and then copy to the Termux directory on the…

↯ Llama↯ Gemma↯ Gemma 3gemmallama

Hi r/LocalLLaMA ! I built a 5M Llama model with HF Transformers on 2x T4 in Kaggle to see, if it is able to be as good as my previous Apex 350M model (https://huggingface.co/LH-Tech-AI/Apex-1.6-Instruct-350M).

↯ Llama↯ GPT 2llama

I'm getting ~13 tps on Q8_0, with a context window of 128000, K Q8_0, V Q8_0 this is on 3x GPUS (1x2060super 8gb, 2x5060ti 16gb), via llamacpp unsure if this is slow or to be expected? */llama-server --port 8080 --model */llama.cpp/Qwen3.6…

↯ Llama↯ Qwen 3.6llama

Memory-augmented Large Language Models (LLMs) are essential for developing capable, long-term AI agents. Recently, applying Reinforcement Learning (RL) to optimize memory operations, such as extraction, updating, and retrieval, has emerged…

↯ Llama↯ Fine Tuningfine-tuningllama

Hello HN, I built Minute because I wanted searchable meeting notes without sending recordings or transcripts to a cloud service. It captures microphone and optional system audio, transcribes locally with Whisper, and generates summaries, d…

↯ Llamallamacodex

A proposed class action filed against Meta Platforms in New York federal court targets not only the company and its CEO Mark Zuckerberg but also two former senior AI researchers by name—an unusual move that could signal a new front in the…

↯ Llamallama

https://huggingface.co/stevelikesrhino/gemma-4-31B-it-nvfp4-GGUF/blob/main/gemma4-improved.jinja Yall are more than welcome to try it out and provide feedback. In my own testing in Pi-coding-agent I no longer have the "forgot to close thin…

↯ Llama↯ Gemma↯ Gemma 4gemmallama

https://github.com/ggml-org/llama.cpp/pull/23269

↯ Llamallama

So as we know.. llama.cpp separates the vision or other multimedia from the main weights.

↯ Llama↯ Mistralmistralllama

Hello, I'm currently using Ollama / lm studio for things like code inference and proof reading emails, etc. Definitely not experienced in this space but looking to grow.

↯ Ollama↯ Llama↯ Vllm↯ Gemma↯ Gemma 4vllmollamagemma+3

PLEASE KEEP IN MIND BOTH OF MY CARDS ARE POWER LIMITED TO 150W (i hate noise) ------- Just wanted to share my current setup, that might help some users out there... services: llama-server: image: ghcr.io/ggml-org/llama.cpp:full-cuda12-b912…

↯ Llama↯ Qwen 3.6llama

Few days ago I posted about my DeepSeek V4 Pro at home - now time for an update. Yesterday I finally managed to run this model in ktransformers (sglang + kt-kernel).

↯ Llama↯ DeepSeek 4deepseekllama

I have a docker stack with a bunch of AI services and llama.cpp server is the brain. I've got a working vulkan yml snippet for llama.cpp but out of curiosity, I flipped it to ROCM (latest build) and did not see ANY performance improvement.

↯ Llamallama

This is follow up from previous post: https://www.reddit.com/r/LocalLLaMA/comments/1t5ageq/ There have been many improvements to the MTP pull request and the llama.cpp main branch, such as image support and various bug fixes. I recently ma…

↯ Llamallama

If anyone is looking for a good high-speed setup with ~190k context, this config has been working insanely well for me. I’m using my laptop as a server over Tailscale.

↯ Llama↯ Claude 4.6↯ Claude 4.6moellamaopus

We took a recently released Bonsai 1.7B ternary model from PrismML (https://github.com/PrismML-Eng/Bonsai-demo) and ran our agentic evolution search on it for 6 hours to optimize the Metal kernels. The search was fully autonomous.

↯ Llamallamaagentic

Model: Abiray-Qwen3.6-27B-NVFP4.gguf Specs: - Legion 7i Gen10 - NVIDIA GeForce RTX™ 5090 - Intel® Core™ Ultra 9 275HX × 24 - RAM 32.0 GiB llamacpp settings: ./build/bin/llama-server \ -m ~/.lmstudio/models/lmstudio-community/Qwen3.6-27B-GG…

↯ Llama↯ Qwen 3.6llama

https://huggingface.co/z-lab/gemma-4-31B-it-DFlash I guess we'll have to wait until this PR is merged before we can test it. https://github.com/ggml-org/llama.cpp/pull/22105

↯ Llama↯ Gemma↯ Gemma 4gemmallama

Hi, here you can find experimental llama.cpp support for DeepSeek v4, and here there is the GGUF you can use to run the inference with "just" (lol) 128GB of RAM. The model, even quantized at 2 bit, looks very solid in my limited testing, a…

↯ Llama↯ DeepSeek 4deepseekllamaqwen

The new dense model is great, but I’m trying to figure out how to increase PP and Token generation speed. I’m running Q8 quants across 3 7900xtx GPUs and I’m consistently only getting 18-20 t/s generation speed and ~650 t/s prompt processi…

↯ Llama↯ Qwen 3.6llamaqwen

There have been many TurboQuant implementations recently in llama.cpp, mlx, vllm, and sglang, but a lot of the discussion and code around them feels pretty noisy and looks to be AI-generated. I’m trying to understand which claims from the…

↯ Llama↯ Vllmvllmllama

I've been running 5-8 Claude Code sessions at a time and got tired of tab-switching to approve tool calls. So I built claudectl — a TUI that sits on top of all your sessions and lets a local LLM (ollama/llama.cpp) handle approvals for you.

↯ Ollama↯ Llamaollamallamaclaude-code

https://preview.redd.it/24uvk7o4sy2h1.png?width=1440&format=png&auto=webp&s=542570e3057b6f44c1e7e8d92130f575fb69cfa2 https://preview.redd.it/l4bbm7o4sy2h1.png?width=1440&format=png&auto=webp&s=3dc0edd978da23fecf81e86a269a06de643247d1 I was…

↯ Llamallama

https://preview.redd.it/sm4ysgdw1w2h1.png?width=1376&format=png&auto=webp&s=3705932403919814fbf2008a1cba189d17e0591e Thanks everyone for the advice on my previous post (24/7 Headless AI Server on Xiaomi 12 Pro (Snapdragon 8 Gen 1 + Ollama/…

↯ Ollama↯ Llama↯ Gemma 4ollamallama

This is for all with 12GB VRAM. Hi, I created a fork of llama.cpp with an experimental implementation of experts instead of layers.

↯ Llama↯ Qwen 3.6moellama

This has turned out to be useful to many of my friends so I thought I'd share here as well. I created a tool and documentation page for most major open-souce project's adherence to 'OpenAI compatibility' after seeing inconsistencies betwee…

↯ Llama↯ Vllmvllmllamaopenai+1

I thought it had horrible performance and was a nothingburger and had spent like an hour benchmarking it. Updated it yesterday and received a like 1.5-1.8x token boost.

↯ Llamallama

Someone suggested I give Continue (Vscode extension) a try. I've been using Roo / Zoo now and liking it but it is pretty tough on context and I was told continue has more control over it.

↯ Llama↯ Qwen 3.6continue-devmoellama+1

CPU is just a secondhand 10900x. Using 128k context, unquantized kv cache.

↯ Llama↯ Minimaxminimaxmoellama

Hi r/LocalLLaMA - I've been paying close attention to the edge AI ecosystem because it's an area where i see huge potential and where I truly believe AI will become more useful for day to day tasks. Around the gemma 4 release I was already…

↯ Llama↯ Gemma↯ Gemma 4gemmallama

Hey all, I have built myself a WebUI for configuring and managing llama-server sessions, and want to share the code and concept. Python and a bit of JS.

↯ Llamallama

Inspired by https://www.reddit.com/r/LocalLLaMA/comments/1tayu5t/stop_wasting_electricity/ I've decided to put my 5090 to test and see how do the curves look like for the device and whether there were any obvious sweet spots (apart from se…

↯ Llamallama

Hello there people. So I have noticed that people are pretty much ignoring Llama 3 plus 3.1, 3.2, and 3.3 these days.

↯ Llama↯ Fine Tuning↯ Llama 3.3fine-tuningllama

Bigger ubatch made gpt-oss-120b prompt processing much faster on my RTX 3090 I was tuning gpt-oss-120b-F16.gguf with llama.cpp on a 24 GB RTX 3090 and found that increasing the physical micro-batch size (-ub) can massively improve prompt p…

↯ Llamamoellama

I'm running opencode and llama-server locally. I have 32gb ram and 780m igpu.

↯ Llama↯ Qwen 3.6llama

Disclosure: I am the author of this evaluation SDK. I released an independent TurboQuant-compatible KV backend evaluation package for compressed-KV ABI testing, smoke tests, and partial attention decode experiments.

↯ Llamallama

I struggle to wrap my head around all this. My goal is local agent to solve low complexity tasks, in the same harness where I would use frontier models.

↯ Llama↯ Qwen 3.6llamaqwen

Hi everyone, I’m the maintainer of Box — a fork of Google’s AI Edge Gallery that I’ve been extending into a fully offline AI assistant for Android. Full disclosure: I built this project.

↯ Llamallama

I'm just looking for some advice on optimally setting up Qwen3.6 27B for OpenCode. The VRAM is a little bit scarce, but I ended up with this so far: llama-server --model models/Qwen3.6-27B-IQ4_XS.gguf \ --port 8080 \ --host 127.0.0.1 \ --t…

↯ Llama↯ Qwen 3.6moellama

Both llama.cpp and ik_llama.cpp now have FP4 support — but with different flavors worth knowing about. llama.cpp recently merged NVFP4 (Nvidia's block-scaled FP4, `GGML_TYPE_NVFP4 = 40`), with CUDA kernels landing in `mmq.cuh`, `mmvq.cu`,…

↯ Llamallama

MLX https://github.com/Blaizzy/mlx-vlm?tab=readme-ov-file#turboquant-kv-cache vLLM https://github.com/vllm-project/vllm/pull/38479 MLX & vLLM users, please share your experience with benchmarks(t/s). Adding llama.cpp Links related to Turbo…

↯ Llama↯ Vllmvllmllama

Here is my experience, I've been using LM Studio with RTX 5080 and 64GB RAM using Windows 11. I'm very happy with LM Studio except the speed.

↯ Llama↯ Gemma↯ Gemma 4gemmallamaqwen

Guard llama A guard llama is a llama that is used in farming to protect sheep, goats, hens or other livestock from canids such as coyotes, dingos, dogs, foxes and other predators.[1][2] In the past, a single gelded (castrated) male was rec…

↯ Llamallama

I built this because I see that grocery savings are achievable in NYC. People usually just go to the store they're used to going to, and it's rarely worth the effort of combing through card cashback, weekly coupons, CPG rebates.

↯ Llamallama

I've been using lm studio for a few months. I want to try hermes agents with Qwen 3.6 MoE, so I'm switching to llama.cpp and I don't understand well how the server slots -np and the context size -c interact.

↯ Llama↯ Qwen 3.6moellamaqwen

I'm building a local-first agent — a plain ReAct loop (think, pick a tool, observe, repeat) on a llama.cpp backend — and I want to be precise about a question that usually just gets answered with "it depends." It does depend. So let me spl…

↯ Llama↯ Qwen 3.6moellama

https://preview.redd.it/kx39ammxno2h1.jpg?width=1080&format=pjpg&auto=webp&s=d1a2d5b27920a5b61a50547a6e70a6378445cae4 SupraLabs released a new model! - Supra-50M Supra-50M is a compact 50M-parameter causal language model (BASE and INSTRUCT…

↯ Llamallama

Well I ordered a 3090 today. I plan on pairing it with a 3060 I have for 32gb combined VRAM.

↯ Llamallama

Qwen 3.6 27B Q8 on four Nvidia RTX A4000 (16GB each) with Llama.cpp and MTP enabled My setup is heterogenous, I originally acquired my server (Lenovo ThinkStation P3 Tower Gen 2) to run OpenShift/K8s clusters (because I work on that), and…

↯ Llama↯ Qwen 3.6llamaqwen

We've got great outputs for 27B via club 3090, but what about those of us who love the blazing speed of 35B on dual 3090s? I was getting 1500 p/p and 120 t/g with split layers, but MTP slowed it down to 80 t/g when I tested last week.

↯ Llama↯ Qwen 3.6llamaqwen

Been fine-tuning llama 3.1 8b with Qlora for a classification task using about 8k samples. I was getting bad eval results for a while and kept thinking something was wrong with my data.

↯ Llama↯ Fine Tuningfine-tuningllama

I got Qwen 3.6 35B-A3B and Gemma 4 26B-A4B running on a $200 secondhand machine (i7-6700 / GTX 1080 / 32 GB RAM) using llama.cpp (the TurboQuant/RotorQuant KV cache quantisation allows 128k context within the 8 GB VRAM). Results (Q4_K_M mo…

↯ Llama↯ Gemma↯ Qwen 3.6gemmamoellama+1

I mean; don't get me wrong, I love me some improvements and enhancements and it keeps on giving... and with MTP making its way to llama.cpp soon, a lot of you who aren't already running custom compiles are about to get a boost in inference…

↯ Llamallama

Hi, I'm trying to use MTP with llama.cpp, I built from source the mtp-pr, download an MTP model from huggingface https://huggingface.co/unsloth/Qwen3.6-27B-GGUF-MTP/resolve/main/Qwen3.6-27B-Q6_K.gguf But when I run the model I have an erro…

↯ Llama↯ Qwen 3.6llama

llama-server.exe --model "H:\gptmodel\AesSedai\MiMo-V2.5-GGUF\MiMo-V2.5-IQ3_S-00001-of-00004.gguf" --ctx-size 1048576 --threads 16 --host 127.0.0.1 --no-mmap --jinja --fit on --flash-attn on -sm layer --n-cpu-moe 0 --threads 16 --parallel…

↯ Llama↯ Minimaxminimaxmoellama

I have RX 7900 XTX, running Qwen3.6 27B Q4_K_XL. got 400ish pp and 30s tps.

↯ Llama↯ Qwen 3.6llama

I gave some math problems to Qwen 3.5 27B and Qwen 3.6 27B and they got all of them right, pretty smart models I would say, but very slow and electricity consuming, they took like 5 mins with my GPU at 120 W to solve a problem. The MoE mod…

↯ Llama↯ Qwen 3.6moellamaqwen

Did some quick tests after building llama.cpp with ROCm 6.4.2 and latest Vulkan for my 6900 XT gemma4 E2B Q4_K ubatch ROCm pp512 Vulkan pp512 ROCm tg128 Vulkan tg128 32 1536.60 1423.49 151.92 174.59 64 1590.65 1930.60 151.41 173.76 128 265…

↯ Llama↯ Gemma↯ Qwen 3.5gemmallamaqwen

I have a 4 x R9700 system on Threadripper pro, but I have never been happy with the performance of my GPUs in vLLM. I have started benchmarking any new model I try out with llama-benchy so that I can get a better idea of how models of diff…

↯ Llama↯ Vllm↯ Qwen 3.6vllmllama

Hey all: I am trying to set up claude code to work with llama.cpp, I am using the Qwen3.6-35B-A3B. I usually use claude code + ZLM subscription i got lucky with $30 yearly - the set up is very simple with their automated script, but for th…

↯ Llama↯ Qwen 3.6llamaclaude-code

What would you say is the minimum amount of tokens per second you would tolerate for your local agent workflows? I have been trying pi.dev connected to a llama.cpp instance running Qwen3.6-27B-Q6_K_L with 200K context running on an RTX A60…

↯ Llama↯ Qwen 3.6llamaanthropicclaude-code

↯ Llamallama

↯ Llama↯ Vllm↯ Fine Tuningvllmfine-tuningllama+1

I tried local model couple weeks ago. At the beginning, I tried Ollama, but reddit says better to switch to llama.ccp.

↯ Ollama↯ Llama↯ Qwen 3.6ollamamoellama+2

I've seen a ton of PR, and a bunch of failed PR with some interesting additions. I was wondering what other people's commands are looking like now, what they are running for llama.cpp I'm still running: CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6 l…

↯ Llama↯ Qwen 3.5llama

I run it on 2xRTX 3090. This is part of my llama-server presets file: [Qwen3.5-27B-bartowski] load-on-startup = true alias = Qwen3.5-27B-bartowski hf = bartowski/Qwen_Qwen3.5-27B-GGUF:Q8_0 hfd = bartowski/Qwen_Qwen3.5-2B-GGUF:Q8_0 draft-mi…

↯ Llama↯ Qwen 3.5llamamcp

Adaptive KV Streaming for llama.cpp This branch adds an experimental, block-granular KV cache streaming path to the CUDA llama-server. It is intended for running long contexts when model weights leave too little VRAM for the complete KV ca…

↯ Llama↯ Qwen 3.8llamaqwen

K.A.S — Kasra's Agentic Shell. Run open models locally — on Apple Silicon (MLX) or NVIDIA (llama.cpp/GGUF) — behind an Anthropic Messages-compatible server, driven by an agentic TUI.

↯ Llamallamaanthropicagentic

I've been running Qwen3.6-35B-A3B locally on llama.cpp and noticed that prompt processing throughput gets too low with MTP. I got nerd-sniped.

↯ Llama↯ Qwen 3.6moellama

Selora AI Local is an open-source, Qwen-based model for Home Assistant. Specs: Qwen3 1.7B base model (Q6 quantized~1.6GB) Four Home Assistant-specific LoRA adapters: - Answers - Clarifications - Automations - Commands ~3.5 GB total downloa…

↯ Llama↯ Qwen 3llamaqwen

llama.cpp Manifesto / ggml / ops LLM inference in C/C++ Recent API changes Changelog for libllama API Changelog for llama-server REST API Hot topics Hugging Face cache migration: models downloaded with -hf are now stored in the standard Hu…

↯ Llamallama

Appears thay have been cooking and we might see a fix soon released for crashes on split mode tensor Multi-gpu folks keep watch - ( In my tests SM Tensor has a ~35% uplift in TG over Layer but ofc crashes every 90-120 minutes due to vram e…

↯ Llamallama

It does not like anything I have tried, including what works with llama-server. is it not built to work with speculative decoding?

↯ Llamallama

I’m using vim lmao with a custom made plugin for completing text, so I was curious what yall use. Llama-server seems like a sensible default but it seems limited

↯ Llamallama

Hi everyone, I’ve just published the first public release of llampart 1.0.0: https://github.com/mchowy-troll/llampart llampart is a standalone local web UI designed to work with `llama-server`. It started from the `llama-ui` work in the `l…

↯ Llamallamamcp

I'm running the 122-billion Qwen 3.5, specifically Qwen3.5-122B-A10B-Q5_K_M, on DGX Spark (128 GB contiguous memory). I'm (very!) impressed with the general knowledge output.

↯ Llama↯ Qwen 3.5llamaqwen

Simple question: Has anyone run two or more of either of these on Ubuntu ? Intel Arc Pro B70 (32 GB) Intel Arc Pro B65 (32 GB) Running llama or vLLM etc., Any thoughts

↯ Llama↯ Vllmvllmllama

Llama-server does not accept audio input (or video for that matter) with Nemotron 3 nano omni (unsloth). I’m on a recent build of llamacpp and I redownloaded Nemotron, and I have the mmproj loaded too.

↯ Llama↯ Gemma 4llama

Hello, After almost two years of on-and-off development, 5 complete architectural rewrites, and hitting a few brick walls, I’m finally open-sourcing a project I built to scratch my own privacy-paranoia itch: Nexidion. GitHub Repo: https://…

↯ Ollama↯ Llamaollamallamaopenai

TL;DR I spent a few crazy evenings this past week seeing if I could get Gemma4 running with proper turbo quant and rotating KV cache support. The answer was yes, and I'm now able to run Gemma4 26b on my MacBook Air M5 at 128k context with…

↯ Llama↯ Gemma 4moellama

I’m using llama-swap with llama.cpp. I mainly use opencode + pi.dev and I’m seeing frequent massive prompt reprocessing / prefills even tho the prompts are very similar between requests.

↯ Llamallama

Hi, i just wanted to share what im playing with for last couple weaks. I built my own AI harness: TinyHarness My main goal was low memory footprint, it is not written in Typescript/Javascript/Python, leaving as much memory as possible for…

↯ Ollama↯ Llama↯ Vllmvllmollamallama

About Press Copyright Contact us Creators Advertise Developers Terms Privacy Policy & Safety How YouTube works Test new features NFL Sunday Ticket © 2026 Google LLC

↯ Llamallama

Long post, but hopefully helps somebody. Llama-cpp vulkan server running single AMD R9700.

↯ Copilot↯ Llama↯ Qwen 3.6copilotllamachatgpt

How is this dual setup's performance? Is it difficult to set-up everything with for example llama.cpp?

↯ Llama↯ Qwen 3.6llamaqwen

Just a quick note that I got a nice result using am17an's MTP branch of llama.cpp on v100 32GB SXM module using one of those pcie card adapters. Pulled and built in one shot, and llama-server ran without a hitch.

↯ Copilot↯ Llama↯ Qwen 3.6copilotllamaqwen

Very quick initial test of Gemma 4 new MTP model via Ollama (llama.cpp doesnt support yet) https://blog.google/innovation-and-ai/technology/developers-tools/multi-token-prediction-gemma-4/ Running in Open Webui to view token/s output and I…

↯ Ollama↯ Llama↯ Gemma↯ Qwen 3.6ollamagemmallama

Qwen3.6 35B A3B UD IQ4_NL_XL. 512k context tokens for 4 parallel processing, key cache quantized to Q_8 and value cache quantized to Q_4.

↯ Llama↯ Qwen 3.6llama

Mistral Medium 3.5 128B with 4x3080 20GB with layer split: CUDA_VISIBLE_DEVICES=0,1,2,3 ./build/bin/llama-bench --model /data/huggingface/Mistral-Medium-3.5-GGUF/Mistral-Medium-3.5-128B-IQ4_XS-00001-of-00003. gguf -ngl 99 -d 0,16384 -fa 1…

↯ Llama↯ Mistral↯ Qwen 3.5mistralllamaqwen

Been building this for a while and finally cleaned it up enough to share. voice-agents-from-scratch is a numbered, chapter-by-chapter repo that walks the full real-time pipeline: Microphone capture Whisper for STT Local GGUF LLM (via llama…

↯ Llamallama

Hello everyone. Over the last couple months I have been assembling my local AI setup for personal use, and I thought to write a post here, firstly to collect some thoughts on the whole concept, and secondly to perhaps gather some feedback.

↯ Llama↯ Qwen 3.6llamaqwen

I'm testing running local LLMs on a gaming mini PC (AMD 7840HS, 32 GB RAM) paired with an eGPU (Radeon 9060XT with 16 GB VRAM). Since I'm not very familiar with using llama.cpp, I kept getting unsatisfactory results, but with the recent Ge…

↯ Llama↯ Gemma↯ Gemma 4gemmallama

I wanted to see how much of my coding-agent workflow I could move local instead of paying for hosted tools forever. There was another push: Anthropic's own April 23 postmortem confirmed product-layer regressions through March/April.

↯ Llama↯ Qwen 3.6llamaanthropicclaude-code

TLDR : Should an RTX 3090 + T4 be faster than a P40 + T4 for OpenCode with Qwen3.6 35B A3B ? --- Hi, Nowadays, I have an architecture running : A Tesla P40 w/ 24GB VRAM A Tesla T4 w/ 16GB VRAM I mainly use this setup to run models like GPT…

↯ Llama↯ Qwen 3.6llama

↯ Glm↯ Llama↯ GLM 5.1glmmoellama

↯ Llamallama

Advertisement Alibaba’s Qwen family captures over 50% of global open-source downloads, report finds Qwen hits nearly 1 billion cumulative downloads, far surpassing rivals like Meta Platforms’ Llama and DeepSeek, researchers say 2-MIN READ2…

↯ Llamadeepseekllamaqwen

my current build is just a 16GB 5060Ti running on a 3800X with 32GB DDR4. not really anything special, but I only really use it right now for Qwen3-VL-8B-Instruct at INT8 to do handwriting transcription (and it works great for that). someo…

↯ Llama↯ Vllm↯ Qwen 3.5vllmllama

Intel Releases OpenVINO 2026.1 With Backend For Llama.cpp, New Hardware Support Intel's OpenVINO toolkit for optimizing and deploying AI inferencing across their range of hardware platforms is out with its newest quarterly feature update.…

↯ Llamallama

Benchmarking Local LLM Servers: llama.cpp, llamafile, LM Studio, and Ollama Benchmarking Local LLM Servers evaluates llama.cpp, llamafile, LM Studio, and Ollama across Mac, Linux, and Steam Deck. The study reveals build flags and configura…

↯ Ollama↯ Llamaollamallama

Self-hosted deploy console — your own Vercel.

↯ Llamallama

Open models behind one OpenAI-compatible endpoint. No account yet?

↯ Llama↯ Gemmagemmallamaqwen+1

I wanted to run AI from inside the JVM. I started out with the standard REST sidecar, ripped that out to use Project Panama (Foreign Function & Memory API) in the new JDK versions to interface directly with llama.cpp.

↯ Llamallama

How do people measure power usage of GPUs at large (32x) self-hosted setups or small multi-rack setups? I've seen some PDUs which collect and transmit data, but I'm unsure of the processes and if/how people do this on small builds.

↯ Llama↯ Mistral↯ Gemmamistralgemmallama

Second Brain 🧠 Your Invisible AI Interview Assistant English | Português | Español | Français | Italiano Second Brain is a 100% free, invisible desktop assistant built to help professionals worldwide ace their job interviews. It runs quiet…

↯ Copilot↯ Llamacopilotllama

Claude Code like agentic workflow ai too costly for me.Any LLM can I run with VSCode at the below setup? 16ram Intel core i7 h processor 13gen 512gb NVMe SSD I want to run the ai as local agentic workflow with Vscode.I want use LLAMA agent…

↯ Llama↯ Qwen 3.5↯ Qwen 3.5↯ Qwen 3.5↯ Qwen 3.5↯ Qwen 3.5↯ Qwen 3.5↯ Qwen 3.5llamaagenticclaude-code

The Winamp Skin Museum really whips the llama's ass Over 65,000 skins to browse! In the late nineties and early noughties, no video game forum was complete without a 'post your desktop' thread, and no desktop screenshot was complete withou…

↯ Llamallama

llama.cpp now has an official website: llama.app Our goal is to make local AI accessible to everyone, and improving the user experience is a big part of that. On the new landing page you’ll find a single-line cross-platform installer.

↯ Llamallama

I'm using llama.cpp, and I've tried Bartowski's and my own quants. When using Qwen3.5-122B or Qwen3.6-27B, I'm seeing really low draft acceptance in chats with interleaved code snippets (chatting with the LLM about programming / a code pro…

↯ Llama↯ Qwen 3.6↯ Qwen 3.6↯ Qwen 3.6↯ Qwen 3.6↯ Qwen 3.6↯ Qwen 3.6↯ Qwen 3.6↯ Qwen 3.6↯ Qwen 3.6↯ Qwen 3.6llama

TL;DR. Shard is a drop-in HuggingFace Cache that makes Llama-3.1-8B's KV memory about 10× smaller at 8K context (11× at 32K) without measurable hits to NIAH or LongBench.

↯ Llamallama

I've fine-tuned Qwen 3.5 0.8B on the dataset provided by Pangram with their EditLens paper. It's available via a Chrome extension; you can just click selected text and it's going to give you the probability distribution of how likely it is…

↯ Llama↯ Fine Tuning↯ Gemmagemmafine-tuningllama+1

I just published a repo called MCP from Scratch that teaches the Model Context Protocol by building it step by step in plain Node.js. Most of the repo is about understanding MCP itself, but the later modules may be relevant here: I added a…

↯ Llama↯ Model Context Protocolmodel-context-protocolllamamcp

I've burned a week trying to customize my agent manually - building my own front end - but I've gotten to the point where I'm just exhausted and willing to try a harness, but need the right one. I read posts all the time, but I have a spec…

↯ Llama↯ Qwen 3.6llamaqwen

I'm still in my learning process and so far I've been able to make satisfying use of my setup (4070 with 12GB VRAM + 32GB RAM and iGPU for my GUI). I've been able to run both Gemma4 26B and Qwen 3.6 35B MoEs up to high quants with large co…

↯ Llama↯ Qwen 3.6llamaqwen

As some other fellow lllmers I've discovered few days ago that the amazing llama.cpp project has just added native tools functionalities into the server. After having enabled the relative options into llama-server and played a bit with the…

↯ Llamallamarag

Everyone remembers that sneaky download of Gemini Nano earlier this month? and if you talk to it, it will happily tell you it’s a Gemma.

↯ Llama↯ Vllm↯ Gemma↯ Gemma 4vllmgemmallama+1

Run production LLMs on your own hardware A Kubernetes operator for self-hosted LLM inference. vLLM, llama.cpp, TGI, NVIDIA, Apple Silicon.

↯ Llama↯ Vllmvllmoperatorllama

Following on from club-5060ti, I’ve been doing some testing with my desktop AMD GPU and wanted to make a similar repo for 16GB Radeon cards. Repo: https://github.com/5p00kyy/club-rdna16 Pages/results: https://5p00kyy.github.io/club-rdna16/…

↯ Llama↯ Qwen 3.6llama

Introducing WebGPU support for llama.cpp

↯ Llamallama

Title. I've got a llama.cpp server running a model being accessed across a number of scripts, and some of them are easier for the model than others, and those easier ones are also latency dependent.

↯ Llamallama

I posted earlier about RTX 5060 Ti local LLM testing, and I have cleaned the repo up quite a bit since then. The project is now a more structured benchmark/recipe repo rather than scattered notes.

↯ Llama↯ Vllm↯ Qwen 3.6vllmllama

Hi all, Not sure many people are aware so wanted to give a word about Fara-1.5 release. => this release will likely be the big sister of Fara-7B and built on top of Qwen3.5 Actual Fara-7B performs not bad at all but actually requires a pro…

↯ Llama↯ Vllm↯ Qwen 3.6vllmllama

I'd love to hear from developers who use big context windows if they notice a difference? Obviously I would love to cut the KV cache VRAM requirement in half, but I'm worried about quality especially when we enter into 50k+ context territo…

↯ Llama↯ Qwen 3.6moellamaqwen

Just an idea and a prototype (made by Qwen3.6-27B-UD-Q6_K_XL via OpenCode) for allowing users to add custom sampling logic to llama-server without having to maintain their own entire fork and without having to make a wrapper that reimpleme…

↯ Llama↯ Qwen 3.6llama

It is suppose to be 2-4x faster but i am only getting 6TK/s on Gemma4-31B . What am i doing wrong?

↯ Llama↯ Gemma↯ Gemma 4gemmallama

TL;DR: I got TBQ4 KV cache + MTP working on AMD ROCm for RX 7900 XTX / RDNA3 / gfx1100 in llama.cpp. Main win: 64k context fits on 24 GB VRAM and remains usable.

↯ Llama↯ Qwen 3.6llama

Implemented Multi-Token Prediction for QWEN on LLaMA.cpp with TurboQuant. +40% performance!

↯ Llama↯ Qwen 3.6llamaqwen

I would like to dedicate a budget of about 500 euros to upgrade my workstation and run inference on the qwen 3.6 27b and gemma 4 31b models. I currently have an RTX 5060Ti 16GB.

↯ Copilot↯ Llama↯ Vllm↯ Gemma↯ Qwen 3.6vllmgemmacopilot+2

An unofficial Pokedex for AI models. Compare GPT, Claude, Gemini, Llama, DeepSeek and more, with types, evolutions, base stats, and simulated token-burning battles.

↯ Llamadeepseekllamagemini

Hi, I recently tried to get llama.cpp with SYCL running on an Arrow Lake system but gave up halfway through since Vulkan is just way easier to set up. But, the pp/tg I'm getting on Vulkan w/ Arc 130T is disgustingly bad - 100 tokens/s for…

↯ Llama↯ Gemma↯ Gemma 4gemmallama

I'm running qwen3.6-35b with llama.cpp connected to openwebui. And I noticed the model fails the number guessing game test on openwebui while it works perfectly with the llama.cpp web ui.

↯ Llama↯ Qwen 3.6llama

Let me first say I am not doing anything with parallelism so these benchmarks and tests are not for you. That said if your hobbyist like me that is left wondering if can I use the GPUs my other PCs then I have some answers and but I'm stil…

↯ Llamallama

Did anyone evaluate HIPfire for long context sizes (100k+) and quality, for Strix Halo? It apparently promises large performance increase over llama.cpp and the like.

↯ Llamallama

Looking for suggestions. Current setup llama.cpp and ran qwen 3.5 397b 256k context.

↯ Llama↯ Qwen 3.6llamaqwen

specs : core i5 14400F 32gb ram d4 3200mhz rtx 4060 current speeds 30tps in output 500 tps in prefill command i currently use .\llama-server.exe ` >> -m "H:\model\unsloth\Qwen3.6-35B-A3B-GGUF\Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf" ` >> --host 0.…

↯ Llama↯ Sonnet 4.5moellamasonnet

Can llama.cpp run MTP for this model?

↯ Llama↯ Qwen 3.5llamaqwen

Hey folks, looking for advice before I delete or keep a huge model file. I’m testing local coding/agentic workflows on an RTX 5080 16GB + 96GB RAM.

↯ Llama↯ Qwen 3.6moellamaagentic

Noob here, Running Qwen3.6 35B A3B in LM Studio on a 3080 10GB + Ryzen 5 3600 on Windows 10. Tried some unsloth quants with identical settings (GPU offload 40, MoE layers to CPU 40, context 8192, flash attention on).

↯ Llama↯ Qwen 3.6moellama

Got 47 repos that start with 'just playing with Claude' or 'testing Llama 4 on'. Every single one dead after three commits.

↯ Llama↯ Llama 4llama

Hello everyone. I just thought of something that seems so obvious but from what I’ve been able to find it doesn’t seem like anyone has done it or at least not openly disclosed it if they have.

↯ Llamallama

I was curious to understand what people are doing for this use case to get the best trade-off of convenience and performance. Private backhaul on the 10GbE?

↯ Llamallama

Hi all I'm running Qwen3.6-27B-UD-Q6_K_XL.gguf using llama swap and llama-server with these parameters (actually stolen for some posts on this subreddit.) llama-server \ -m /models/Qwen3.6-27B/Qwen3.6-27B-UD-Q6_K_XL.gguf \ --mmproj /models…

↯ Llama↯ Qwen 3.6llama

It looks like finally we have it! Time to test!!!

↯ Llamallama

Tutorial from the Google guy, I use very similar setup (llama.cpp instead of lmstudio)

↯ Llama↯ Gemma↯ Gemma 4gemmallama

At this moment, the models such as Qwen 3.6 35b/27b crush the competition, yet I can't help, but notice this pattern. While the local RP scene is abundant with the Western model tunes: LLaMA, Mistral (all sizes), Nemo and more recently Gem…

↯ Llama↯ Mistral↯ Gemma↯ Qwen 3.6mistralgemmallama+1

I've heard mostly bad opinions about multiple slots with llama.cpp (--parallel > 1). I guess comparing to vLLM it might be worse at this, but I recently tried vLLM on 4 slots and it indeed improved the overall speed significantly (150-170t…

↯ Llama↯ Vllmvllmllama

I'm building a small text-based game where the gameplay loop is "talk an NPC into revealing a secret." It's basically a 20+ turn roleplay stress test: the model needs to stay in character, remember what the player said earlier, and refuse…

↯ Tool Use↯ Llama↯ Qwen 3.5tool-usellamarag

Somehow I cannot get KV resume for my Qwen3.5 model with lama-server: Save/restore works for tokens, but KV cache is never reused — is this expected? How to enable real resume?

↯ Llama↯ Qwen 3.5llama

Can someone who owns a R9700 (single GPU enough) to add a llama-bench output with Qwen3.6-35B-A3B Q5_K_P here in the thread? Other benchmarks are also welcome :) I just want to see the t/s and compare it with my local solution, because I m…

↯ Llama↯ Qwen 3.6llama

↯ Llamallamaagentic

↯ Llama↯ Gemma 4llama

↯ Llama↯ Qwen 3.6llama

Complete llama.cpp tutorial for 2026. Install, compile with CUDA/Metal, run GGUF models, tune all inference flags, use the API server, speculative decoding, and benchmark your hardware.

↯ Llamallama

I cannot find a definive answer. I think the following should be able to do FIM: Qwen 2.5 coder Qwen 3 coder Qwen 3-2507 instruct Qwen 3.5 Qwen 3.6 What I verified: Qwen3-32B: no Qwen3-4B-Instruct-2507: yes Qwen3.5-27B: yes Qwen3.6-35B-A3B…

↯ Llamallamaqwen

Has anyone been able to solve or mitigate context checkpoints being erased during single user inference, specifically when function calling is part of the chat history? I've been using Qwen 3.5 35B A3B for some time (now using 3.6), tested…

↯ Llama↯ Function Calling↯ Qwen 3.6function-callingllamaqwen

I noticed that groups have changed to Matrix , to allow concurrent models. Currently i use llama-swap for my models and an individual instance of llama-server for embedding and reranking all for Openweb UI.

↯ Llamallama

Ryzen AI MAX+ 395, Bosgame M5, 128GB LPDDR5x. Proxmox VE 9.1 LXC containers with GPU passthrough.

↯ Llama↯ Minimax↯ Gemma↯ Qwen 3.5minimaxgemmallama

Hey! I was wondering if anyone of you have used Qwen3.5-27B-NVFP4-GGUF on RTX5090 on llama.cpp?

↯ Llama↯ Qwen 3.5llama

Right now I have 3 GPUs, 5060 Ti 16G, 2 x 4060 Ti 16G, and may get a used 3090 24G that I found. I could build a janky open rack system using M.2 and PCI risers with a 1600W PSU or try something like putting 2 GPUs in 2 systems using the f…

↯ Llamallama

Pure C# pipeline Tokenizer, sampler, scheduler, kernels — all C#. No Python, no foreign runtime, no llama.cpp wrapper.

↯ Llamallama

Hi, my current system hardware RTX 3090 24GB VRAM & Sysrem RAM 64GB using windows 11 been playing around with hermes agent and local llm (Qwopus3.5-27B-v3-GGUF & gemma-4-26B-A4B-it-GGUF) when i try asking the hermes agent to do a task with…

↯ Llama↯ Gemma↯ Gemma 4gemmallamaqwen

I bought this 7900XTX for 905 euro in Spain, and wondering if can I combine them together to run Qwen 3.5 27B for example ? Using a MSI B650 Gaming Plus Wifi and 64gb DDR5 6400mt/s

↯ Llama↯ Qwen 3.5llamaqwen

using z790 prime p d4 with 128gb ddr4 3200mhz ram. 1x3090 in main PCIe5 16x slot and 2x3090 in chipset PCIe4 4x slots.

↯ Llama↯ Qwen 3.5llama

Frontier open-weight language models increasingly use Mixture-of-Experts (MoE) architectures to expand model capacity while activating only a small subset of experts per token. Local inference must nevertheless keep the complete expert poo…

↯ Llamamoellama

Making a vintage LLM from scratch; Take #2 Three and a half months ago I wrote about making my own vintage LLM from scratch, trained only on texts written before the year 1900. That project ended with a Llama-340M params base model that co…

↯ Llamallama

Eris A local-first agent in a single Rust binary: your Markdown vault as memory, grammar-enforced tool calls on llama.cpp, and nothing leaves your machine unless you say so. Eris runs a local LLM as a personal agent over a plain-Markdown v…

↯ Llamallama

Long story short, about a year ago, in spite of everybody bashing gpt-oss for broken tool calling and refusals, I thought there's something there worth exploring. Model hit a sweet spot for me in that it was the first time I could run full…

↯ Llama↯ Vllmvllmllamaopenai

Hi HN — I built VelocityNote, a compact, Markdown cross-platform notebook. The desktop app is under 100 MB.

↯ Llamallama

GenOffice (local-LLM fork) A free, open-source AI Office suite — this fork drops the cloud-account requirement and talks to any OpenAI-compatible endpoint instead: a local server (Ollama, LM Studio, vLLM, llama.cpp server, text-generation-…

↯ Ollama↯ Llama↯ Vllmvllmollamallama+1

Ayman Aboskhroun · Accountant I’ve been using Whizi for my accounting work at the office for the past year, and the results have been unlike anything else. Whizi · The multi-model AI workspace GPT, Claude, Gemini, Llama, and top image-gene…

↯ Llamallamagemini

So i had been building a side project screenmind ,sort of local ai desktop assistant, and hit a confusing multimodel failure... texts worked,but vision and image failed no crash just returning <unsused49>.

↯ Llama↯ Gemma↯ Gemma 4gemmallama

Ferrox: Building a Rust Inference Engine That Matches llama.cpp I’ve spent the last few days building Ferrox, a pure-Rust inference engine for running open LLMs locally — dense models and Mixture-of-Experts, on CPU, Apple Metal, or CUDA. N…

↯ Llamallama

homebench **Benchmark the local LLMs you already have — speed, memory, and quality — as a live terminal leaderboard. homebench is a single-command TUI that discovers the models installed in your local runner (Ollama, LM Studio, llama.cpp,…

↯ Ollama↯ Llamaollamallama

I'm a mechatronics designer with a background in control systems, robotics, PCB design, and embedded hardware. I design physical systems: motors, sensors, microcontrollers, and real-time control loops.

↯ Glm↯ Llamamixture-of-expertsglmmoe+1

There is a setting in LM Studio and llama.cpp that can make a local model generate 1.5 to 2.5 times faster without changing a single word of its output. Not a lower quant, not a smaller model, the same model producing mathematically identi…

↯ Llamallama

tiny-llama A minimal LLaMA-style inference engine for MiniCPM5-1B, built purely in Rust. I built this project to study the fundamentals of large language models.

↯ Llamallama

could not extract summary

↯ Llamallama

Darby is an open source Hugo docs theme with the polish of the paid docs platforms: clean typography, dark mode, full-text search, code blocks with copy and filename tabs, callouts, tabs, beautifully rendered mermaid diagrams, auto sidebar…

↯ Llamallama

Local-first · Private · Open source One native home forevery local model on your Mac Every text, image, and speech model already on your Mac — pulled through Ollama, cached by Hugging Face, or dropped in by hand — discovered and run in one…

↯ Ollama↯ Llamaollamallama

Open models at $0 rate 118+ open-source models hosted on NVIDIA NIM — Llama 3.1 70B, Mixtral, Nemotron, CodeLlama — bill at our open-model rate ($0 input, $0 output) once your account is activated with the $5 starter pack. InferAll is an A…

↯ Llamallamaopenaianthropic

A lean, fully-tested LLM inference server built on llama.cpp — designed for the hardware you already have: shared vCPUs, free tiers, 2-core ARM boxes. Reame is not the first inference server.

↯ Llamallama

Hey HN! We were part of YC S24, ended up pivoting around.

↯ Llamallamaagentic

Ever thought of running a local Large Language Model (LLM) on a vintage PC running DOS? Now you can!

↯ Llamallama

EdgeSync-LLM — KV Fragment Engine for Local LLMs A engine-agnostic KV cache fragment system for on-device LLM inference. Designed for ARM64 Android (Cortex-A55/A78), portable to any platform running llama.cpp, MLC-LLM, or ONNX Runtime.

↯ Llamallama

I've been running Qwen3.6-35B-A3B locally on llama.cpp and noticed that prompt processing throughput gets too low with MTP. I got nerd-sniped.

↯ Llama↯ Qwen 3.6moellama

I got tired of sending every text I translate to Google/DeepL. Even with all the opt-out options and privacy policies, it never felt right especially for some work documents, personal writing, or anything sensitive.

↯ Ollama↯ Llama↯ Gemma↯ Gemma 4ollamagemmallama

gateway online|running 3 · peer 2|req/s 0.87 LOCAL INFERENCE CONTROL PLANE One control plane for local inference. Monitor models, requests, API keys, routing rules, and proxy metrics from one dashboard for llama-swap and compatible upstrea…

↯ Llamallama

Back in 2022 and 2023 there were two big branches of machine learning happening at Meta1. The LLM work that led to Llama was a clean, smooth stack of repeated Transformer modules; the recommendation systems graphs were, by contrast, terrif…

↯ Llamallama

Llama.cpp (LLaMA C++) allows you to run efficient Large Language Model Inference in pure C/C++. Download llama.cpp for Windows, Linux and Mac.

↯ Llamallama

Hi folks, I found this setup on consummer hardware that seems to have great results on local hardware. - qwen 3.6 q6 - 450 K context using turboquant turbo3 mode llama.cpp fork - multimodal support This AI generated blog article is a kind…

↯ Llama↯ Qwen 3.6llamaqwen

Apples® to Apples®: MLX vs llama.cpp for Gemma 4 12B on an M1 16GB A matched-quant MLX-vs-raw-llama.cpp benchmark for Gemma 4 12B on one M1 16GB - decode is a tie, both pinned at the bandwidth wall. The cost that differs is startup and CPU…

↯ Llama↯ Gemma↯ Gemma 4gemmallama

mochallamaA local, tool-calling LLM inside your JVM The only in-process, tool-calling local LLM for the JVM — Spring-first, OpenAI-compatible, llama.cpp-backed via Project Panama FFM. No JNI, no daemon, no native-install dance.

↯ Llamatool-callingllamaopenai

llama.cpp VRAM estimator for normal people. Assumes single GPU, all layers offloaded.

↯ Llamallama

gemma-4-12B-it-GGUF Recommended way to run this model: llama-server -hf ggml-org/gemma-4-12B-it-GGUF Then, access http://localhost:8080

↯ Llama↯ Gemma↯ Gemma 4gemmallama

Hardware accelerated local LLM inference for Go with llama.cpp integration.

↯ Copilot↯ Llamacopilotllama

I built thaw because forking an LLM agent is absurdly wasteful today. When an agent explores N branches — RL rollouts, best-of-N, parallel coding attempts — each branch re-runs prefill over the same shared context.

↯ Llamallama

Context: I am testing and running local LLM on Linux for some months, first with llama.cpp and now with vLLM for better concurrent capabilities. I use llama-swap in front of either vLLM or llama.cpp in order to have thinking and non-thinki…

↯ Llama↯ Vllmvllmllama

Hi all, I'm somewhat new to the scene (been lurking for maybe 4-5 months now), but i think I have all the basics figured out. My setup: 9800x3d with 64GB of RAM, 6900xt with 16GB VRAM.

↯ Llama↯ Qwen 3.6↯ Qwen 3.6llamachatgptmcp

Best I tried so far is https://huggingface.co/nsparks/DeepSeek-V4-Flash-FP4-FP8-GGUF with the custom llama.cpp fork, but it suffers from low quality and random incoherent output. VLLM wouldn't support anything other than H100s for DS4.

↯ Llama↯ Vllm↯ DeepSeek 4↯ DeepSeek 4vllmdeepseekllama

Running llama.cpp on a Mac Pro 6,1 with Dual FirePro D700s on Ubuntu A D700-specific guide to running llama.cpp with Vulkan on the 2013 Mac Pro: dual 6 GB FirePro cards, Ubuntu, RADV, full GPU offload, cooling, and the traps that make old…

↯ Llamallama

Hi Reddit, I am planning on running Qwen 3.6 27b NVFP4 via vLLM on my 5090 but was wondering if something like 35b a3b at Q8 on Llama would produce better results for agentic coding and utilize the system memory. My research says no but if…

↯ Llama↯ Vllm↯ Qwen 3.6vllmllamaqwen+1

I've been using Qwen3.5 122B A10B (Q3_K_XL) a lot lately for coding, and it's been pretty incredible overall like it feels not far off from frontier-level for most tasks -- but I've been noticing that usually once I hit around 75-80k conte…

↯ Llama↯ Qwen 3.5llama

I'm usually not posting about Harbor releases out of the respect for the community here, but I think v0.4.19 might save a lot of people some time. Harbor can now launch your local agentic coding tools with local inference backends.

↯ Llama↯ Vllm↯ Qwen 3.5vllmllamacodex+1

Wondering what’s the best coding model that can fit on a RTX 3060 (12GB). Has anyone been able to do something useful with it?

↯ Llama↯ Vllmvllmllama

I'm running Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf on 12 GB VRAM and 32 GB RAM via the TurboQuant variant of llama.cpp. I increased the --n-cpu-moe value from 8 to 30, and my inference rate doubled!

↯ Llama↯ Qwen 3.6moellama

My use-cases will be to test open-weight LLMs and work on harnesses, inference systems and possibly other non-ML workflows (CS-related) in the future. Fine-tuning would not be something I do locally because I can rent a B200 from RunPod fo…

↯ Llama↯ Vllm↯ Minimax↯ Fine Tuning↯ DeepSeek 4minimaxvllmfine-tuning+2

I'm on Macbook M5 Max with 128GB RAM Running a test in openwebui using llama-server (llama.cpp): unsloth/Qwen3.6-27B-UD-Q6_K_XL.gguf (non MTP): 19tps unsloth/Qwen3.6-27B-UD-Q6_K_XL.gguf (MTP): 22.3tps So nothing like the massive improvemen…

↯ Llama↯ Qwen 3.6llama

could not extract summary

↯ Llamallama

Hey everyone, For people building agents with LangGraph, CrewAI, AutoGen, OpenAI Agents SDK, Claude MCP/SDK, Google ADK, or LlamaIndex — how are you managing LLM API costs? Agent workflows can get expensive fast because of: tool calls retr…

↯ Llamadeepseekllamarag+5

Finally got Llama 3 70B running locally and wired up function calling so it can search the web. First tried self-hosting SearXNG, but the results are pretty messy.

↯ Llama↯ Function Callingfunction-callingllama

Long story short, I am running Qwen3.5-35B-A3B (GGUF format) and other models on MacOS and getting around 1500 tokens/sec for prompt processing and around 35-50 tokens per second for prompt processing. I'm using the latest version of llama…

↯ Llama↯ Qwen 3.5llama

I posted recently about EvalShift, an OSS CLI for regression-testing LLM model changes. A few people pointed out that for LocalLLaMA, the more interesting use case may be quantization regression: Q8 -> Q4_K_M Same base model, same prompts,…

↯ Ollama↯ Llama↯ Vllmvllmollamallama

# ternative Inference engine for ternary-weight LLMs with runtime LoRA — the llama.cpp of BitNet models. Loads a BitNet I2_S base GGUF + a separate LoRA adapter GGUF, merges them at full F32 precision, and serves the result via an OpenAI-…

↯ Llamallamaopenai

Hi, I'm using llama.cpp with qwen3.6 35B A3B on two different machines. I noticed that on both machines tokens per second is better while using Q4_K_S and Q4_K_M quants than lower Q3_K_M quants.

↯ Llama↯ Qwen 3.6llama

Benchmarking llama.cpp's brand-new MTP support on Strix Halo PR #22673 landed in llama.cpp on May 16. It adds first-class Multi-Token Prediction (MTP) speculative decoding for models that ship with an MTP head, including Qwen3.6 27B dense…

↯ Llama↯ Qwen 3.6llama

Does anyone know why qwen 3.6 MTP spec decoding won't work with Tesla P40 when the K cache is quantized? I was able to get mtp qwen 3.6 27B Q5 running at 20t/s on my tesla p40.

↯ Llama↯ Qwen 3.6llamaqwen

Is there an easy way to know if a model is using CPU/RAM (and not only GPU/VRAM)? (I think standard verbose output, which got shorter, says nothing about this, but I may be missing something)

↯ Llamallama

I'll be UPDATING this as it seems I was benchmarking and testing Just before the UPDATE LOL TL;DR If you're running rigid agent frameworks locally with mtp on consumer hardware: drop your draft window to 3, lock parallel slots to 1, and co…

↯ Llama↯ Qwen 3.6llamaqwenagentic

testing in progress ...we all need an increase in pp 😆 https://github.com/ggml-org/llama.cpp/releases/tag/b9200 u/am17an am17an commented 13 hours ago • Overview Avoid copying the logits for every token in the batch when doing prompt proce…

↯ Llamallama

Thank you to the people at ik_llama and llama.cpp. It's amazing how far you've all pushed mtp and other tech so that I can run 27B and 35B Qwen3.6 models on an old gaming laptop with a RTX2060 mobile at 6GB VRAM and 32GB RAM.

↯ Llama↯ Qwen 3.6llamaopusagentic

In my real-world usage (opencode) and in my synthetic benchmarks, Coder-Next (Q5) demolishes the whole Qwen3.6 family including the 27B Dense model (All Q8). Everybody else is hailing that 27B is superior and is an amazing model, but I hav…

↯ Llama↯ Qwen 3.6llamaqwen

Here are some results (llama.cpp)! Task 1: write a short poem 27B Dense: 12.5 tokens/s 27B Dense MTP: (spec-draft-n-max 6): 14.5 tokens/s 27B Dense MTP (spec-draft-n-max 3): 18.7 tokens/s Task 2: edit a hello word html artifact 27B Dense:…

↯ Llama↯ Qwen 3.6llamaqwen

With the MTP merge into mainline llama.cpp I wanted to try out some other optimizations i could think of. Ended up tested backends, mtp, and bumping to ROCm nightlies.

↯ Llamamoellama

I'm running Qwen 35B A3B via llama-server with reasoning budget set to -1 (unlimited) for testing. In every client I've tried, the model just thinks endlessly before responding.

↯ Llamallamaqwen

https://github.com/ggml-org/llama.cpp/releases

↯ Llamallama

https://github.com/ollama/ollama/releases/tag/v0.30.0-rc15 Hopefully this has more devs come to llama.cpp to support Day 1 releases due to Ollama now moving to using llama.cpp directly. Additionally, I hope that Ollama makes it clear that…

↯ Ollama↯ Llamaollamallama

Anthropic's first open weight models, Natural Language Autoencoders, are just finetunes of popular open weight models. They do not modify architecture and modeling code so inference with llama.cpp is mostly trivial.

↯ Llamallamaanthropic

Hi all I'm trying to disable reasoning for quicker outputs in llamacpp-server. I remember using LM studio and that having a think button in the gui that could be toggled but later I tried the unsloth ggufs but they don't have that button f…

↯ Llama↯ Qwen 3.5llama

EDIT: Edited to provide more clarity It occurred to me, that perhaps the same draft model used for speculative decoding would be completely adequate if we just used it's output as-is for reasoning, without validating the results against th…

↯ Llama↯ Vllmvllmllama

Hi all I recently started a new job and we're doing python development for a ci cd metadata consolidation library for analytics and we cannot use no stuff like claude code or codex or gh copilot or any model APIs (free or paid). I got a la…

↯ Copilot↯ Ollama↯ Llama↯ Qwen 3.5ollamacopilotllama+3

I spent the past 5+ months building a pipeline that creates hybrid GGUF quant mixes. I also built it to learn from Unsloth (or other) models by utilizing their quant to tensor assignment.

↯ Llama↯ Qwen 3.6llama

I am looking for recommendations on the best way to run local LLMs on NVIDIA DGX Spark. Which stack makes the most sense in practice: TensorRT-LLM, vLLM, or llama.cpp?

↯ Llama↯ Vllmvllmllama

If you are using a MoE model that does not fully fit in your GPU, some of the experts must stay on the CPU. Putting the experts that you will actually need on the GPU will give you GPU inference speeds.

↯ Llamamoellama

basically what I'm doing here is trying to validate whether or not it's a reasonable idea to get a couple of V100s, either SXMs with PCIe adapters or straight-up PCIe cards in the first place, for the sake of running this model or models l…

↯ Llama↯ Qwen 3.6llama

You open Claude.ai, chatgpt.com, gemini, whatever LLM provider you use. You type something: "What is the capital of France?" You hit enter.

↯ Llamallamageminichatgpt

In llamacpp I'm getting 12tok/s, does this number look right to you and what can I do to increase this number (if possible)? cd ~/llama.cpp && ./build/bin/llama-server -m models/qwen-3.6-27b-abliterated-q3.gguf -ngl 999 -c 65536 (i need th…

↯ Llama↯ Qwen 3.6llamaqwen

Hello there (beginner here) I've been unable to build myself llama.cpp for my Strix Halo (Windows 11) (cmake errors, I have not digged too much into it, already burned hours...), so I was wondering when an official release for Vulkan/HIP w…

↯ Llamallama

I remember a year or so ago when DeepSeek R1 came out and it was pretty quickly distilled into Llama 3 8b and Qwen 2.5 (?) 7b. Why don’t we see more distilled models?

↯ Llama↯ Qwen 2.5↯ Qwen 2.5↯ Qwen 2.5deepseekllamaqwen

Hey everyone! I’ve just uploaded a GGUF version of nvidia/Gemma-4-26B-A4B-NVFP4.

↯ Llama↯ Gemma↯ Gemma 4gemmallama

I was asked for this guide, so here it is. Some overlap with someone else’s post from yesterday.

↯ Llama↯ Qwen 3.6moellamaopus

UPDATE: i have switched to vulkan (image: ghcr.io/ggml-org/llama.cpp:server-vulkan-b9014) and now i am getting prompt eval: 591.01 tok/s generation: 41.90 tok/s which is faster than rocm new config: services: llama-cpp: container_name: lla…

↯ Llama↯ Qwen 3.6llama

A lot of the posts in this sub is about advice about which hardware to buy, what settings to use and what speed to expect. There are a lot of excellent replies spread all over the place, but alot of it is also just vague indications like ~…

↯ Llamallama

So I saw an article recently about exo disaggregated prefill with DGX Spark and M3 Ultra - prefill on one machine and decode on another. DGX Spark apparently has 4x matmul performance over an M3 Ultra - same as the M5 Ultra should have.

↯ Llama↯ Minimaxminimaxllamaqwen

llmconfig Local Large Model Config — manage local inference with llama.cpp, stable-diffusion.cpp, and whisper.cpp from a single YAML file and a single CLI. llmconfig up gemma # or just: llmc up gemma ✓ gemma is ready at http://127.0.0.1:80…

↯ Llama↯ Gemmagemmallama

I have been trying Mistral 3.5 on my 4x RTX 3090 rig with llama.cpp. Inference is slow (about 11 t/s) even without anything being offloaded to the CPU.

↯ Llama↯ Vllm↯ Mistral↯ Mistral 3.5mistralvllmllama+1

Access Claude, Llama, Mistral, Nova and more through a single OpenAI-compatible API. Start for as little as €1.

↯ Llama↯ Mistralmistralllamaopenai

Hey fellow Llamas, thank you for all the nice words and great feedback on the last post I made. We have something new we thought would be useful to share.

↯ Llama↯ Qwen 3.6llama

https://preview.redd.it/7yei65sbugyg1.png?width=1703&format=png&auto=webp&s=ad388c51dd10cb44b41a99876d28797e006fd138 Stanford's Generative Agents = one LLM cosplaying 25 personas. I wanted agents that actually become different people — dif…

↯ Llama↯ Qwen 3llama

I have (free) access to a SLURM cluster with 8x NVIDIA A100 80GB GPUs (=640 GB VRAM) on a single task, and I want to run an open-weight model locally with llama.cpp for data generation, not coding. My use case is generating teacher data fo…

↯ Llama↯ Fine Tuningfine-tuningllama

Curious what stacks people are actually using right now, and where you're hitting walls. Some things I've been observing while testing combos: - Deepgram Nova-3 still the best STT for English, Cartesia is closing the gap on streaming - Ele…

↯ Llama↯ Llama 3llamaopenaianthropic

Whew! Big work today getting optimized llama.cpp MIPS R8000 kernel running on the SGI Power Challenge deskside from 1995 with Gemma 3 270M.

↯ Llama↯ Gemmagemmallama

I have an MI50 that I use with llama.cpp/Vulkan, however some models run quite slowly, so I'd like to try the ROCm backend, but no matter what I try it doesn't work. Downloading the missing files from ArchLinux package doesn't work.

↯ Llamallama

https://github.com/ggml-org/llama.cpp/pull/22196 And somehow we already got some GGUFs for it! https://huggingface.co/CISCai/gemma-4-31B-it-NVFP4-turbo-GGUF https://huggingface.co/stevelikesrhino/gemma-4-31B-it-nvfp4-GGUF (the below one is…

↯ Llama↯ Gemma↯ Qwen 3.5gemmallama

I now use (mostly) Gemma 4 and Qwen 3.5 models *. And seems that all of them, after context grows a bit, after providing long output for me and getting a short prompt in response, are starting to process many new tokens as input and I have…

↯ Llama↯ Gemma↯ Qwen 3.5gemmallamaqwen

TLDR: I've been running gemma4 e2b extensively on iOS with llama.cpp and found some interesting quirks and info you guys may like! These are specifics for the iPhone and what I've found worked across 20+ devices.

↯ Llama↯ Gemma↯ Gemma 4gemmallama

I have a dedicated linux box I run all my stuff on. I occasionally see the 'zomg 35b can't call tools?!' posts here and chuckle to myself in a *zero issues here* way.

↯ Llamallama

It looks as if OpenCode introduces an artificial delay in agentic coding. Have you noticed similar issues?

↯ Llamallamaagentic

At least in open-webui. Nothing has changed except for the backend update.

↯ Llama↯ Qwen 3.6llamaqwen

Repo: statisticalplumber/kanban at pi-agent-integration Hi Guys, To test Qwen 3.6’s potential, I also wanted the Cline Kanban project to have an open-source agent to work with. The last time I tested Cline Kanban, it didn’t support agents…

↯ Llama↯ Qwen 3.6clinemoellama+3

I'm curious if any brave soul has upgraded their computer (especially if it's Strix Halo) from Ubuntu 24.04 -> 26.04 and seen a significant performance improvement for inference with VLLM, llama-server, and/or LM Studio.

↯ Llama↯ Vllmvllmllama

Decided to try out the new --spec-type ngram-mod feature in llama.cpp using Qwen3.6 27B during an OpenCode bug chasing session. TLDR: Performance is variable, but so far it seems to provide a nice speed increase for working on the same cod…

↯ Llama↯ Qwen 3.6llama

I'm a daily llama-cpp user and was hoping to try MTP on vLLM. Unfortunately, pipeline parallelism + MTP does not seem to work with this model in vLLM.

↯ Llama↯ Vllm↯ Qwen 3.6vllmllama

https://preview.redd.it/tblmrwxkbexg1.png?width=1193&format=png&auto=webp&s=6dea1e6684e75e22852d57c0c72e9171deb56ae2 I have experimented how to run Qwen3.6-27B on my laptop with an A5000 16GB GPU. I have created an own IQ4_XS GGUF "qwen3.6…

↯ Llama↯ Qwen 3.6llama

Hey folks — looking for some advice on improving my local LLM setup (and also exploring agentic coding workflows). Current setup: GPU: RTX 3090 (24GB VRAM) RAM: 64GB Using llama.cpp with a Qwen3.6 27B Q6 model (GGUF) Running through OpenCo…

↯ Llama↯ Qwen 3.6llamaagentic

Hi! We, Vincenzo and Riccardo, built Doxa as an agnostic engine for emergent simulations with agents for constrainted scenarios (like geopolitical, economics, ...) and work well with LLMs like Qwen2.5:7B, Llama but also cloud models such a…

↯ Llama↯ Qwen 2.5llamagemini

I usually just throw models into LM Studio but I decided to finally compile llama.cpp on my hardware to get some extra speed and to hopefully replace my increasingly unreliable cloud subscription. I have a RTX 4080 and Ryzen 5 7600 with 32…

↯ Llamallama

I tried working on a local LLM project today and honestly ended up pretty frustrated. I tested several approaches, but none of them worked reliably.

↯ Llama↯ Qwen 3.6llamaqwen

I knew there would be a speed penalty when switching the KV cache quantization from F16 to Q8, but I never expected it to be this significant at longer context sizes. I ran a test with Qwen 3.5 122B on my MacBook M2 Max using llama.cpp.

↯ Llama↯ Qwen 3.5llamaqwen

Do I understand correctly, based on this comment, that I can potentially fit Qwen 3.6 27B FP8 precision model and have around 256K context available and fit it fully in my RTX 5090 VRAM? Of course with the help of TurboQuant compression, a…

↯ Llama↯ Qwen 3.6llamaqwen

Selam, Bir süredir üzerinde çalıştığım bir projeyi paylaşmak istiyorum. Sıfırdan geliştirdiğim bir Türkçe dil modeli var: Marul V7 Model tamamen bağımsız şekilde eğitildi.

↯ Llamallama

↯ Llamallama

↯ Llama↯ Minimaxminimaxllama

↯ Llama↯ Qwen 3.6llamaqwen

↯ Llama↯ Qwen 3.6moedeepseekllama+1

↯ Llama↯ Qwen 3.6moellamaqwen

↯ Llamallamamcp

↯ Llama↯ Gemma↯ Qwen 3.6gemmallamaqwen

↯ Llamallama

Hi guys, just want to share with you guys a Frankenstein build I put together that is surprisingly decent I have a i5 12400 / B660 / 32GB DDR4 build that was previously paired with a 3060ti. Last Christmas I upgraded it to a RX9070, then I…

↯ Llama↯ Qwen 3.6llamaqwen

I have Gemma4-E2B working within home assistant as STT, and E2B seems fast and accurate for STT (maybe a bit better than Parakeet), however, it responds with the entire thought process: https://preview.redd.it/v8zhb5elltvg1.png?width=599&f…

↯ Llama↯ Gemma↯ Gemma 4gemmallama

I'm running a local AI setup and want to make sure I'm using my hardware to the absolute maximum. If you have tips on better models, smarter configurations, or services I'm missing, drop them in the comments.

↯ Llamaopenclawllama

https://w418ufqpha7gzj-80.proxy.runpod.net Started for myself, but since Im not using it continuously, sharing it: Open Access Qwen3.6-35B-A3B-UD-Q5_K_M with TurboQuant (TheTom/llama-cpp-turboquant) on RTX 3090 (Runpod spot instance). 5 pa…

↯ Llama↯ Qwen 3.6llama

I want to create a local GenAI. Tell me the server machine requirements.

↯ Llamallama

It's been some time now, surely some have tinkered with it more and optimised it already

↯ Llamallama

https://preview.redd.it/4906akj9dovg1.png?width=1527&format=png&auto=webp&s=c49e255ac79a3c5455f44603422f8af7ddc12594 First of all can we make https://www.youtube.com/watch?v=2lUC8Gimxz8 Angine de Poitrine this subs official band? Those guy…

↯ Llama↯ Qwen 3.6llamaqwen

Basically I am asking the model to describe an image, but it says it can't process the images. The weird thing is that if I send the image encoded directly on the prompt, it works just fine, I am using llama-server with qwen3.5 (tried all…

↯ Llama↯ Qwen 3.6llamaqwencodex

I am an local LLM beginner and I found this Reddit while looking for help. (Please understand that I am unfamiliar with Reddit.) (system- i5 4440 1.8GHz/b85m ds3h/DDR3 32GB/128GB SSD/Ubuntu 25.10 questing) I loaded Qwen3.5 27B Q4_K_M onto…

↯ Llama↯ Qwen 3.5llama

Hi! I hope its okay for me to ask this here.

↯ Llama↯ Qwen 3.5llamaqwen

I am playing around with Intel Arc B70, still trying to decide whether I keep it or not. After some battle, I got it working with Radeon 5500 and B550M, now I am on to the fun part of getting software to work.

↯ Llama↯ Vllm↯ Qwen 3.5vllmllama

It seems we have entered a period of accelerating innovation regarding the KV cache. Someone mentioned this post's paper in the Github issue of llama.cpp for implementing Turbo Quant.

↯ Llamallama

If you pull https://github.com/ggml-org/llama.cpp/releases/tag/b8786 and try to build with Vulkan support on Fedora, you may hit this error: [ 39%] Building CXX object ggml/src/ggml-vulkan/CMakeFiles/ggml-vulkan.dir/multi_add.comp.cpp.o /h…

↯ Llamallama

Introducing dotLLM - Building an LLM Inference Engine in C# If you’ve been building .NET applications and wanted to run LLMs locally, your options have been… limited. You could wrap llama.cpp through LLamaSharp, deal with ONNX Runtime, or…

↯ Llamallama

Due to costs I am running on some older hardware. Looking for suggestions on supported models for my particular stack.

↯ Llamallama

TokenMonopoly Live leaderboard of AI API deals — pricing, subscriptions, and SWE-bench scores for Claude, GPT, Gemini, Kimi, DeepSeek, Llama and more. Compare 27 benchmarked models across 96 hosts by price-per-performance, refreshed daily.

↯ Llama↯ Swe Benchswe-benchdeepseekllama+1

The introduction of TurboQuant, PolarQuant, and QJL (Quantized Johnson-Lindenstrauss) by Google Research represents more than just a technical optimization. At Vucense, we view this as a landmark moment for Inference Sovereignty https://vu…

↯ Ollama↯ Llamaollamallama

I benchmarked the Qwen3.6-35B-A3B-MTP-UD-Q4_K_M.gguf model (a 35B Mixture-of-Experts model with 3B active parameters, 22 GiB quantized) across all available llama.cpp compute backends on an Intel Panther Lake laptop. The goal: find out whi…

↯ Llamallama

Hi everyone, Been working on Otis, an open-source ai agent that gives you one minimal experience across local and hosted open-weight models, privacy-focused by design. On setup it recommends a local model based on the hardware Otis is runn…

↯ Ollama↯ Llamaollamallama

llmash An Ollama-compatible server and command line for Windows, built on llama.cpp. It serves your GGUF files through llama-server and keeps Ollama's commands, API and model store, so anything already pointed at Ollama keeps working.

↯ Ollama↯ Llamaollamallama

A dead simple local audit trail for Claude Code tool calls. Energized by a rule-based risk analyser.

↯ Llamallamaclaude-code

LlamaRack A self-hosted control plane and OpenAI-compatible gateway for llama.cpp. LlamaRack manages GGUF models, durable llama-server Instances, GPU placement, automatic loading and unloading, Hugging Face downloads, request observability…

↯ Llamallamaopenai

MyHandler is the private AI assistant and gatekeeper that handles your email, meetings, and calls. Local-first, encrypted, you stay in control.

↯ Llamallama

PicoLM is an LLM inference engine written in C99. It currently supports llama-2, GPT-2, Qwen 3.6/3.8(+MoE) and Gemma-3n models.

↯ Llama↯ Gemmagemmamoellama+2

a11ce.com/llama-3.1-405b.html Llama 3.1 405b (base) is no longer hosted by any public inference provider. This page has instructions for running it yourself on an on-demand GPU instance for ~$20/hr and 10 minutes of setup.

↯ Llamallama

ggml-axcl — llama.cpp Axera NPU backend A custom llama.cpp backend (ggml-axcl) that runs Qwen3-0.6B directly from GGUF on an Axera AX8850 NPU accelerator card (M5Stack LLM-8850: 24 TOPS INT8, 8 GB LPDDR4x) hosted on a Raspberry Pi 5. The G…

↯ Llamallama

Llama Llama is a macOS menu bar app for running local LLMs. Watch a 2-minute intro 📽️ Install brew install --cask llama-app Or download from Releases.

↯ Llamallamamcpagentic

privibe CLI coding agent for private, local-first development. privibe is a fork of Mistral Vibe reworked to not do any call back home of any kind and then run against local models first, I want to be able to use it and know it will not be…

↯ Llama↯ Mistral↯ Qwen 3mistralllama

DLLM - D Language 🤖 on 🦙.cpp A minimal, clean D language agent built directly on llama.cpp via importC. No Python, no bindings, no overhead.

↯ Llamallama

MOST POPULAR AI - AI and ML The future is for billionaires – the rest of us will get open weight AI models, maybe Mark Zuckerberg muses about 'superintelligence' and 'arc of human civilization' - AI AND ML Zuck rekindles open weights Llama…

↯ Llamallama

The LLAMA tests A suite of language learning tests. The LLAMA test battery consists of four tasks.

↯ Llamallama

Come help push the frontier of token speed across local models and hardware with your agents! Current frontier Laguna XS 2.1 · AMD R9700 (llama.cpp HIP) +31.14% 143.3 tok/s Laguna XS 2.1 · DGX Spark GB10 (vLLM NVFP4) +5.28% 37.3 tok/s Lagu…

↯ Llama↯ Vllm↯ Qwen 3.8↯ Qwen 3.8↯ Qwen 3.8↯ Qwen 3.8↯ Qwen 3.8vllmllama

A local LLM runtime that boots from USB and runs without a conventional operating system. Real boot, one cut: loading and prefill sped up, generation at actual speed (Llama 3.2 1B, QEMU/KVM, 8 cores).

↯ Llamallama

Lexicon is a rich-text editor with grammar checking and AI writing tools (rewriting, tone shifting, and summarizing) that run entirely on your machine. No account, no API calls, and nothing uploaded.

↯ Ollama↯ Llamaollamallama

I'm the founder of Scalattice, this is my second company, third total product. I'm a 2x founder building some challenging software, some easy software, and some curiosity based tools that I've just always wanted to be a part of!

↯ Llama↯ Gemmagemmadeepseekllama+1

Inspiration Large language models running across multiple GPUs are often limited not only by computation, but by communication between GPUs. This becomes especially visible on systems without NVLink or with slow PCIe connections.

↯ Llamallama

cpubrrr https://github.com/user-attachments/assets/a390cb6e-86b8-41e5-9ef6-957c94dabe19 From-scratch CPU-only LLM inference that beats llama.cpp's CPU path — on both quant formats it runs, no GPU. cpubrrr is a research runtime that runs fr…

↯ Llamallama

Diktafon is a voice-memo app: each topic is a cassette, memos are appended to its tape, and a cassette plays back as one continuous recording. Every memo is transcribed and every cassette summarised, fully on-device: whisper.cpp for transc…

↯ Llamallama

I had codex make this for me so I could see what llama.cpp was doing on my mac. Vibecoded in swift, hit 1 for more processor details, hit m for more memory details.

↯ Llamallamacodex

BigMoeOnEdge Run Mixture-of-Experts models far bigger than your edge device's RAM. The result: a ~60 GB model on a 12 GB phone: 1.3 tok/s lossless, byte-identical to running from RAM, 2.2 tok/s with one speed knob.

↯ Llamamoellama

LLM Red Team Lab A hands-on kit for educational, authorized red teaming of any locally-run LLM. It works with any OpenAI-compatible model — Llama, Mistral, Qwen, Gemma, DeepSeek R1, and more — and covers the two ways an LLM system gets exp…

↯ Security↯ Llama↯ Mistral↯ Gemma↯ Jailbreakred-teammistraljailbreak+6

Hi HN, I started this because I wanted something in the spirit of llama.cpp and stable-diffusion.cpp, but for local 3D generation model: native executables, no Python/PyTorch runtime, and both CUDA and Vulkan backends. I’ve mainly tested i…

↯ Llamallama

i open-sourced bonsai-turbo -- a batch-1 decode engine that runs @PrismML's Bonsai 27B 1.76x faster than the official llama.cpp fork. same outputs, token for token H100, tg128, greedy: ternary 85.5 &gt;&gt; 151 tok/s.

↯ Llamallama