#vllm

311 items

Qwen/Qwen3.8-2.4T-A95B (huggingface.co via hn) +14950 5w

Qwen3.8-2.4T-A95B [!Note] This repository contains model weights and configuration files for the post-trained model in the Hugging Face Transformers format. These artifacts are compatible with vLLM, SGLang, TokenSpeed, etc.

↯ Vllm vllm qwen
Qwen 3.6 27B on 24GB VRAM setup: backend comparisons, quant choice and settings (llama.cpp, ik_llama.cpp, BeeLlama, vllm) (www.reddit.com) +10548 17w

TL;DR best setup I tested on a RTX 3090 24 GB: ik_llama.cpp + Qwen3.6-27B-MTP-IQ4_KS.gguf 156k context, q8_0/q8_0 KV, MTP, vision on CPU benchmark result on a ~5.9k prompt + 1k output: about 1261 tok/s prefill, 72.9 tok/s decode llama.cpp…

↯ Llama ↯ Vllm ↯ Qwen 3.6 vllm llama qwen
What do you want me to try? (www.reddit.com) +7864 21w

Got a new playground at work. Anything I cn help run (via vllm maybe) that you might be curious about.

↯ Vllm vllm
VLLM PR : New MoE model from Cohere soon (github.com via reddit) +639 21w

Easy, fast, and cheap LLM serving for everyone | Documentation | Blog | Paper | Twitter/X | User Forum | Developer Slack | 🔥 We have built a vLLM website to help you get started with vLLM. Please visit vllm.ai to learn more.

↯ Vllm vllm moe
MI50s Qwen 3.6 27B @52.8 tps TG @1569 tps PP (no MTP, no Quant) (www.reddit.com) +5821 18w

TL;DR Results from the title are for single inference with 2 prompt of 1k and 15k tokens. So no MTP (as it’s slower for big prompt), no DFlash (working too but slower for big prompt), no quant used (full precision wanted) and the results a…

↯ Vllm ↯ Qwen 3.6 vllm qwen agentic+1
LiquidAI/LFM2.5-8B-A1B · Hugging Face (huggingface.co via reddit) +4915 16w

looks like you can run it on any potato (A1B)! https://huggingface.co/LiquidAI/LFM2.5-8B-A1B-GGUF from LiquidAI: LFM2.5 is a new family of hybrid models designed for on-device deployment.

↯ Llama ↯ Vllm vllm moe llama+1
KVarN: Native vLLM KV-cache quantization back end by Huawei (github.com via hn) +424 15w

⚡️ Built for agentic and long-context workloads. 💡 KVarN delivers 3-5x more KV-cache capacity and up to ~1.3x the throughput of FP16, so you fit far longer contexts and serve more concurrent requests, with FP16-level accuracy.

↯ Vllm vllm agentic
Qwen 3.6: worse adherence? (www.reddit.com) +3727 22w

Just swapped Qwen 3.5 for the 3.6 variant (FP8, RTX 6000 Pro) using the same recommended generation settings. My stack is vLLM (v0.19.0) + Open WebUI (v0.8.12) in a RAG setup where the model has access to several document retrieval tools.

↯ Vllm ↯ Qwen 3.6 vllm rag qwen
vLLM ROCm has been added to Lemonade as an experimental backend (www.reddit.com) +339 19w

vLLM has the ability to run .safetensors LLMs before they are converted to GGUF and represents a new engine to explore. I personally had never tried it out until u/krishna2910-amd/ u/mikkoph and u/sa1sr1 made it as easy as running llama.cp…

↯ Llama ↯ Vllm vllm llama
FastDMS: 6.4X KV-cache compression running faster than vLLM BF16/FP8 (www.reddit.com) +305 19w

Last year researchers affiliated with NVIDIA, University of Warsaw, and University of Edinburgh published Dynamic Memory Sparsification (DMS), a KV-cache sparsification technique using learned per-head token eviction, reporting up to 8x KV…

↯ Llama ↯ Vllm vllm llama
Need a second pair of eyes, this Qwen3.6 27B quant recipe consistently thinks less and is correct (www.reddit.com) +298 18w

Ok, hear me out. This all started when I was trying to understand why this Qwen3.6 27B INT8 Autoround (https://huggingface.co/Minachist/Qwen3.6-27B-INT8-AutoRound/tree/main) recipe was performing so much better than any other Qwen3.6 27B q…

↯ Llama ↯ Vllm ↯ Qwen 3.6 vllm llama
Inside vLLM: Anatomy of a High-Throughput LLM Inference System (2025) (www.aleksagordic.com via hn) +231 6w

Inside vLLM: Anatomy of a High-Throughput LLM Inference System From paged attention, continuous batching, prefix caching, specdec, etc. to multi-GPU, multi-node dynamic serving at scale August 29, 2025 In this post, I'll gradually introduc…

↯ Vllm vllm
Here are my KV cache quantization benchmarks: TurboQuant is overrated but saved by TCQ, q5 deserves more attention, and symmetric q8 might be a waste of VRAM (www.reddit.com) +2320 17w

Greetings from former TurboQuant's biggest defender, now middle-sized niche-aware TurboQuant defender. Today I'm presenting to you the results of me thoroughly exploring the world of PPL and KLD benchmarks with my single RTX 3090 using Bee…

↯ Vllm ↯ Qwen 3.6 vllm qwen
Is using vLLM actually worth it if you aren't serving the model to other people? (www.reddit.com) +2126 18w

So, as most of us here are, I'm a llama.cpp loyalist. Easy to understand, great configuration, relatively stable, etc.

↯ Llama ↯ Vllm vllm llama
Built an open-source one-prompt-to-cinematic-reel pipeline on a single GPU — FLUX.2 [klein] for character keyframes, Wan2.2-I2V for animation, vision critic with auto-retry, music + 9-language narration in the same pipeline (www.reddit.com) +202 18w

Shipped this for the AMD x lablab hackathon. Attached video is one of the actual reels the pipeline produced - one English sentence in, finished mp4 with characters, story, music, and voice-over out (fast demo video, not the best quality).

↯ Vllm ↯ Qwen 3.5 vllm moe
Follow-up: Qwen3.6-27B on 1× RTX 3090 — pushing to ~218K context + ~50–66 TPS, tool calls now stable (PN12 fix) (www.reddit.com) +207 20w

Following up on our previous post about running Qwen3.6-27B on a single RTX 3090 (~125K context, higher TPS). We’ve been pushing further on both context length and stability for tool-agent workloads.

↯ Vllm ↯ Qwen 3.6 vllm
Qwen3.6-27B at ~80 tps with 218k context window on 1x RTX 5090 served by vllm 0.19 (www.reddit.com) +201 21w

Qwen3.6-27B is out for a few days and the NVFP4 with MTP is dropped earlier on HF: https://huggingface.co/sakamakismile/Qwen3.6-27B-Text-NVFP4-MTP Can follow the same recipe I used for Qwen3.5-27B to achieve ~80 tps on a single RTX 5090 at…

↯ Vllm ↯ Qwen 3.6 vllm
Speculative Decoding in vLLM on AMD GPUs (vllm.ai via hn) +183 12d

Exploring Speculative Decoding in vLLM on AMD GPUs TL;DR: Speculative decoding allows vLLM to verify multiple drafted tokens in a single target-model pass. In our experiments, its effect on output-token throughput varied across drafting me…

↯ Vllm vllm
Show HN: Tiny-vLLM – high performance LLM inference engine in C++ and CUDA (github.com via hn) +182 16w

tiny-vllm You're going to build a high performance LLM inference engine with C++ and CUDA - tiny-vllm, a younger and smaller sibling of vLLM We will learn a lot along the way, make mistakes and derive the ideas and maths from scratch This…

↯ Vllm vllm
Gemma 4 MTP vs DFlash on 1x H100: dense vs MoE results (www.reddit.com) +1712 18w

Benchmarked Gemma 4 MTP and z-lab's DFlash on a single H100 80GB using vLLM and NVIDIA's SPEED-Bench qualitative dataset. Setup: Hardware: 1x H100 80GB Runtime: vLLM Dataset: SPEED-Bench qualitative Prompts: 880 total, 80 prompts across ea…

↯ Vllm ↯ Gemma ↯ Gemma 4 vllm gemma moe
Gemma 4 26B Hits 600 Tok/s on One RTX 5090 (www.reddit.com) +1714 19w

I ran a benchmark to see how much DFlash speculative decoding actually helps in vLLM. Setup: GPU: RTX 5090, 32GB VRAM vLLM: 0.19.2rc1 Main model: cyankiwi/gemma-4-26B-A4B-it-AWQ-4bit Draft model: z-lab/gemma-4-26B-A4B-it-DFlash Workload: r…

↯ Vllm ↯ Gemma ↯ Gemma 4 vllm gemma
Show HN: Free Inference Engineer and Model Training Roadmap (inferquest.org via hn) +167 3w

Free, gamified roadmaps for LLM engineering: an Inference Engineering path (KV caches, CUDA kernels, production vLLM serving) and a Model Training path (pretraining on a budget, scaling laws, SFT/DPO/GRPO) — 183 tasks with auto-verified mi…

↯ Vllm ↯ Dpo dpo vllm
DeepSeek-V4-Flash W4A16+FP8 with MTP self-speculation: 85 tok/s @ 524k on 2× RTX PRO 6000 Max-Q (www.reddit.com) +156 18w

TL;DR: DeepSeek-V4-Flash running at 85.52 tok/s @ 524k ctx and ~111 tok/s @ 128k single-stream on 2× RTX PRO 6000 Max-Q pasta-paul's DeepSeek-V4-Flash-W4A16-FP8 quant is great, but its MTP head silently gets stripped at load time (HF trans…

↯ Vllm ↯ DeepSeek 4 vllm deepseek
Kv cache quantization: ignorance, or malice? (www.reddit.com) +1533 20w

I run Qwen-3.6 27B FP8 on vllm for long-horizon agentic coding harness workloads with high context window and concurrent sub-agents. On two 3090s that aren’t used for anything else, it seems reasonable to expect a good balance between spee…

↯ Vllm ↯ Qwen 3.6 vllm qwen agentic
Running the new Qwen3.6-35B-A3B at full context on both a 4090 and GB10 Spark with vLLM and Llama.cpp (www.reddit.com) +144 22w

Here is how to run the new Qwen3.6-35B-A3B > At full context on a 4090 - IQ4_XS gguf with llama cpp > At full context on a Spark - FP8 with a tweaked vLLM Here is the docker compose with llama cpp services: llamacpp: container_name: llamac…

↯ Llama ↯ Vllm ↯ Qwen 3.6 vllm llama
Qwen 3.8-27B goes openweight in 2 days (huggingface.co via hn) +1313 5w

Qwen3.8-27B [!Note] This repository contains model weights and configuration files for the post-trained model in the Hugging Face Transformers format. These artifacts are compatible with Hugging Face Transformers, vLLM, SGLang, TokenSpeed,…

↯ Vllm ↯ Qwen 3.8 vllm qwen
z-lab released gemma-4-26B-A4B-it-DFlash. Anybody tried it yet? (huggingface.co via reddit) +135 19w

Past few days, its all been about MTPs. Somehow people missed out the fact that Z lab released the Dflash for Gemma4 26B a couple of days ago.

↯ Vllm ↯ Gemma ↯ Qwen 3.6 vllm gemma qwen
Those of you running minimax 2.7 locally, how are you feeling about it? (www.reddit.com) +1221 22w

Im running the raw version straight from the minimax release on hugging face (https://huggingface.co/MiniMaxAI/MiniMax-M2.7) on 3 rtx pro 6000's on vllm. So no quantization.

↯ Vllm ↯ Minimax ↯ MiniMax 2.7 minimax vllm
Eagle 3.1: Collaboration Between the EAGLE Team, vLLM Team, and TorchSpec Team (vllm.ai via hn) +113 16w

EAGLE 3.1: Advancing Speculative Decoding Through Collaboration Between the EAGLE Team, vLLM, and TorchSpec The EAGLE series — including EAGLE 1, EAGLE 2, and EAGLE 3 — has become one of the most widely adopted and practically deployed fam…

↯ Vllm vllm
Simple to use vLLM Docker Container for Qwen3.6 27b with Lorbus AutoRound INT4 quant and MTP speculative decoding - 118 tokens/second on 2x 3090s (github.com via reddit) +115 20w

Qwen3.6-27B vLLM Docker Docker-based vLLM serving for Qwen3.6-27B with Lorbus AutoRound INT4 quant and MTP speculative decoding. Model is downloaded at runtime and stored on a host volume so the container can be upgraded without redownload…

↯ Vllm ↯ Qwen 3.6 vllm
Qwen3.6-27B-INT4 clocking 100 tps with 256k context length on 1x RTX 5090 via vllm 0.19 (www.reddit.com) +114 20w

Thanks to the community the Qwen3.6-27B speed keeps getting better. The following improves upon my recipe from yesterday and delivered a whopping 100+ tps (TG).

↯ Vllm ↯ Qwen 3.6 vllm
Ran the same models across Strix Halo, RTX 3090, and RTX 5070 because I wanted my own numbers (www.reddit.com) +106 18w

I kept seeing inference-speed claims for these models and wanting an apples-to-apples comparison on the hardware I actually have. So I built a harness and a public page that dumps every run as YAML.

↯ Vllm ↯ Gemma ↯ Qwen 3.5 vllm gemma moe+1
Finding the 4x 3090 Sweet Spot (www.reddit.com) +1010 18w

https://preview.redd.it/8o43bjhe9d1h1.png?width=5346&format=png&auto=webp&s=1c87c2ee8b8ffff43495f543266056b0e26d3947 In another post I had someone ask me about the power draw of the 4x 3090 setup so I'm sharing a a full test I conducted to…

↯ Vllm ↯ Qwen 3.6 vllm
vLLM Just Merged TurboQuant Fix for Qwen 3.5+ (www.reddit.com) +101 19w

Previously it was throwing a 'Not Implemented' error due to Mamba layers. Going to test it now!

↯ Vllm ↯ Qwen 3.5 vllm qwen
Qwen3.6-27B at 72 tok/s on RTX 3090 on Windows using native vLLM (no WSL, no Docker), portable launcher and installer (www.reddit.com) +104 20w

The angle here is native Windows, no WSL. Simple installation, open source, no telemetry.

↯ Vllm ↯ Qwen 3.6 vllm openai
Intel B70: LLama.ccp SYCL vs LLama.cpp OpenVino vs LLM-Scaler (www.reddit.com) +101 20w

In case anyone is interested, I decided to test out LLama.cpp's new OpenVino backend to see how it compares on Intel GPUs. At first glance, it stomps all over the previous best-case, SYCL, but lags behind LLM-Scaler (Intel's VLLM fork), li…

↯ Llama ↯ Vllm vllm llama
VLLM gives 5x speed of llama but quants not available (unsloth/gguf). What to do? (www.reddit.com) +939 16w

EDIT - IGNORE. I MADE A MISTAKE.

↯ Llama ↯ Vllm ↯ Qwen 3.6 ↯ Qwen 3.6 ↯ Qwen 3.6 ↯ Qwen 3.6 ↯ Qwen 3.6 ↯ Qwen 3.6 ↯ Qwen 3.6 ↯ Qwen 3.6 ↯ Qwen 3.6 ↯ Qwen 3.6 ↯ Qwen 3.6 vllm llama
Show HN: NanoRL – RL training for LLMs in ~1,800 lines (github.com via hn) +8 5w

The smallest async RL trainer I could write: one loop that runs REINFORCE on CartPole on a laptop and async GRPO on a cluster (e.g. 8xH100 trainer, 8 vLLM workers, ran as a [SkyPilot job group](https://docs.skypilot.ai/en/latest/examples/j…

↯ Vllm vllm
Question: Llama cpp, whats good right now for: MTP, KV cache quant, Long context. (www.reddit.com) +819 16w

Used the vllm version of https://github.com/noonghunna/club-3090 It worked fine for myabe 20 40k context, havent tried the new one. Anyone used the new llama.cpp patched one for single 3090?

↯ Llama ↯ Vllm ↯ Qwen 3.6 ↯ Qwen 3.6 ↯ Qwen 3.6 ↯ Qwen 3.6 ↯ Qwen 3.6 ↯ Qwen 3.6 ↯ Qwen 3.6 ↯ Qwen 3.6 vllm llama qwen
club-5060ti: practical RTX 5060 Ti local LLM notes and configs (github.com via reddit) +83 18w

I put together a small public repo for RTX 5060 Ti 16GB local LLM setups: I took inspiration from the club-3090 repo, but this one is focused on documenting what we’ve actually tested on 5060 Ti hardware so the setup details are easier to…

↯ Llama ↯ Vllm ↯ Qwen 3.6 vllm llama openai
qwen3.6 just stops (www.reddit.com) +89 18w

https://preview.redd.it/74cj1xu9pw0h1.png?width=1229&format=png&auto=webp&s=3ae999cc3530ecb4eccf70e25f1a9eb2aa3f2d7b Sometimes qwen 3.6 just stops at the middle of a task, is there a way to avoid it? This is qwen-code CLI, but also happens…

↯ Vllm ↯ Qwen 3.6 vllm qwen openai
Exaggerated PCI-E bandwidth concerns? (www.reddit.com) +815 19w

I frequently see (both here and on r/LocalLLM ) comments that multi-gpu setups are complex, problematic and typically bottlenecked by PCI-E bandwidth on consumer motherboards. I am running 2x RTX 5060 TI 16gb ( and about to add a third ),…

↯ Vllm vllm
Throughput and TTFT comparisons of Qwen 3.6 27B, Qwen 3.6 35B A3B and Gemma 4 models on H100 (www.reddit.com) +85 21w

I wanted to figure out which of the newer small and mid-size models are actually worth running on a single H100, so I put 8 of them through a proper vLLM benchmark and recorded what came out. The setup was simple.

↯ Vllm ↯ Gemma ↯ Qwen 3.6 vllm gemma qwen
Bench 8xMI50 MiniMax M2.7 AWQ @ 64 tok/s peak (vllm-gfx906-mobydick) (www.reddit.com) +88 22w

Inference engine used (vllm fork): https://github.com/ai-infos/vllm-gfx906-mobydick/tree/main Huggingface Quants used: cyankiwi/MiniMax-M2.7-AWQ-4bit Relevant commands to run: docker run -it --name vllm-gfx906-mobydick-mixa3607 -v ~/llm/mo…

↯ Vllm ↯ Minimax minimax vllm
Alibaba open-sources Qwen3.6-35B-A3B, a 35B MoE model with 3B active parameters (huggingface.co via hn) +8 22w

Qwen3.6-35B-A3B [!Note] This repository contains model weights and configuration files for the post-trained model in the Hugging Face Transformers format. These artifacts are compatible with Hugging Face Transformers, vLLM, SGLang, KTransf…

↯ Vllm ↯ Qwen 3.6 vllm moe
Free vLLM Course: Inference, Compression, Benchmarks (www.deeplearning.ai via hn) +7 15w

Apply quantization to shrink a model's memory footprint, then measure the accuracy tradeoff. Fast & Efficient LLM Inference with vLLM Instructor: Cedric Clyburn Earn an accomplishment with PRO - Intermediate - 1h38m - 9 Video Lessons - 3 C…

↯ Vllm vllm
Looking to migrate off of Ollama and LMStudio (www.reddit.com) +79 17w

Hello, I'm currently using Ollama / lm studio for things like code inference and proof reading emails, etc. Definitely not experienced in this space but looking to grow.

↯ Ollama ↯ Llama ↯ Vllm ↯ Gemma ↯ Gemma 4 vllm ollama gemma+3
Qwen 3.6 27B in Claude Code says it will do something then stops and prompts for user reply (not failing a tool call) (www.reddit.com) +714 20w

I'm running Qwen/Qwen3.6-27B-FP8 via vLLM using this command: vllm serve Qwen/Qwen3.6-27B-FP8 --tensor-parallel-size 4 --gpu-memory-utilization 0.95 --max-num-seqs 8 \ --enable-auto-tool-choice --tool-call-parser qwen3_xml \ --enable-prefi…

↯ Vllm ↯ Qwen 3.6 vllm qwen claude-code
Reproduction of TurboQuant (www.reddit.com) +75 22w

There have been many TurboQuant implementations recently in llama.cpp, mlx, vllm, and sglang, but a lot of the discussion and code around them feels pretty noisy and looks to be AI-generated. I’m trying to understand which claims from the…

↯ Llama ↯ Vllm vllm llama
Show HN: MemStitch – Zero-copy context bridging for vLLM (25x TTFT speedup) (github.com via hn) +61 9w

Context-Stitcher 🪡 Zero-Copy Context Bridging Gateway for Multi-Agent GPU Inference. 💡 Core Value Proposition In multi-agent collaborative workflows, separate agents often process the same long text context sequentially.

↯ Vllm vllm
'Am I OpenAI compatible' - a tool and documentation for unified api signatures in open source AI. (www.reddit.com) +61 17w

This has turned out to be useful to many of my friends so I thought I'd share here as well. I created a tool and documentation page for most major open-souce project's adherence to 'OpenAI compatibility' after seeing inconsistencies betwee…

↯ Llama ↯ Vllm vllm llama openai+1
Final Monster: 32x AMD MI50 32GB at 9.7 t/s (TG) & 264 t/s (PP) with Kimi K2.6 (www.reddit.com) +67 20w

32 MI50 32GB setup moonshotai/Kimi-K2.6 int4 @ 9.7 tok/s (output of 136 tok) and 263 tok/s (input of 14564 tok) on vllm-gfx906-mobydick Github link of vllm fork: https://github.com/ai-infos/vllm-gfx906-mobydick Power draw: ~640W (idle) / ~…

↯ Vllm vllm deepseek
Qwen3.6 27B on dual RTX 5060 Ti 16GB with vLLM: ~60 tok/s, 204k context working (www.reddit.com) +64 20w

I’ve been testing Qwen3.6 27B on a pretty non-standard local setup and figured the numbers might be useful for anyone looking at the newer 16GB Blackwell cards. Hardware: 2x RTX 5060 Ti 16GB 32GB total VRAM Proxmox LXC 16 vCPU ~60GB RAM CU…

↯ Vllm ↯ Qwen 3.6 vllm
Qwen3.6 uncensored AWQ (www.reddit.com) +63 21w

I have tested Qwen3.6-27B-Uncensored-HauhauCS-Aggressive-Q5_K_P.gguf on my 4x3090 system (opencode) and find it really good and fast. However, I can't find any uncensored models for vllm (preferably as AWQ).

↯ Vllm ↯ Qwen 3.6 vllm
TurboQuant on MLX & vLLM (www.reddit.com) +65 22w

MLX https://github.com/Blaizzy/mlx-vlm?tab=readme-ov-file#turboquant-kv-cache vLLM https://github.com/vllm-project/vllm/pull/38479 MLX & vLLM users, please share your experience with benchmarks(t/s). Adding llama.cpp Links related to Turbo…

↯ Llama ↯ Vllm vllm llama
Show HN: Run open-weight OCR, VLM and vision models behind one API (www.vlmrun.com via hn) +5 2w

Hey HN. We built an openai-compatible API for running open-weight VLMs, OCR VLMs and ViT-based vision models.

↯ Vllm vllm openai
A production-grade OCR pipeline on Kubernetes with vLLM and Rust (github.com via hn) +5 8w

📄 Production-Grade SLM-Powered OCR Course 📄 Build a self-scaling, event-driven OCR pipeline on Kubernetes (AKS / GKE) with Qwen 3.5 + the GLM-OCR SDK Table of Contents Table of Contents Course Overview Who is this course for? Course Breakd…

↯ Glm ↯ Vllm vllm glm qwen
Qwen 3.6 benchmarks on 2x RTX PRO 6000 (www.reddit.com) +56 16w

Got a chance to play around with 2x RTX PRO 6000 setup so sharing some number for Qwen 3.6. All these were run using latest stable VLLM backend.

↯ Vllm ↯ Qwen 3.6 vllm qwen
For the 5 people here running vLLM on multiple R9700s, you need to patch in support for AITER Unified Attention. (www.reddit.com) +59 20w

I have a 4 x R9700 system on Threadripper pro, but I have never been happy with the performance of my GPUs in vLLM. I have started benchmarking any new model I try out with llama-benchy so that I can get a better idea of how models of diff…

↯ Llama ↯ Vllm ↯ Qwen 3.6 vllm llama
Qwen3.6-35B-A3B KLDs - INTs and NVFPs (www.reddit.com) +52 21w

https://preview.redd.it/c76w57d1yexg1.png?width=1482&format=png&auto=webp&s=1164d8bc3e2e8a4157f26dd5583238a736474932 KLD for INTs and NVFP4s. AS ALWAYS - Use Case is important.

↯ Vllm ↯ Qwen 3.6 vllm
RTX PRO 5000 (48GB) vs MacBook Pro M5 MAX (128GB RAM) - The choice for fine-tuning & agentic coding (www.reddit.com) +527 21w

↯ Llama ↯ Vllm ↯ Fine Tuning vllm fine-tuning llama+1
Show HN: Harbor v0.4.19 – harbor launch –back end vLLM –web codex (github.com via hn) +4 16w

https://github.com/user-attachments/assets/e4897391-c5a8-4391-93c3-9f8b76155f11 Setup your local LLM stack effortlessly. Starts fully configured Open WebUI and Ollama harbor up Now, Open WebUI can do Web RAG and TTS/STT harbor up searxng s…

↯ Ollama ↯ Vllm vllm ollama rag+1
Opinions/improvements for my Qwen3.6-35B-A3B-FP8 + Hermes Agent setup on NVIDIA DGX Spark? (www.reddit.com) +4 17w

I’m running Hermes Agent on a single NVIDIA DGX Spark using vLLM with: docker run --gpus all \ --name qwen36-aggressive \ --restart unless-stopped \ -p 8000:8000 \ --ipc=host \ --ulimit memlock=-1 \ --ulimit stack=67108864 \ --shm-size=32g…

↯ Vllm ↯ Qwen 3.6 vllm qwen openai
Using Intel Arc Pro series, any thoughts ? (www.reddit.com) +41 18w

Simple question: Has anyone run two or more of either of these on Ubuntu ? Intel Arc Pro B70 (32 GB) Intel Arc Pro B65 (32 GB) Running llama or vLLM etc., Any thoughts

↯ Llama ↯ Vllm vllm llama
My own local first ai harness (www.reddit.com) +44 18w

Hi, i just wanted to share what im playing with for last couple weaks. I built my own AI harness: TinyHarness My main goal was low memory footprint, it is not written in Typescript/Javascript/Python, leaving as much memory as possible for…

↯ Ollama ↯ Llama ↯ Vllm vllm ollama llama
New Qwen3.6 27b Autoround Quant (int4) Best Recipe (www.reddit.com) +46 18w

I've been using the int4 Autoround quant from "Lorbus/Qwen3.6-27B-int4-AutoRound" and it has been pretty good! Great quality and performance on an RTX 5090 vllm.

↯ Vllm ↯ Qwen 3.6 vllm
The GB10 Solution Atlas is now open source, the inference engine made for the community with breakneck inference speeds (Qwen3.6-35B-FP8 100+ tok/s) (www.reddit.com) +4 19w

Some of you saw our post a couple weeks back about hitting 102 tok/s stable on Qwen3.5-35B on a DGX Spark. A lot of you asked "cool, where's the code?" Today's the day: Github Atlas is open source.

↯ Vllm ↯ Qwen 3.6 vllm
Whats the latest status on 7900xtx multi-GPU setups? (www.reddit.com) +43 20w

I am currently running dual RTX 5060 ti 16gb (both of which are easy to sell or re-use in other PCs at home) and monitoring the used market for more of the same and alternatively RTX 3090. I couldn't help but notice that sometimes some qui…

↯ Vllm vllm
Mixing 3090 with 3080 20G (modded) for vllm (www.reddit.com) +47 20w

Has anyone tried mixing 3090s with 3080 20G for vllm using tensor parallelism? I know vllm normally discourages mixing GPUs, but given how much 3090 is selling nowadays, the modded 20G 3080s with half the price feel like better deals.

↯ Vllm vllm
Best settings for Qwen 3.6 -27B for 2X3090? (cannot make it to be smarter than Qwen 3.6 35B-A3B! (www.reddit.com) +41 21w

I'm sure people have asked before for settings for these gpu's, but for me, no matter what I do, It doesn't work as good as 3.6 35B! I've tried VLLM and LLAMACPP .

↯ Vllm ↯ Qwen 3.6 vllm qwen
Show HN: Aide – A customizable Android assistant (voice, choose your provider) (aideassistant.com via hn) +41 21w

↯ Ollama ↯ Vllm vllm ollama chatgpt+1
Free hands-on lab: build a ReAct agent 3 ways (create_agent, raw LangGraph with tool-call budget, NVIDIA NAT YAML) (www.reddit.com) +43 21w

↯ Vllm ↯ Dpo dpo vllm agentic
current: 1x 16GB 5060Ti. worth a 2nd for OpenCode? (www.reddit.com) +49 22w

my current build is just a 16GB 5060Ti running on a 3800X with 32GB DDR4. not really anything special, but I only really use it right now for Qwen3-VL-8B-Instruct at INT8 to do handwriting transcription (and it works great for that). someo…

↯ Llama ↯ Vllm ↯ Qwen 3.5 vllm llama
Show HN: OSS, K8s-native AI platform for distributed multi-model inference (github.com via hn) +3 2w

Hi everyone, I’m one of the co-founders of axem. We recently open sourced Shaide, a project we’ve been working on to make running multiple LLMs on your own infrastructure less painful.

↯ Vllm vllm
New OllamaMQ v0.3.0 (news.ycombinator.com) +3 4w

- loading / unloading models - ollama, lm-studio, vllm - optional security tokens and visibility and many more on - https://github.com/Chleba/ollamaMQ

↯ Ollama ↯ Vllm vllm ollama
Lossless Inference (runinfra.ai via hn) +3 6w

Quantization became the default way to speed up inference because it is the easiest one. In vLLM it is one flag or one swapped checkpoint (docs.vllm.ai), while doing the real optimization work like writing better kernels takes months.

↯ Vllm vllm
Why self-hosted inference is essential (www.redhat.com via hn) +3 7w

Learn about Red Hat's approach to self-hosted inference with vLLM, addressing the reliability gap between open-weight models and hosted frontier models for agentic workloads.

↯ Vllm vllm agentic
Profile(v2.1.4) physics-aware optimizer for vLLM (31→470 tok/s on A100) (github.com via hn) +3 13w

Profile A physics-grounded, cost-aware optimization loop for vLLM inference servers. The Problem: vLLM exposes numerous configuration flags (e.g., --max-num-seqs, --enable-prefix-caching).

↯ Vllm vllm
Running Gemma4 31b-it on vLLM 0.21.0 A100s (bad quality or what am I doing wrong) (www.reddit.com) +31 16w

Okay fun time I got access to two Nvlinked A100s for some research project I benchmarked my work against the Gemma 4 31b-it available through Google, but my dataset is rather massive, so I need to run it on the "local" resources. Basically…

↯ Vllm ↯ Gemma ↯ Gemma 4 ↯ Gemma 4 ↯ Gemma 4 ↯ Gemma 4 vllm gemma openai
Output Length Constrained Summarization using GRPO on tiny LLMs | smolcluster (www.reddit.com) +3 16w

Just released a blog on a side research project I have been doing for the past two months and would love for you all to check out and see how it is! It's about output length-constrained summarization using LLMs with GRPO.

↯ Vllm vllm qwen
Run Chrome’s tiny Gemma4 (aka Gemini Nano) directly on PC without GPU (www.reddit.com) +36 17w

Everyone remembers that sneaky download of Gemini Nano earlier this month? and if you talk to it, it will happily tell you it’s a Gemma.

↯ Llama ↯ Vllm ↯ Gemma ↯ Gemma 4 vllm gemma llama+1
LLMKube – A Kubernetes operator for local LLMs across Nvidia and Mac fleets (llmkube.com via hn) +3 17w

Run production LLMs on your own hardware A Kubernetes operator for self-hosted LLM inference. vLLM, llama.cpp, TGI, NVIDIA, Apple Silicon.

↯ Llama ↯ Vllm vllm operator llama
club-5060ti follow-up: cleaner RTX 5060 Ti local LLM recipes, benchmark explorer, and CUDA GPU compatibility notes (www.reddit.com) +35 17w

I posted earlier about RTX 5060 Ti local LLM testing, and I have cleaned the repo up quite a bit since then. The project is now a more structured benchmark/recipe repo rather than scattered notes.

↯ Llama ↯ Vllm ↯ Qwen 3.6 vllm llama
While waiting for Fara-1.5 for my coding harness (www.reddit.com) +31 17w

Hi all, Not sure many people are aware so wanted to give a word about Fara-1.5 release. => this release will likely be the big sister of Fara-7B and built on top of Qwen3.5 Actual Fara-7B performs not bad at all but actually requires a pro…

↯ Llama ↯ Vllm ↯ Qwen 3.6 vllm llama
RTX 5060Ti 16GB or RTX 3080 20GB? (www.reddit.com) +319 18w

I would like to dedicate a budget of about 500 euros to upgrade my workstation and run inference on the qwen 3.6 27b and gemma 4 31b models. I currently have an RTX 5060Ti 16GB.

↯ Copilot ↯ Llama ↯ Vllm ↯ Gemma ↯ Qwen 3.6 vllm gemma copilot+2
Should I sell my RTX3090s? (www.reddit.com) +331 19w

I have a GPU server (4 × RTX3090s) that I've been using for research and PoC in the past 2 years. Mostly running vLLM for Qwen, GPT-OSS, and Gemma.

↯ Vllm ↯ Gemma vllm gemma qwen
Load balancer for vLLM server instances? (www.reddit.com) +31 20w

Hello all, the docs for the vLLM production stack suggested autoscaling the vllm worker instances based on the number of waiting requests, but it seems like this would only help with new coming requests? We are having burst LLM calls which…

↯ Vllm vllm
Will llama.cpp multislot improve speed? (www.reddit.com) +36 20w

I've heard mostly bad opinions about multiple slots with llama.cpp (--parallel > 1). I guess comparing to vLLM it might be worse at this, but I recently tried vLLM on 4 slots and it indeed improved the overall speed significantly (150-170t…

↯ Llama ↯ Vllm vllm llama
What are your most interesting and hard Vision use cases? I plan to do side by side comparison of Gemma 4 (31B) vs Qwen 3.6(27B) Vision and I look for inspiration (www.reddit.com) +318 21w

Hey guys, I built a custom vLLM pipeline to run Gemma 4 (31B FP8) and Qwen 3.5 side-by-side locally to see how they actually perform in the wild with preprocessing of audio and images. But of course new model Qwen 3.6 27B came out just whe…

↯ Vllm ↯ Gemma ↯ Qwen 3.6 vllm gemma qwen
DeepSeek V4 in vLLM: Efficient Long-Context Attention (vllm-website-pdzeaspbm-inferact-inc.vercel.app via hn) +3 21w

DeepSeek V4 in vLLM: Efficient Long-context Attention We are excited to announce that vLLM now supports the DeepSeek V4 family of models (deepseek-ai/DeepSeek-V4-Pro and deepseek-ai/DeepSeek-V4-Flash ). These models feature an efficient lo…

↯ Vllm ↯ DeepSeek 4 vllm deepseek
Gemma 4 vs Qwen 3.5 Vision on vLLM — 5 things I learned benchmarking them side-by-side (Reasoning budgets, FP8, pre-processing the input). (www.reddit.com) +32 21w

Hi guys, I’ve been running side-by-side experiments on Gemma 4 (31B FP8) and Qwen 3.5 Vision for the last few days using vLLM in Docker to see how they actually handle real-world images and video. A few things I found out: 1.

↯ Vllm ↯ Gemma ↯ Qwen 3.5 vllm gemma qwen
Trained a Qwen2.5-0.5B-Instruct bf16 model on Reddit post summarization task with GRPO written from scratch in PyTorch - updates! (www.reddit.com) +32 22w

So, yesterday run was a success and I did get an avg rollout length of about 64 tokens as attached in the image! This was with quality_reward + length_penalty (more info below!) Next, I'll be going with length penalty as the reward and wit…

↯ Vllm ↯ Qwen 2.5.0 vllm
DGX Spark just arrived — planning to run vLLM + local models, looking for advice (www.reddit.com) +35 22w

Just got a DGX Spark set up today and starting to configure it for local LLM inference. Plan is to run: • vLLM • PyTorch • Hugging Face models as a local API backend for an application I’m building (education / analytics use case, trying t…

↯ Vllm vllm
Deep Dive into Efficient LLM Inference with Nano-vLLM (cefboud.com via hn) +3 22w

Deep Dive into Efficient LLM Inference with nano-vLLM A look inside a lightweight implementation of vLLM. KV cache, paged attention, tensor parallelism &multi-GPU support, etc.

↯ Vllm vllm
Turboquant in vllm kv cache - how to implement ? (or any other rotational kv cache) (www.reddit.com) +37 22w

Hi folks - is there any "standard" (acceptable) vllm way of implementing turboquant or a similar rotational quant for vllm's kvcache? I found https://github.com/mitkox/vllm-turboquant - but this seems inactive.

↯ Vllm vllm
Using older vLLM version via Docker -- how do you use GGUF quants? (www.reddit.com) +31 22w

So vLLM recently added the feature to use GGUF quants with the syntax author/model:quant format. I was just wondering if people were able to use the quants on older vLLM versions.

↯ Vllm vllm
Going local with old GPUs (www.reddit.com) +311 22w

I'm an ex crypto miner with remnant mining parts so I threw them together into a franken hydra case. I've been using claude oath previously, but they just shut that door last week or so.

↯ Ollama ↯ Vllm vllm ollama
Show HN: Don't trust me bro: fixing GPT-OSS (3.49B tokens, 1k GPU hours, 1x3090) (github.com via hn) +2 2w

Long story short, about a year ago, in spite of everybody bashing gpt-oss for broken tool calling and refusals, I thought there's something there worth exploring. Model hit a sweet spot for me in that it was the first time I could run full…

↯ Llama ↯ Vllm vllm llama openai
Tencent Hy4 Preview LLM (github.com via hn) +2 3w

中文｜ English 🖥️ Official Website | 💬 GitHub Table of Contents Model Introduction A New Flagship Generation Built for Productivity Benchmark Appendix Known Limitations News Model Links Quickstart Deployment vLLM SGLang Finetuning Quantizati…

↯ Vllm vllm
vLLM-iOS: 88% Faster Multi-Agent Inference on iOS (jonready.com via hn) +23 3w

TL;DR: I improved multi-agent batching on iOS by implementing vLLM-style continuous batching in native Swift on MLX. I've written a lot about the tokenomics of local LLMs and agent swarms.

↯ Vllm vllm
Native vLLM and ROCm 7.15 for RX 6000 (RDNA2) on Windows 11 – 26 Tflops FP16 (github.com via hn) +2 4w

vllm-rocm-windows-RDNA2-oneclick []() []() []() []() []() Native vLLM + ROCm 7.15 (TheRock) for the whole AMD Radeon RDNA2 family on Windows — no WSL2, no NVIDIA, no compiler. One-click installer, everything prebuilt, and an OpenAI-compati…

↯ Vllm vllm openai
GenOffice fork that works with any local LLM instead of a cloud account (github.com via hn) +2 4w

GenOffice (local-LLM fork) A free, open-source AI Office suite — this fork drops the cloud-account requirement and talks to any OpenAI-compatible endpoint instead: a local server (Ollama, LM Studio, vLLM, llama.cpp server, text-generation-…

↯ Ollama ↯ Llama ↯ Vllm vllm ollama llama+1
Show HN: Kvcachescope – Why Nvidia-smi is blind to vLLM KV cache leaks (github.com via hn) +2 5w

kvcachescope A logical memory profiler and state inspector for PagedAttention inference engines (vLLM, SGLang). Standard GPU profilers (nvidia-smi, nsys, torch.cuda.memory_allocated()) observe physical VRAM allocations at the PyTorch tenso…

↯ Vllm vllm
Show HN: Proxima serves 4x more requests with no hardware change on vLLM (github.com via hn) +21 5w

hey everyone, i decided to make a vLLM plugin that implements the Star-KV paper. the results are quiet promising with a decode kernel thats faster than FA2 in higher batch sizes.

↯ Vllm vllm
Show HN: Open-source verification and tuning layer for self-hosted LLMs (github.com via hn) +2 9w

Hi HN, Thomas here, I'm a Machine Learning, and I've been deploying models in prod for the last 5 years and decided it was time to open-source some of my learnings! In Coloma you'll find my vLLM tuner, I use it to ensure models don't OOM a…

↯ Vllm vllm
Veta: AI agent that QA-tests Android apps (github.com via hn) +2 9w

# Veta Autonomous AI agent swarm for visual, functional, and accessibility testing of Android apps and mobile web — 100% of AI inference runs on AMD GPUs (Fireworks AI on AMD Instinct / self-hosted vLLM on ROCm), with containerized Android…

↯ Vllm vllm
vLLM prefill paired with TileRT decode (vllm.ai via hn) +2 9w

vLLM x TileRT: Specialized Decode for Latency-Critical Serving Disaggregated serving, which separates the compute-bound prefill phase from the memory-bandwidth-bound decode phase, has become an increasingly standard pattern for serving lar…

↯ Vllm vllm
AKM-CLR – pre-inference governance for vLLM-style LLM serving (github.com via hn) +2 10w

AKM-CLR — Tenant/Task Governance Layer for Shared LLM Serving AKM-CLR is a validated prototype governance/control layer for multi-tenant LLM infrastructure. It sits above OpenAI-compatible serving engines such as vLLM and helps decide whet…

↯ Vllm vllm openai
How to become an AI infrastructure engineer? (news.ycombinator.com) +2 13w

Hi, I currently work on a GenAI platform for one of the largest local industrial companies. My daily work mostly involves building inference infrastructure on top of a 48x H200 GPU, Kubernetes and vLLM.

↯ Vllm vllm
IndexCache: Accelerating Sparse Attention via Cross-Layer Index Reuse (github.com via hn) +21 13w

IndexCache Accelerating Sparse Attention via Cross-Layer Index Reuse Tsinghua University & Z.ai This repository provides a patch for SGLang and vLLM that enables IndexCache inference acceleration for models using DeepSeek Sparse Attention…

↯ Vllm vllm deepseek
Sors: a Rust proxy that reorders prompts to maximize vLLM prefix cache hits (github.com via hn) +2 13w

Sors - reorders prompts for LLM to maximize prefix cache hit. A minimal reverse proxy that reorders prompt content to maximize prefix cache hits in LLM inference engines (vLLM, SGLang, or any OpenAI-compatible backend with prefix caching e…

↯ Vllm vllm openai
67% Cost Savings with PD Disaggregation Using Ray and vLLM on AMD MI325X (www.anyscale.com via hn) +2 13w

Achieving Up to 67% Cost Savings with Prefill-Decode Disaggregation Using Ray + vLLM on AMD MI325X In LLM serving, the optimization objective is deceptively simple: given a set of latency SLA targets – time to first token (TTFT), time per…

↯ Vllm vllm
Fast and Efficient LLM Inference with vLLM: A New Course with Deeplearning.ai (vllm.ai via hn) +2 15w

Fast & Efficient LLM Inference with vLLM: A New Course with DeepLearning.AI We're excited to announce, with Red Hat and Andrew Ng's DeepLearning.AI, a hands-on course that walks through LLM fundamentals and the full optimize, deploy, and b…

↯ Vllm vllm
Local run for multi users: which software set? (www.reddit.com) +211 16w

Context: I am testing and running local LLM on Linux for some months, first with llama.cpp and now with vLLM for better concurrent capabilities. I use llama-swap in front of either vLLM or llama.cpp in order to have thinking and non-thinki…

↯ Llama ↯ Vllm vllm llama
Looking for a working Deepseek-v4-Flash quant (www.reddit.com) +25 16w

Best I tried so far is https://huggingface.co/nsparks/DeepSeek-V4-Flash-FP4-FP8-GGUF with the custom llama.cpp fork, but it suffers from low quality and random incoherent output. VLLM wouldn't support anything other than H100s for DS4.

↯ Llama ↯ Vllm ↯ DeepSeek 4 ↯ DeepSeek 4 vllm deepseek llama
Looking for Suggestions — Single 5090 & 64gb DDR5 (www.reddit.com) +211 16w

Hi Reddit, I am planning on running Qwen 3.6 27b NVFP4 via vLLM on my 5090 but was wondering if something like 35b a3b at Q8 on Llama would produce better results for agentic coding and utilize the system memory. My research says no but if…

↯ Llama ↯ Vllm ↯ Qwen 3.6 vllm llama qwen+1
Harbor v0.4.19 - vllm/sglang/llama.cpp launch codex/claude/pi/opencode (www.reddit.com) +21 16w

I'm usually not posting about Harbor releases out of the respect for the community here, but I think v0.4.19 might save a lot of people some time. Harbor can now launch your local agentic coding tools with local inference backends.

↯ Llama ↯ Vllm ↯ Qwen 3.5 vllm llama codex+1
I Built MagesticAI. A Cloud Web-Based Agentic DevOps Orchestrator that actually helped me develop Itself. (www.reddit.com) +22 16w

Posted on other feeds last week and figured some of you out here might be interested as well; Someone commented asking if it supported OpenAI-compatible endpoints (LM Studio, vLLM, OpenRouter, Together, Groq, LocalAI…), so i have spent few…

↯ Ollama ↯ Vllm vllm ollama gemini+3
Best coding model on RTX 3060 (www.reddit.com) +22 16w

Wondering what’s the best coding model that can fit on a RTX 3060 (12GB). Has anyone been able to do something useful with it?

↯ Llama ↯ Vllm vllm llama
numind/NuExtract3 · Hugging Face (huggingface.co via reddit) +2 16w

NuExtract3 is a unified 4B vision-language reasoning model for document understanding. It combines strong structured information extraction with high-quality image-to-Markdown conversion, making it suitable for extraction pipelines, OCR, a…

↯ Vllm vllm rag
What workstation to get for ~13k EUR? (www.reddit.com) +28 16w

My use-cases will be to test open-weight LLMs and work on harnesses, inference systems and possibly other non-ML workflows (CS-related) in the future. Fine-tuning would not be something I do locally because I can rent a B200 from RunPod fo…

↯ Llama ↯ Vllm ↯ Minimax ↯ Fine Tuning ↯ DeepSeek 4 minimax vllm fine-tuning+2
Do smaller quants silently break tool calls / JSON output? (www.reddit.com) +25 17w

I posted recently about EvalShift, an OSS CLI for regression-testing LLM model changes. A few people pointed out that for LocalLLaMA, the more interesting use case may be quantization regression: Q8 -> Q4_K_M Same base model, same prompts,…

↯ Ollama ↯ Llama ↯ Vllm vllm ollama llama
5060ti chads -> gemma-4-31b-it-nvfp4 + vllm + mtp (www.reddit.com) +22 17w

Hey all, While nvfp4 still seems to be a work in progress, the latest version of vllm 0.21 finally has mtp working for gemma. With all the talk of qwen being badass I thought I would revisit gemma.

↯ Vllm ↯ Gemma ↯ Gemma 4 vllm gemma qwen
Is it possible to exclusively use a draft model for reasoning to speed up generation? (www.reddit.com) +218 18w

EDIT: Edited to provide more clarity It occurred to me, that perhaps the same draft model used for speculative decoding would be completely adequate if we just used it's output as-is for reasoning, without validating the results against th…

↯ Llama ↯ Vllm vllm llama
TensorRT-LLM vs vLLM vs llama.cpp on NVIDIA DGX Spark? (www.reddit.com) +2 18w

I am looking for recommendations on the best way to run local LLMs on NVIDIA DGX Spark. Which stack makes the most sense in practice: TensorRT-LLM, vLLM, or llama.cpp?

↯ Llama ↯ Vllm vllm llama
Show HN: Granite Switch - compose multiple LoRA adapters to one deployable model (github.com via hn) +2 19w

Granite Switch is an open-source IBM Research project for composing several task-specific LoRA adapters into a single deployable Granite model checkpoint. The idea is to get the accuracy benefits of multiple fine-tuned models without havin…

↯ Vllm vllm
[Help] Running big dense models faster (www.reddit.com) +26 20w

I have been trying Mistral 3.5 on my 4x RTX 3090 rig with llama.cpp. Inference is slow (about 11 t/s) even without anything being offloaded to the CPU.

↯ Llama ↯ Vllm ↯ Mistral ↯ Mistral 3.5 mistral vllm llama+1
What’s up with mobile LLMs? (www.reddit.com) +26 20w

I see a lot of support for running LLMs on PCs with ollama to vLLM. Whats the current state for running on mobile?

↯ Ollama ↯ Vllm vllm ollama
Ubuntu 26.04 vs 24.04 speed improvements for inference? (www.reddit.com) +24 20w

I'm curious if any brave soul has upgraded their computer (especially if it's Strix Halo) from Ubuntu 24.04 -> 26.04 and seen a significant performance improvement for inference with VLLM, llama-server, and/or LM Studio.

↯ Llama ↯ Vllm vllm llama
Does anyone have a usable vLLM setup with Qwen3.6 27B + pipeline parallelism + MTP? (www.reddit.com) +28 20w

I'm a daily llama-cpp user and was hoping to try MTP on vLLM. Unfortunately, pipeline parallelism + MTP does not seem to work with this model in vLLM.

↯ Llama ↯ Vllm ↯ Qwen 3.6 vllm llama
To run deepseek v4 flash how much max vram we need? 175 gb or 320gb? (www.reddit.com) +2 21w

As far as i know the weight is of 160gb + 9.6gb needed for max 1 million token window + 5 gigs overhead = 175gb vram. But vllm and othere sources said "To use the full 1M context, you need 4x A100 80G" --> thats a 320gb vram ??

↯ Vllm ↯ DeepSeek 4 vllm deepseek
LLM performance benchmarking update (www.reddit.com) +22 21w

↯ Vllm vllm
Please help me pick the right Qwen3.5-27B format/quant for RTX5090 (www.reddit.com) +21 22w

Hi all, first post here. I've started a project in OpenClaw a month ago, and it's been a very "intense" 4 weeks to say the least...

↯ Vllm ↯ Qwen 3.5 vllm openclaw sonnet+2
Anybody got Qwen3.5-27B working with Intel Arc B70 (or similar) and proper optimization? (www.reddit.com) +215 22w

I am playing around with Intel Arc B70, still trying to decide whether I keep it or not. After some battle, I got it working with Radeon 5500 and B550M, now I am on to the fun part of getting software to work.

↯ Llama ↯ Vllm ↯ Qwen 3.5 vllm llama
Show HN: Fleet Watch – preflight guard for local AI inference on Apple Silicon (github.com via hn) +21 22w

Fleet Watch Process governance for AI workloads on a single machine. The Problem You're running MLX, Ollama, vLLM, Candle/Cake, experiment runners, and AI coding agents on the same machine.

↯ Ollama ↯ Vllm vllm ollama
Dynamic tool lists vs KV cache: how do you handle this trade-off in LLM agents? (www.reddit.com) +28 22w

I’m working on an LLM agent setup (using Qwen-style chat templates with tool calling), and I ran into a design trade-off that I’d like to get some insights on. In these templates, the full tool definitions (JSON schemas) are injected into…

↯ Vllm vllm qwen mcp
Keeping vLLM's Prefix Cache Warm Between Agent Turns (doug.sh via hn) +1 1d

Keeping vLLM's Prefix Cache Warm Between Agent Turns From 55% to 95% cached 🔗 I’ve been playing with a few different ways to host Qwen3.8 locally. I’m aiming for something that can replace Claude Code for most of my tasks.

↯ Vllm ↯ Qwen 3.8 vllm claude-code
Show HN: Self-adjusting vLLM at production scale (twitter.com via hn) +1 4d

𝗥𝘂𝗻𝗻𝗶𝗻𝗴 @vllm_project 𝗶𝘀 𝗲𝗮𝘀𝘆. 𝗞𝗲𝗲𝗽𝗶𝗻𝗴 𝗹𝗮𝘁𝗲𝗻𝗰𝘆 𝗽𝗿𝗲𝗱𝗶𝗰𝘁𝗮𝗯𝗹𝗲 𝘄𝗵𝗲𝗻 𝟯𝟬𝟬 𝗰𝘂𝘀𝘁𝗼𝗺𝗲𝗿𝘀 𝗮𝗿𝗿𝗶𝘃𝗲 𝗮𝘁 𝗼𝗻𝗰𝗲 𝗶𝘀 𝘁𝗵𝗲 𝗵𝗮𝗿𝗱 𝗽𝗮𝗿𝘁.

↯ Vllm vllm
Neural Nova – GPU optimization benchmarks for LLM workloads (www.neural-nova.com via hn) +1 9d

Performance Benchmarks Explore Optimized AI models recipes across GPUs, frameworks, and deployment configurations. BENCHMARKED MODELS REASONING vLLM · 8× NVIDIA H100-80GB Qwen3-235B-A22B +138.7% token/s +58% Cost Savings Qwen3-235B-A22B ·…

↯ Vllm ↯ Qwen 3 vllm
Public vs. Private LLM (wirt.ee via hn) +1 10d

LiteLLM in front of vLLM Context: the vLLM serving entry covers the engines. This is the front door — a LiteLLM proxy in front of them: one OpenAI-compatible endpoint, virtual keys per consumer, spend tracking, health-check routing between…

↯ Vllm vllm openai
Show HN: MaskShift – a maximalist coding agent with zero NPM dependencies (github.com via hn) +1 13d

MaskShift is a local-first coding agent harness. Features: - Works with models that have no tool-calling API.

↯ Copilot ↯ Ollama ↯ Vllm tool-calling vllm ollama+6
trie stands for trace replay inference evaluation (github.com via hn) +1 2w

trie trie stands for trace replay inference evaluation and is a lightweight benchmarking harness that exercises OpenAI-compatible inference servers with synthetic workloads derived from production traces. It targets backends like vLLM, SGL…

↯ Vllm vllm openai
Show HN: Building AI agents client-side JavaScript (news.ycombinator.com) +12 2w

Hey HN, most agent systems default to server-side Python inside containers and chain frameworks. I wanted to see how far we could push agent in the browser with vanilla JavaScript https://buttercup.sh The reason this is interesting is beca…

↯ Ollama ↯ Vllm ↯ Function Calling function-calling vllm ollama
Hot reload vLLM and sglang configs (github.com via hn) +1 2w

trimtab Change SGLang and vLLM scheduler settings while the server runs. No restart, weights never leave the GPU.

↯ Vllm vllm
Efficient Decode Context Parallelism with vLLM for Long Context Workloads (vllm.ai via hn) +1 3w

Efficient Decode Context Parallelism with vLLM for Long Context Workloads 1. Introduction Long-context inference is becoming essential for agentic AI, where assistants may need to reason over large code repositories and long chat histories.

↯ Vllm vllm agentic
Four reproducible vLLM parser failures that return 200 with the wrong tool call (ingot.tools via hn) +1 3w

vLLM can drop or garble a tool call and still return 200 Four reproducible parser failures in vLLM 0.26.0, 0.27.1 and 0.28.0, run on CPU with no GPU or weights. As of 2026-08-27 one is fixed on main but not in any release; the rest are ope…

↯ Vllm vllm
LLMPanel Deploy vLLM to RunPod or Vast.ai Without Kubernetes (llmpanel.io via hn) +1 3w

Deploy LLMs on any GPU, anywhere Pick a model, pick a GPU, hit deploy. LLMPanel provisions the container, exposes an OpenAI-compatible endpoint, and streams every GPU metric back to one dashboard — on your own hardware or any cloud.

↯ Vllm vllm openai
Ask HN: Are there any production LLM pipeline setups to learn from? (news.ycombinator.com) +1 5w

I believe vLLM has been the gold standard for inferencing but I'd like to learn how vLLM, langgraph, langfuse, other observability and business logic is meld into a single setup.

↯ Vllm vllm
Inferact vLLM creators are hiring (twitter.com via hn) +1 5w

The @vllm_project maintainers at @inferact 🚀 are some of the most cracked engineers in the world. They’re building one of the inference engines that powers much of the world’s intelligence—and doing so with remarkable dedication, kindness,…

↯ Vllm vllm
vLLM Serving Experiments on H100s – config beats the baseline on p95 TTFT,ITL (efficientagent.substack.com via hn) +1 5w

A Better Knob Beats More Silicon: 12 vLLM Serving Experiments on H100s Doubling my GPU count raised throughput 61% — and still lost to a one-line config change. A roofline-guided tour of what actually moves production serving metrics.

↯ Vllm vllm
LettuceDetect v2 in Semantic Router: Gen. Hallucination Detection vLLM Endpoint (vllm-sr.ai via hn) +1 6w

LettuceDetect v2 in Semantic Router: Generative Hallucination Detection as a vLLM Endpoint Semantic Router can now verify grounded responses with a generative span detector served by vLLM. The new endpoint detector backend runs LettuceDete…

↯ Hallucination ↯ Vllm vllm hallucination
C++ Version of vLLM (github.com via hn) +1 6w

Same tokens as vLLM. Same throughput.

↯ Vllm vllm
DeepSeek-V4-Flash-0731-Latent-Reasoning. A model thinking in latent space (blog.n.ichol.ai via hn) +11 6w

DeepSeek-V4-Flash-0731-Latent-Reasoning. A self-contained model that does thinking in latent space, NVFP4-quantized, with a production vllm form for serving runtime.

↯ Vllm ↯ DeepSeek 4 vllm deepseek
The Inference Engine Guide for K3 Deployment (twitter.com via hn) +1 6w

Kimi K3 is here: a 2.8T-parameter MoE with a 1M-token context and native vision, running on vLLM from day 0. Here is our canonical deployment guide: architecture, kernels, recipes, and the flags to run it in production.

↯ Vllm vllm moe
Show HN: Gainz.fast – Local Inference, Faster (gainz.fast via hn) +1 6w

Come help push the frontier of token speed across local models and hardware with your agents! Current frontier Laguna XS 2.1 · AMD R9700 (llama.cpp HIP) +31.14% 143.3 tok/s Laguna XS 2.1 · DGX Spark GB10 (vLLM NVFP4) +5.28% 37.3 tok/s Lagu…

↯ Llama ↯ Vllm ↯ Qwen 3.8 ↯ Qwen 3.8 ↯ Qwen 3.8 ↯ Qwen 3.8 ↯ Qwen 3.8 vllm llama
Kimi K3 on vLLM: Up to 370 Tokens/sec (vllm.ai via hn) +1 7w

Kimi K3 Is Here: Efficient Day-0 Support on vLLM We're thrilled to announce efficient day-0 vLLM support for Kimi K3, one of the most powerful open-weight models ever released. Last week, we previewed the production-scale integration work…

↯ Vllm vllm
AI Firewall – Security Gateway and Reverse Proxy for LLM Traffic (github.com via hn) +1 8w

🛡️ AI Firewall — Security Gateway & Reverse Proxy for LLM Traffic AI Firewall is a security gateway that sits between your applications and LLM APIs (OpenAI, Anthropic, Gemini, Ollama, vLLM). It inspects prompts before they leave your peri…

↯ Ollama ↯ Vllm vllm ollama gemini+2
Hy3 (295B A21B), a reasoning and agent model with great cost efficiency (github.com via hn) +1 10w

中文｜ English 🖥️ Official Website | 💬 GitHub Table of Contents Model Introduction Stronger Agent Capabilities More Reliable Product Experiences Benchmark Appendix News Model Links Quickstart Deployment vLLM SGLang Finetuning Quantization Li…

↯ Vllm vllm
Automatic Prefix Caching – vLLM (docs.vllm.ai via hn) +1 11w

Automatic Prefix Caching¶ Prefix caching kv-cache blocks is a popular optimization in LLM inference to avoid redundant prompt computations. The core idea is simple – we cache the kv-cache blocks of processed requests, and reuse these block…

↯ Vllm vllm
Show HN: ParseHawk – 100% Local Document AI with API, CLI, and Web UI (github.com via hn) +1 12w

I just released ParseHawk v0.1.0: Apache-2.0 licensed 100% local document AI platform that extracts JSON from PDFs, images etc. It builds on top of NuMind's NuExtract3 but additionally enforces a provided JSON schema with constrained decod…

↯ Vllm vllm
vLLM Recipes (recipes.vllm.ai via hn) +1 12w

vLLM Recipes — Deploy any model on any hardware with vLLM Pick a model, adjust for your GPUs, copy the vllm serve line that runs. Community-maintained recipes for NVIDIA H100/H200/B200/B300, Grace-Blackwell, and AMD MI300X/MI325X/MI355X.

↯ Vllm vllm
petite-vllm Part 2: KV Cache & Paged Attention (kristenmcintosh.dev via hn) +1 13w

petite-vllm Part 2: KV Cache & Paged Attention Building an LLM Serving Engine from Scratch · 12 min readKV Cache In part 1 we implemented a simple autoregressive loop and LLM interface. There was no KV caching, which means that at each tok…

↯ Vllm vllm
Running GLM-5.2 5x faster at 500tps with limitation (abhishek.it via hn) +1 13w

Running GLM-5.2 5× faster than vLLM, on a runtime that doesn't support it I rented an 8×B200 and tried to run GLM-5.2 on TileRT, the runtime MiMo used to push a 1T model past 1000 tok/s. TileRT doesn't support GLM-5.2, so I reverse-enginee…

↯ Glm ↯ Vllm ↯ GLM 5.2 vllm glm
Monitoring LLM Inference with Prometheus and Grafana (vLLM, TGI, Llama.cpp) (www.glukhov.org via hn) +1 13w

Monitor LLM Inference in Production (2026): Prometheus & Grafana for vLLM, TGI, llama.cpp Monitor LLM with Prometheus and Grafana LLM inference looks like “just another API” — until latency spikes, queues back up, and your GPUs sit at 95%…

↯ Llama ↯ Vllm vllm llama
Nvidia DGX Spark GB10 – AI Models and Guide with vLLM and Autonomous Script (github.com via hn) +1 15w

NVIDIA DGX Spark GB10 — AI Models & Inference Guide Welcome to my repository and guide for running, optimizing, and benchmarking state-of-the-art AI models on the NVIDIA DGX Spark deskside supercomputer, powered by the cutting-edge NVIDIA…

↯ Vllm vllm
vLLM: An Efficient Inference Engine for Large Language Models [pdf] (www2.eecs.berkeley.edu via hn) +1 15w

vLLM: An Efficient Inference Engine for Large Language Models by Woosuk Kwon A dissertation submitted in partial satisfaction of the requirements for the degree of Doctor of Philosophy in Computer Science in the Graduate Division of the Un…

↯ Vllm vllm
Show HN: LLMhop – A tiny, stateless router for LLMs with a NixOS module (github.com via hn) +1 15w

LLMhop is a tiny stateless proxy for LLM inference servers. It tackles an issue I faced when trying to serve more than one local LLM at once which is not natively supported by vLLM.

↯ Llama ↯ Vllm vllm llama
Nvidia H100(94GB VRAM) - should I run llama.cpp or vllm for 30 users inference? (www.reddit.com) +11 16w

I was given the great opportunity to borrow a H100 with 94GB VRAM at work until it is needed by a customer. (No idea how much system ram I will get, but I guess they are a bit flexible on this).

↯ Llama ↯ Vllm ↯ Qwen 3.6 vllm llama agentic
Sharing INT4-W4A16 version of Jackrong/Qwopus3.6-27B-v2 for VLLM/SGLang users (www.reddit.com) +11 16w

link: https://huggingface.co/JC1DA/Qwopus3.6-27B-v2-INT4-W4A16-Autoround Super surprised how good Jackrong's model is... It's taking so much time to evaluate the all the base qwen3.6-27B, Jackrong's version and other's quantized models but…

↯ Vllm ↯ Qwen 3.6 vllm
$340 opus bill made me rethink how I route agent tool calls (www.reddit.com) +15 17w

Looked at my coding agent's bill last month: $340 for repo maintenance across three repos, each around 15k lines. Most of those tool calls were just grep and file reads.

↯ Haiku ↯ Vllm ↯ DeepSeek 4 vllm haiku deepseek+1
Cannot get NCCL test to run in docker with 2 x 6000 Pro connected x8 to AM4 CPU (www.reddit.com) +11 17w

nvidia-smi topo -m is showing the both GPU as PHB (i.e. via CPU) connected as expected but I cannot get NCCL all_reduce_perf to run at all, it always hangs after starting up.

↯ Vllm vllm
40+tok/s - optimized recipe for Qwen 3.5 122B Int4 on a single DGX Spark with vLLM (www.reddit.com) +14 17w

Hello guys, two days ago i ran the spark-arena for my Qwen 3.5 122B Recipe on a single DGX Spark and I got the highest score on speed for any context length and concurrency across all 3.5 122B Int4 Recipes. Just wanted to share if somebody…

↯ Vllm ↯ Qwen 3.5 vllm qwen
Need help getting 7900 XTX PyTorch performance metrics (www.reddit.com) +11 17w

I'm on a quest to profile and benchmark different GPUs for PyTorch, vLLM, and llama.cpp. Cannot find the high-end AMD consumer cards for rent anywhere online and interested in the PyTorch ROCm performance of the 7900 XTX (if you want to co…

↯ Llama ↯ Vllm vllm llama
Benchmarking vLLM vs SGLang vs llama.cpp on a mixed Blackwell/Ada cluster (www.reddit.com) +1 17w

I have been running some benchmarks on a heterogeneous 7-GPU cluster to see how different inference engines handle long context prefill using pipeline parallelism. My setup consists of a mix of Blackwell and Ada cards: one RTX PRO 6000 96G…

↯ Llama ↯ Vllm vllm llama
KV Cache Is Becoming the Memory Hierarchy of Inference (touchdown-labs.com via hn) +1 17w

A briefing on the inference memory hierarchy: prompt layout, host-side shared KV, distributed lookup, RDMA transfer, encoder reuse, and evidence discipline. Covers vLLM × Mooncake, LMCache MP, LMCache CacheBlend, SGLang, NVIDIA Dynamo, and…

↯ Vllm vllm
Show HN: Per-request emotion steering for vLLM, with batching preserved (github.com via hn) +1 19w

emotion-steering Extract and serve CAA-style emotion steering vectors for any HuggingFace causal LM, with a fast vLLM path for Qwen3. ┌────────────┐ ┌────────────────────┐ labeled │ extract │ vectors + AUC report │ serve │ contrasts ├─────…

↯ Vllm ↯ Qwen 3 vllm
A plug-n-play open-source pruning tool that is workload-aware (www.reddit.com) +11 19w

This project was born out of time I spent digging into a biologically inspired algorithm I was using to measure co-activation for placement of experts and ranks onto chips. The default scheduling that vllm provides can end up causing laten…

↯ Ollama ↯ Llama ↯ Vllm vllm ollama llama
Advice needed on eGPU and Mini PC (www.reddit.com) +15 19w

Hi all, I come across to relatively niche problem and could not find much useful posts or guides about it. I have a mini pc (Beelink Ser 8, 8745HS and 32GB 5600 DDR5 SODIMM) headless server for hosting some routing services, and I am wonde…

↯ Vllm ↯ Qwen 3.6 vllm moe qwen
Show HN: Valkyr LM Inference with Realtime Guarantees (github.com via hn) +1 19w

Valkyr is a fresh take on LM Inference runtimes. It's quite different from llama.cpp, vLLM, or ZINC for example.

↯ Llama ↯ Vllm vllm llama openai
How can I locally run Deepseekv4 1.6T? I can use a VPS. (www.reddit.com) +11 20w

I wanted to use vast.ai, but ollama doesnt have it, and when i used vLLM I didn't have success. I genuinely don't know what failed.

↯ Ollama ↯ Vllm ↯ DeepSeek 4 vllm ollama
Is Mistral-3.5-Medium-128B broken in Llama CPP? (www.reddit.com) +14 20w

Trying some if Bartowski's Q4 quants. Using Vulkan with the latest main branch as of a few hours ago.

↯ Llama ↯ Vllm ↯ Mistral ↯ Swe Bench ↯ Mistral 3.5 swe-bench mistral vllm+1
5060ti quad-chads - vllm (the reluctant arc) - pp and tg talk (www.reddit.com) +1 20w

Okay, so I have this quad 5060ti setup and for forever I have had people nagging me to try vllm. I thought it was too complicated, like varsity golf or putting on both legs of pants at the same time.

↯ Vllm vllm
vLLM-Compile: Bringing Compiler Optimizations to LLM Inference (docs.google.com via hn) +1 20w

vLLM-compile: Bringing Compiler Optimizations to LLM Inference Luka Govedič vLLM Committer Senior Machine Learning Engineer, Red Hat 1

↯ Vllm vllm
3.6 27B Tool Calling Issues (vLLM) (www.reddit.com) +1 20w

Has anyone got a reliable vLLM recipe for 3.6 27B that fixes the tool calling issues? I am getting "Not let me..." - then nothing.

↯ Vllm ↯ Qwen 3.6 vllm qwen
Disaggregated Serving for Hybrid SSM Models in vLLM (vllm-website-lx4pji0mz-inferact-inc.vercel.app via hn) +1 20w

Disaggregated Serving for Hybrid SSM Models in vLLM Introduction Hybrid architectures that interleave Mamba-style SSM layers with standard full-attention (FA) layers — such as NVIDIA Nemotron-H — are gaining traction as a way to combine th…

↯ Vllm vllm
which is faster and better for coding? Luce-Org/Dflash or noonghunna/qwen36-27b-single-3090 (www.reddit.com) +19 20w

Anyone have experience with both? Luce is llama.cpp with custom dlflash and noonghunnas project is vllm with patches.

↯ Llama ↯ Vllm ↯ Qwen 3.6 vllm llama qwen
Power-limit vs TG/s for 2x3090 (www.reddit.com) +12 20w

Trying to find the sweet-spot to tradeoff between power and tg/s. 250W seems to be a sweet spot for Qwen3.6-27B.

↯ Vllm ↯ Qwen 3.6 vllm openai
your daily driver stack, what's it look like? and why? (www.reddit.com) +1 20w

What it says in the title, I'm interested in hearing what you all have landed on as a workable / useful stack for you. Mine looks like this: back end inference servers - llama.cpp, vLLM | V hermes-agent - cron jobs + OpenAI compatible endp…

↯ Cowork ↯ Llama ↯ Vllm vllm cowork llama+2
Qwen3.6-27B-FP8 - JS file is too long and causing JSON truncation (www.reddit.com) +14 20w

Apologies in advance, if this is a newbie question. When running Qwen3.6-27B-FP8 using the below command on an Nvidia RTX PRO 5000, in opencode, I am seeing errors such as: "The issue is that the JS file is too long and causing JSON trunca…

↯ Vllm ↯ Qwen 3.6 vllm qwen
ASUS Ascent GX10 - Having tons of issues (www.reddit.com) +110 21w

Hi all, Looking for some advice with a GX10 I purchased about 4 months ago. I've been having all kind of issues trying to run local models on this device.

↯ Ollama ↯ Vllm vllm ollama
What are your favorite LLMs for translation/docuement work? (www.reddit.com) +14 21w

I am currently working on a system to translate books/web novels. I got a working prototype, but now I am looking into optimizing it.

↯ Vllm ↯ Gemma vllm gemma
Short term access to 4x rtx6000pro... Suggestion on what to try/test? (www.reddit.com) +15 21w

Always been stuck with models that fit on my 16gb .... Going to have about a week for free with 4x rtx6000pro .

↯ Vllm ↯ Gemma ↯ Gemma 4 vllm gemma claude-code
Self-hosted AI: What is the way to go? (www.reddit.com) +19 21w

Hi everyone, I’m building a small support chatbot in Symfony for a limited group of users (around 300 people). For the MVP, I’m running everything locally on an NVIDIA DGX Spark with the GB10 Grace Blackwell superchip, using vLLM.

↯ Vllm vllm openai
Do you have any go-to utility LLM-related tools that are less commonly discussed? (www.reddit.com) +110 21w

↯ Ollama ↯ Llama ↯ Vllm vllm ollama openclaw+3
Deploying Gemma 4 26B A4B on a single RTX 5090 — ~196 tok/s with AWQ + vLLM on RunPod Serverless (www.reddit.com) +12 21w

↯ Vllm ↯ Gemma ↯ Gemma 4 vllm gemma moe
A Debugging Story: Getting Claude Code to Work with Local vLLM When the Docs Don't (www.reddit.com) +11 21w

↯ Vllm ↯ Qwen 3.5 vllm qwen claude-code
Multi GPU setup help (www.reddit.com) +12 22w

Hi guys I managed to get a multi GPU setup going with a 3090 and three 3060 bringing my vram to 60gb along with 64gb ddr5. The objective is to run the largest coding model I can at a respectable token speed of over 20 tokens / second.

↯ Vllm vllm moe
Training Qwen2.5-0.5B-Instruct on Reddit post summarization with GRPO on my 3x Mac Minis — add METEOR as quality reward! (www.reddit.com) +11 22w

Setup: 3x Mac Minis in a cluster running MLX. One node drives training, two push rollouts via vLLM.

↯ Vllm ↯ Qwen 2.5.0 vllm
ReDraft, Don't Just Distill: Reference-Driven Revision for Continual VLLM Post-Training (arxiv.org) 3d

Continual post-training of large multimodal models should add new capabilities while preserving those from pre-training, and the two goals pull in opposite directions. SFT gives explicit target supervision that learns a task from near-zero…

↯ Vllm vllm
Validating Hybrid-State Cache Recovery for GLM-5.3-Flash with vLLM and LMCache (arxiv.org) 4d

External cache transfers can succeed while a hybrid language model resumes from an inconsistent state. We examine the full 45-layer GLM-5.3-Flash model, using the RedHatAI/ GLM-5.3-Flash-NVFP4 quantized checkpoint with vLLM and LMCache und…

↯ Glm ↯ GLM 5.3 ↯ Vllm vllm glm
Building py-kvcache: A Performance Characterization of External KV Caching for vLLM with NVMe SSDs (arxiv.org) 8d

Prefix caching can reduce the time to first token (TTFT) of long-context LLM requests by reusing previously computed key-value (KV) states, but for short prefixes or fast GPUs, recomputation can be faster than loading from an external cach…

↯ Vllm vllm
Serving LLMs on Tenstorrent Hardware: Inside the vLLM TT Plugin (vllm.ai) 10d

Serving LLMs on Tenstorrent Hardware: Inside the vLLM TT Plugin Today we are introducing vLLM TT Plugin, which brings Tenstorrent accelerators to vLLM through the standard out-of-tree platform plugin mechanism. Install it alongside vLLM an…

↯ Vllm vllm
I was tired of my AI agents and dev tools leaving random processes and ports running in the background, then having no easy way to figure out what was eating resources. So I built Port Visualizer. (www.reddit.com via reddit) 3w

I always run so many AI agents that I keep ending up with mystery processes, stray localhost ports, and no quick way to tell what started what. So I built Port Visualizer, a free open-source Windows app that shows which process owns each T…

↯ Ollama ↯ Llama ↯ Vllm vllm ollama llama
GLM-5.3-Flash (FP8) on 4 x RTX6000 Pro (www.reddit.com via reddit) 3w

I've forked https://github.com/tonyd2wild/GLM-5.3-Flash-NVFP4-2x-DGX-Spark and make it run on sm120. I'm using it right now - got 1,4M context (5,45 sessions 262k each) 3,7kt/s PP and 160 - 230t/s TG (MTP enabled) You can make vllm Docker…

↯ Glm ↯ Vllm ↯ GLM 5.3 vllm glm
GLM-5.3-Flash @ DGX Station GB300: ~206 tok/s (single stream), 1M context (www.reddit.com via reddit) 3w

Hey all! I'm finally doing some cool stuff with my "thinking heater" (h/t u/-TV-Stand-).

↯ Glm ↯ Vllm ↯ GLM 5.3 vllm glm
Two vLLM Recipes for Blackwell: More Context, More Concurrency, Less VRAM Waste (www.reddit.com via reddit) 3w

Blackwell vLLM Recipes: Why NVFP4 KV Matters I’ve released two vLLM recipes for Blackwell GPUs: MTP-3 + NVFP4 KV DFlash2 + NVFP4 KV Both are built around the same idea: Blackwell can do native FP4 well, so the KV cache should benefit from…

↯ Vllm vllm
Questions on optimism speed/intelligence on this rig (www.reddit.com via reddit) 3w

Rig: 3945WX (12C, 2 CCDs, no AVX-512) · 8×32GB DDR4-3200 · 4× 5060 Ti 16GB · PCIe 4.0. Agentic workload (Hermes Agent).

↯ Vllm ↯ DeepSeek 4 ↯ DeepSeek 4 ↯ DeepSeek 4 ↯ DeepSeek 4 ↯ DeepSeek 4 ↯ DeepSeek 4 vllm deepseek agentic
Running local LLM's as agents in Claude Code (www.reddit.com via reddit) 3w

I hit my token limit three times a day on my max subscription - got sick of that and designed this MCP setup to shift some of the coding load to my local Qwen3.8-27B model. I've been iterating on it now for a bit, and thought I'd share it…

↯ Llama ↯ Vllm ↯ Qwen 3.8 vllm llama mcp+2
Qwen3.8 27B C8 at 972 TG / 5,680 PP on 4x MI100 rig ($6.5k) using my new INT8 vLLM fork (www.reddit.com via reddit) 3w

Yet another vLLM fork thread here, but this time its for older INT8-centric hardware. This is a complete INT8 serving stack for Qwen3.8 27B based on vLLM, AITER, and a 27B GPTQ INT8 quant w/ DFlash2.

↯ Vllm ↯ Qwen 3.8 vllm
Qwen3.8 Garbage Outputs after a few hours of use (www.reddit.com via reddit) 3w

Do I have something wrong in my settings? I keep having to restart vLLM to get Qwen3.8-27b-FP8 to stop outputting nonsense / garbage.

↯ Vllm ↯ Qwen 3.8 vllm qwen openai
256GB Mac vs 2 DGX Spark (www.reddit.com via reddit) 3w

Question: if your goal is to run inference, have a bunch of docker containers running your software and bunch of users using your product. What’s the most efficient route given both above options cost the same amount roughly.

↯ Vllm vllm qwen
Fully quantized NVFP4 Qwen3.8-27B with QUASAR QAD (huggingface.co via reddit) 3w

We're releasing a fully quantized NVFP4 version of Qwen3.8-27B. The checkpoint was trained using quantization-aware distillation (QAD) with QUASAR, our new QAT algorithm.

↯ Vllm ↯ Qwen 3.8 vllm
Anyone directly compare 3.8 27B at INT4 vs INT8? (www.reddit.com via reddit) 3w

Man I love the quality of 27B int8 (currently running lued/Qwen3.8-27B-INT8-W8A16-MTP). But man, it is not fast and I am seriously skating a line with memory consumption on my A40 at 256K context (KV at FP8).

↯ Vllm ↯ Qwen 3.8 vllm
Qwen 3.8 27b with tools and directed search on a non-coding professional suite (www.reddit.com via reddit) 3w

Some data showing the value of tools on the professional intelligence of Qwen 3.8 27b. My workload with Qwen focuses on real estate, private equity, and adjacent finance stuff.

↯ Vllm ↯ Qwen 3.8 vllm qwen
Best 64k-128k models/fine-tunes on a single 3090 for PRD planning & ticket creation? (www.reddit.com via reddit) 3w

Hey guys, I'm setting up a local workflow on a single 24GB RTX 3090 to handle project planning—specifically digesting massive (~128k context) requirements documents/PRDs and spitting out a ton of structured .md files to act like Jira ticke…

↯ Vllm ↯ Gemma ↯ Qwen 38 vllm gemma moe+1
2xR9700 - Switching to gen5 x8/x8 from gen4 x16+x4 - I see about 18-32% more performance in vLLM (www.reddit.com via reddit) 3w

I had read that I should not have mismatched PCIe configurations for tensor parallelism, so I upgraded my motherboard from MSI PRO-X670-P-WIFI to Asus ProArt X870E-Creator WIFI. With the old board I had gen4 x16 and gen4 x4.

↯ Vllm vllm
ReBar support for 20gb RTX 3080? (www.reddit.com via reddit) 3w

Has anyone successfully patched the drivers of a modded RTX 3080 20gb to get Rebar support? I already updated my 3090.

↯ Llama ↯ Vllm vllm llama
Qwen 3.8 27B Aider score (www.reddit.com via reddit) 3w

I ran the Aider benchmark on Qwen 3.8 27B FP8 with FP8 KV cache 256K context vLLM. The score: 72.9 This matches Gemini 2.5 Pro from 2025-04-12 which also scored 72.9.

↯ Vllm ↯ Gemini 2.5 aider vllm deepseek+3
Is anybody using Deepseek v4F 0731 with a vision encoder and have had any success? (www.reddit.com via reddit) 3w

Hey all, So I currently use 2x DGX Sparks with Deepseek v4 Flash 0731 and it works fantastic at 1M context, with 1.8M kv, dspark, vllm tp 2, etc. All good.

↯ Vllm ↯ DeepSeek 4 vllm deepseek mcp
R9700 AI Pro TP=2 Qwen3.8-27B-FP8 low speed? Need Advice. (www.reddit.com via reddit) 3w

Hi folks, with tp=2 I get the following logs out of vllm with official Qwen3.8-27b-FP8 with MTP3: [vllm] | (APIServer pid=1) INFO 08-24 05:47:40 [loggers.py:310] Engine 000: Avg prompt throughput: 198.5 tokens/s, Avg generation throughput:…

↯ Vllm ↯ Qwen 3.8 vllm
Qwen3.8 27B VLLM or SGLang Support (www.reddit.com via reddit) 3w

Hi friends, I've just spent the last few hours to ultimately fail to run Qwen3.8 27B in a docker container. Would anyone happen to have a working docker file they could share the contents of?

↯ Vllm ↯ Qwen 3.8 vllm
New qwen3.8:27b on a 39k line C to single-file HTML / three.js port (www.reddit.comhttps) 3w

I was just curious how the new qwen3.8:27b does on a hard C to HTML porting job against Opus 5 in a default Claude Code. The job: my fun side project is a procedural shooter in a single C file.

↯ Vllm ↯ Qwen 3.8 vllm opus claude-code
Ling Tiny, King of Speed (www.reddit.comhttps) 3w

Ling Tiny has now replaced Gemma4-12B in my rig as an auxiliary model doing hindsight operations. This is on a 4060Ti, which is a reasonable GPU available out there, and the speed is phenomenal.

↯ Vllm ↯ Gemma 4 vllm
vLLM + Deepseek harness or hermes? qwen3.8 (www.reddit.com via reddit) 3w

How do you guys set it up , i constantly get the error : I tried increasing the contex to 142k and putting the contex size as 115k in DSH , it still did not compress correctly. I have 0 issues if i run it with llama.ccp , it can work for 2…

↯ Llama ↯ Vllm ↯ Qwen 3.8 vllm deepseek llama
Only ONE 450K session can keep its prefix cache on 2× DGX Spark — a second session wipes it with 43% of the KV pool still free. 6-minute cold prefill every turn. What am I missing? (www.reddit.com via reddit) 3w

**Setup:** 2× DGX Spark (GB10, 121 GiB unified each), TP=2 over 2×200GbE RoCE, vLLM 0.25.2.dev0, DeepSeek-V4-Flash-0731 FP8, `max_model_len=450000`, prefix caching on. KV pool = **1,686,693 tokens**.

↯ Vllm ↯ DeepSeek 4 vllm deepseek agentic
Qwen3.8-27B NVFP4 with vision + 451K token KV-cache on one RTX 5090 (power limited to 400W) at 120 tokens/s average (www.reddit.com via reddit) 3w

Hello, So I've been trying lots of combinations in that never-ending landscape of options and settings. I wanted a proper quant of 3.8 27B running as fast as possible on my 5090 at 400W, with vision and with as much KV-cache as possible an…

↯ Vllm ↯ Qwen 3.8 vllm
I hosted Kimi K3 (2.8T parameters) using 8 B300s. 92 tok/s, $190 per million tokens (www.reddit.comhttps) 3w

What I ran: 8x B300 on Modal, $56.79 per hour, vLLM, tensor parallel 8, native MXFP4 Cold boot ~27 min (1.56 TB load, JIT, 51 CUDA graph captures) TTFT 0.92 to 1.02 s, decode 92 tok/s steady, 83 tok/s average over 4 prompts $190 per millio…

↯ Llama ↯ Vllm vllm llama
i finally switched from windows to linux and got a 30-50% boost in speed. (www.reddit.com via reddit) 3w

This is amazing. All I did was switch from llamacpp on windows to vllm on linux.

↯ Vllm vllm
~3,200 tok/min on Two 3090s, No NVLink — Local Qwen Coding Agent at Full 262K Context (www.reddit.com via reddit) 3w

Qwen3.8-27B (W4A16-AutoRound) on vLLM 0.27.1, two RTX 3090s, TP=2, no NVLink. Agent: Pi, straight to the local endpoint.

↯ Vllm ↯ Qwen 3.8 vllm qwen
Local LLM Inference: A 96GB Blackwell Field Guide (2026) (kitdev.ai via reddit) 4w

Local LLM Inference: A 96GB Blackwell Field Guide (2026) Running local LLMs on 96GB Blackwell: which models fit, vLLM against SGLang, the throughput you actually get, and the places the setup quietly falls over. The verdict.

↯ Vllm vllm
Question for folks with r9700 (www.reddit.com via reddit) 4w

I have dual r9700 set up with Ubuntu. With llama.cpp I'm getting about 40 tokens per second for single instance.

↯ Llama ↯ Vllm ↯ Qwen 3.8 vllm llama agentic
Single RTX 5090: Qwen3.8-27B NVFP4 at a real 262K context in vLLM — 77 tok/s short-context, 64.7 tok/s at 128K (www.reddit.com via reddit) 4w

This is the Qwen3.8-27B setup I actually use every day on one RTX 5090. I wanted to write it down with enough detail that another 5090 owner can reproduce it instead of guessing which memory knobs I used.

↯ Vllm ↯ Qwen 3.8 vllm
I forked Ninfer 3090 and converted it to run on the CMP170HX - doubled my Qwen3.6-35B from llama.cpp (www.reddit.com via reddit) 4w

Good afternoon, everyone! I wanted to show the work I've been doing around porting Ninfer over to the CMP170HX (Github) So, first, I do want to call out the amazing work that Neroued, Sergiuszm and specifically Don-Chad have all done, to n…

↯ Llama ↯ Vllm ↯ Qwen 3.8 vllm llama codex
Helm chart for Qwen 3.8 for B70 users (www.reddit.com via reddit) 4w

I took SergiioB's Intel Arc Pro B70 Inference Cookbook and made it into a Helm chart. It applies the patches onto the pinned vLLM version.

↯ Vllm ↯ Qwen 3.8 vllm qwen
If you want to upgrade from Qwen3.6 27B, but dislike 3.8, give Ornith-1.5-35B-A3B a try. (www.reddit.com via reddit) 4w

Just make sure to leave MTP off. It's a decent upgrade from Qwen3.6 27B, and better than KAT Coder V2.5 Dev (which also requires their custom vLLM).

↯ Vllm ↯ Qwen 3.6 vllm moe
CMP170HX (www.reddit.com via reddit) 4w

For those who bought the CMP170HX and unlocked the memory, I have some questions… are you using any specific llama.cpp or vLLM forks? What are you using for cooling the card?

↯ Llama ↯ Vllm vllm llama
I feel like I finally graduated. (www.reddit.com via reddit) 4w

I finally made the move from LM Studio to vLLM thanks to this post https://www.reddit.com/r/LocalLLaMA/s/NmS9CgHvqz. I may not know what it all means yet but I’m going to start diving into the docs to learn as much as I can.

↯ Vllm ↯ Qwen 3.8 vllm qwen
Engineering Signals of Human-AI Collaboration in the Agentic Coding Era: A Longitudinal Analysis of 33,228 Pull Requests from vLLM and SGLang with Implications for Biomedical AI Agents and Bioinformatics Pipeline Developmen (arxiv.org) 4w

The rapid adoption of AI coding assistants and autonomous agentic development systems has coincided with major changes in the pace and structure of open-source software engineering. Yet empirical longitudinal evidence of these changes at t…

↯ Vllm vllm agentic
Meta is back with Muse Glimmer: local, agentic, multimodal, and open source (huggingface.co) 5w

Meta is back with Muse Glimmer: local, agentic, multimodal, and open source! To celebrate, we are shipping with Meta day-0 support in transformers , llama.cpp , vLLM , Inference Endpoints, and other libraries.

↯ Llama ↯ Vllm vllm llama agentic
What We Observe as LLM Behavior Can Be a Side-effect of Inference Backend (arxiv.org) 6w

Benchmark scores are reported as properties of a model, yet the inference framework used to produce them, such as HuggingFace, vLLM, or Ollama, are considered non-influential and their names and versions are almost never disclosed. In this…

↯ Ollama ↯ Vllm vllm ollama
vLLM for Baidu Kunlun (github.com) 7w

📖 Documentation | 🚀 Quick Start | 📦 Installation | 💬 Slack Latest News 🔥 [2026/07] 🚧 v0.25.1 under development — Added Qwen3.5 / Qwen3.5-MoE, Gemma4 (text and multimodal), GLM MoE DSA, and DFlash speculative decoding [2026/02] ⚡ Performanc…

↯ Glm ↯ Vllm vllm glm moe
Gauge dependence and structured-output corruption in sign-branched repetition penalties: measurements across models, inference stacks, and alternative repetition controls (arxiv.org) 9w

The multiplicative repetition penalty shipped across the LLM inference ecosystem (HuggingFace, vLLM, llama$.$cpp, and a dozen further engines) branches on the sign of each raw logit (divide positives by theta, multiply negatives). But the…

↯ Llama ↯ Vllm vllm llama
Attention to Detail: Evaluating Energy, Performance, and Accuracy Trade-offs Across vLLM Configurations (arxiv.org) 9w

Large Language Models are reshaping how software is developed and maintained. They are typically deployed in production using inference engines such as vLLM, which can efficiently serve pre-trained, highly configurable models.

↯ Vllm vllm
Native-speed vLLM transformers modeling backend (huggingface.co) 10w

Native-speed vLLM transformers modeling backend # Upgrade the vllm pip package uv pip install --upgrade vllm --torch-backend auto The transformers library has become the reference modeling library for Machine Learning. It supports 450+ arc…

↯ Vllm vllm
I rebuilt the Claude Code-style terminal workflow as a hackable multi-provider coding agent (www.reddit.com via reddit) 10w

Hello everyone, After the Claude Code leak started floating around, I spent time studying how the workflow was put together and rebuilt the core experience into my own project. I’m calling it Super Grokie, because it started as a joke but…

↯ Ollama ↯ Vllm vllm ollama grok+4
Can you use Claude Desktop (code) but use local API? (www.reddit.com via reddit) 11w

Hey all, So I tend to favor the Claude Desktop app in Code mode as the GUI does a great job of previewing code, MCP browser interactions/screenshot evals/etc. But I recall people saying they could get Claude Desktop to use a local API.

↯ Ollama ↯ Llama ↯ Vllm vllm ollama llama+1
An Efficient vLLM-Based Inference Pipeline for Unified Audio Understanding and Generation (arxiv.org) 11w

While Large Multimodal Models excel in comprehension, high-throughput inference engines lack native support for multimodal generation. This is severe in Speech Language Models, where generating multi-layered audio tokens via decoupled AR+N…

↯ Vllm vllm
Run a vLLM Server on HF Jobs in One Command (huggingface.co) 12w

Run a vLLM Server on HF Jobs in One Command It's the quickest way to stand up a model for tests, evals, or batch generation. (If you're after a managed, production-ready service instead, that's what Inference Endpoints are for — more on wh…

↯ Vllm vllm
New sampler + verifier *drastically* improves tiny 0.5b model coding performance (arxiv.org via reddit) 12w

I read it with a little bit of effort The tiny model result is insane, theoretically this could make make a 0.5b on-par with a 2/3/4b ish class model in coding with no weights change*. And for large models it could maybe fix let's say 30-5…

↯ Hallucination ↯ Llama ↯ Vllm vllm hallucination llama
DriftSched: Adaptive QoS-Aware Scheduling under Runtime Token Drift for Multi-Tenant GPU Inference (arxiv.org) 12w

The rapid growth of large language model (LLM) inference services has increased the demand for efficient multi-tenant GPU scheduling. While modern inference runtimes such as vLLM improve throughput through continuous batching and optimized…

↯ Vllm vllm
CacheWeaver: Cache-Aware Evidence Ordering for Efficient Grounded RAG Inference (arxiv.org) 13w

Retrieval-Augmented Generation (RAG) improves factual grounding, but it also lengthens prompts and raises prefill cost. Prefix caching in serving engines such as vLLM reduces this cost only when requests share the same token prefix.

↯ Vllm vllm rag
DifussionGemma 4 on 4x7900xtx (www.reddit.com via reddit) 14w

Just got 100 tps on generation, but in total time it around 45-60 t/s in case of prompt processing waiting. Available memory show: GPU KV cache size: 152,671 tokens Maximum concurrency for 131,072 tokens per request: 1.16x amd-smi monitor…

↯ Vllm vllm
Built a minimalist coding agent optimized for memory footprint and speed (www.reddit.com via reddit) 14w

Hi everybody, I spent the last two weeks building zerostack, a coding agent in Rust, focused on memory footprint, shipping with ollama and vLLM integrations. I managed to get it to run at ~16MB (with peaks of 24MB) of RAM usage, and no CPU…

↯ Ollama ↯ Vllm ↯ Mistral mistral vllm ollama
Anyone gotten Gemma 4 12B (unified audio) to actually attend to speech with a large system prompt? (www.reddit.com via reddit) 14w

I'm trying to use Gemma 4 12B — the new encoder-free unified model (audio/vision/text in one) — for a one-pass audio → response voice assistant: feed the recorded WAV + system prompt and get the reply back as text directly, collapsing the…

↯ Vllm ↯ Gemma ↯ Gemma 4 vllm gemma
We are open-sourcing LiteLLM Agent Platform: a self-hosted OSS agent builder for Hermes, OpenCode, Claude Code (bring your own models, Ollama/vLLM work) (www.reddit.com via reddit) 14w

↯ Ollama ↯ Vllm vllm ollama cursor+2
Releasing Cohere North Mini Code (www.reddit.comhttps) 14w

↯ Vllm vllm
I have 4x 128 GB VRAM now , what should i do. (www.reddit.com via reddit) 14w

↯ Vllm ↯ Qwen 3.5 vllm qwen
Fixing single missing quote errors. (www.reddit.com via reddit) 14w

Loving the local AI world and been building out my own the last few weeks. One pesky recurring problem is what seems to be related to how the model produces JSON for its responses.

↯ Vllm vllm
RDNA4 Specific Docker Image vLLM (www.reddit.com via reddit) 14w

You bought RDNA4 with the promise of go-fast, and it doesn't deliver in vLLM. I know the feeling, out of the box vllm is a complete dog on RDNA4...

↯ Vllm vllm
Here are some tips on hitting nearly 200 tok/s for DeepSeek v4 Flash on Hopper (dnhkng.github.io via reddit) 14w

I needed a smarter model for my local Hermes Agent setup, so I moved to DeepSeek v4 Flash. First things first: Running 4 concurrent threads on vLLM, I can hit ~400 tok/s 400 x 60 x 60 x 24 x 30 is ~1B TOKENS per month!!!

↯ Vllm ↯ DeepSeek 4 vllm deepseek
5070 Ti + 5060 Ti on vLLM hangs on GDN with Qwen3.6 (www.reddit.com via reddit) 14w

↯ Vllm ↯ Qwen 3.6 vllm
[2x3090]: SymmMemCommunicator: Device capability 8.6 not supported, communicator is not available. (www.reddit.com via reddit) 14w

Hi all, this is a mere "see what others are doing post" rather than a solution to a problem. As newbie, I put together a 2x3090 box that I run vllm on.

↯ Vllm vllm
OpenEnv is now owned by HF, Torch, Prime Intellect, Unsloth, Modal, Mercor, and more! Use it for training agents. (www.reddit.com via reddit) 14w

OpenEnv is a tool for creating an agentic execution environment like terminals, browsers, or anything an agent can interact with. And today, we’re excited to announce that OpenEnv is becoming even more open, to make the future of training…

↯ Vllm vllm agentic
vllm-doctor — a CLI tool to diagnose and monitor vLLM inference servers (www.reddit.com via reddit) 14w

vllm-doctor reads metrics from a vLLM server's /metrics endpoint or a Prometheus instance and runs rule-based checks to find what is wrong. It detects queue pressure, high TTFT/TPOT, KV cache pressure, and other rules across pods.

↯ Vllm vllm
Breaking the Ice: Analyzing Cold Start Latency in vLLM (arxiv.org) 14w

↯ Vllm vllm
how to run gemma-4-12b-it-qat-w4a16-ct in vllm or any version quantized of the model (www.reddit.com via reddit) 14w

when running by using transformers it runs by using vllm some weird error come up plese can any body share the command of running it on vllm ?

↯ Vllm ↯ Gemma ↯ Gemma 4 vllm gemma
club-3090 adds experimental FP8 support for Qwen3.6-27B! (www.reddit.com via reddit) 14w

It’s finally here! Something many of us running dual RTX 3090 rigs have been anticipating.

↯ Vllm ↯ Qwen 3.6 vllm qwen
dvlt.cu: inference engine written from scratch in CUDA/C++ for NVIDIA's DVLT 3D transformer model (www.reddit.comhttps) 15w

Im into both HPC and 3D reconstruction, so I built this as a side project. dvlt.cu is a single 5MB binary: - No python, torch, TF, ONNX, llama.cpp, vLLM, or huggingface runtime - Nearly no dependencies: only cuBLASLt (shipped with libcuda…

↯ Llama ↯ Vllm vllm llama
Activating MTP for QATGemma4 31b q4_0? (www.reddit.com via reddit) 15w

Has anyone figured out how to activate MTP for Gemma4’s new QAT q4_0 GGUF for 31b? Or is this still not supported in llamacpp?

↯ Vllm ↯ Gemma 4 vllm
Serving TTS/cloning models on llama.cpp? (www.reddit.com via reddit) 15w

Are there any quality voice cloning and speech generation models that already have support in Llama.cpp or, more likely, vLLM-Omni? It would be nice to swap them out like any other inference model and use a common API, rather making a sepa…

↯ Llama ↯ Vllm vllm llama
Built a config sweep CLI for llama.cpp and vLLM and found out Q4_K_M beat Q8_0 by 230ms TTFT on Qwen2.5-7B (www.reddit.com) 19 16w

I have been coming to this subreddit to understand what the optimal config is to run a model on a given hardware setup. I referred to specific benchmarks, but they are too generic and do not consider the underlying hardware.

↯ Llama ↯ Vllm ↯ Qwen 2.5 vllm llama
$16 refactor, 400 steps, 95% routed to open MoE (www.reddit.com) 6 17w

Got tired of $160 Opus bills so I spent a weekend wiring up a routing layer on vLLM 0.8 (2xA100, enable_auto_tool_choice). Getting the tool call parser to cooperate took longer than the actual routing logic.

↯ Vllm ↯ DeepSeek 4 vllm moe deepseek+1
For the users who have add bad luck with QWEN 3.6 27B, and Gemma 4 31B. "Actually..wait..actually". Endless reasoning. Horrible output. I found a solution. rtx pro 6000. (www.reddit.com) 26 17w

Edit: does this happen every time a newbie tries to post here. Getting roasted despite having valid results?

↯ Vllm ↯ Gemma ↯ Qwen 3.6 vllm gemma qwen
Built a self-hosted layer for local agent workflows because retries kept replaying side effects (www.reddit.com) 1 17w

I work on AxonFlow, a source-available (BSL 1.1) runtime for long-running agent workflows. We’ve been running it in front of Ollama-served models and OpenAI-compatible local endpoints (llama.cpp `--server`, vLLM, LM Studio).

↯ Ollama ↯ Llama ↯ Vllm vllm ollama llama+1
dual spark with llama.cpp (www.reddit.com) 6 17w

I'm daily driving dual Asus GX10 (spark) with vllm and it's fantastic. But I want to try model that is GGUF only and won't fit into single spark.

↯ Llama ↯ Vllm ↯ Minimax minimax vllm llama
I built a native Swift macOS AI client that's invisible to screen sharing — works with Ollama, vLLM, llama.cpp [OC] (www.reddit.com) 3 17w

Built this for myself after wanting to use local LLMs during work calls without the window showing up on screen share. Every existing tool was either cloud-only or a 200MB Electron app.

↯ Ollama ↯ Llama ↯ Vllm vllm ollama llama+1
Is this a crazy idea? (www.reddit.com) 8 18w

I’m running locally with 2 RTX 3099s and 128gb of RAM I run my workflows with Hermes/OWUI and use Comfy for media generation. My inference is with LM Studio.

↯ Vllm vllm
vLLM + NVFP4 + Qwen3.6 27B: "Checkpoint does not provide a q scaling factor"? (www.reddit.com) 1 19w

I have been trying various NVFP4 based variations of Qwen 3.6 27B, and I am seeing this for the ones that look most interesting to run on my 2x 16GB VRAM with KV cache fp8. vllm | (Worker_TP0 pid=136) WARNING 05-09 13:49:27 [kv_cache.py:10…

↯ Vllm ↯ Qwen 3.6 vllm qwen
Benchmark Qwen 3.6 27B MTP on 2x3090 NVLINK (www.reddit.com) 19w

TL;DR On 4× RTX 3090 with NVLink bonded between GPU pairs (0↔2 and 1↔3), pinning TP=2 to a NVLinked pair gave +25% throughput at concurrency 1 and +53% at concurrency 4 vs running TP=2 over PCIe. Adding the other two GPUs to make it TP=4 m…

↯ Vllm ↯ Qwen 3.6 vllm qwen
I built an episodic, 2-tier memory for long-running local AI agents - temporal contradiction detection, fiction/roleplay filter, no vector DB required. (www.reddit.com) 19w

I've been running a persistent local agent for about 2 months - hundreds of sessions, mix of local models (llama.cpp/vLLM/lmstudio) and paid (Claude). One of the things that has been driving me nuts with OpenClaw and Hermes is the way memo…

↯ Llama ↯ Vllm vllm openclaw llama
vLLM V0 to V1: Correctness Before Corrections in RL (huggingface.co) 19w

vLLM V0 to V1: Correctness Before Corrections in RL TL;DR. vLLM V1 matched our vLLM V0 reference after we fixed four things: processed rollout logprobs, V1-specific runtime defaults, the inflight weight-update path, and the fp32 lm_head us…

↯ Vllm vllm
Help with GPT-OSS-120B on vLLM (www.reddit.com) 2 19w

Hiya, today I was trying to get a response from GPT-OSS-120B via vLLM - and failed miserably! Has anybody gotten it to work, i.e.

↯ Vllm vllm
Getting unexpected output with Gemma 4 31b-it on vLLM (www.reddit.com) 3 19w

Hey everyone, I'm running into a weird issue and hoping someone here might have a fix or some troubleshooting ideas. I'm currently trying to run the new Gemma 4 31b-it model using vLLM (v0.20.0-cu130) deployed via Helm chart (https://gith…

↯ Vllm ↯ Gemma ↯ Gemma 4 vllm gemma openai
Gemma 4 31B MTP Drafter on H100 -- Real Benchmarks + DFlash Comparison (www.reddit.com) 7 19w

Just tested Gemma 4 31B with the new official MTP Drafter on my H100 today and compared the approach with DFlash to help you decide which one to use. Without drafter: 13.7 tok/s.

↯ Vllm ↯ Gemma ↯ Gemma 4 vllm gemma
Sglang is better for serving a model for a personal agent harness? (www.reddit.com) 3 19w

If one has enough vram, would Sglang be a superior choice than vLLM or llamacpp in terms of inference speed for serving a model dedicated to powering a personal (single user) agent harness like Hermes agent? Sglang has MTP for speculative…

↯ Vllm ↯ Qwen 3.6 vllm
Does running a model (like qwen3.6-27b) on vllm or transformers use less VRAM than llama.cpp? (www.reddit.com) 5 19w

I have been using llama.cpp to run some models recently. For example, I've been running GLM-4.7-Flash with this command .\llama-server.exe -hf unsloth/GLM-4.7-Flash-GGUF:Q6_K_XL --alias "GLM-4.7-Flash" --host 127.0.0.1 --port 10000 --ctx-s…

↯ Glm ↯ Llama ↯ Vllm ↯ Qwen 3.6 vllm glm llama+1
Anyone running HUANANZHI H12D-8D + BMC with 4x RTX 3090 for LLM inference? (www.reddit.com) 2 19w

Hi everyone, I'm considering building a home LLM inference rig around: - HUANANZHI H12D-8D + BMC - AMD EPYC 7002/7003 - 4x RTX 3090 24GB - DDR4 ECC RDIMM, 8-channel - Linux + vLLM / SGLang / llama.cpp - Open frame, PCIe 4.0 x16 risers The…

↯ Llama ↯ Vllm vllm llama
Requesting advice on local AI setup for academic use (www.reddit.com) 2 20w

I'm about to do a clean install of Ubuntu 26.04 on a desktop that has a 5060ti 16gb and a 4060ti 16gb. Can you help me work out the best local AI setup for my use cases?

↯ Llama ↯ Vllm vllm llama
Need advice on Qwen 3.6 27B INT4 quantization (www.reddit.com) 3 20w

Hello everyone, I think Qwen 3.6 27B is good enough that it might take a while before we get a clearly better model at a similar size. I have a single headless RTX 3090 with a 300W power limit.

↯ Llama ↯ Vllm ↯ Qwen 3.6 vllm llama qwen+1
Qwen 3.6 wins the benchmarks, but Gemma 4 wins reality. 7 things I learned testing 27B/31B Vision models locally (vLLM / FP8) side by side. Benchmaxing seems real. (www.reddit.com) 8 20w

Hey guys, A couple of weeks ago, I asked this sub for the hardest Vision use cases you were dealing with to test the newly dropped Qwen 3.6 against Gemma 4. I finally finished running the gauntlet side-by-side locally on vLLM (FP8 quants)…

↯ Vllm ↯ Gemma ↯ Qwen 3.6 vllm gemma qwen
DeepSeek V4 Flash as a cheap worker in your LLM stack: $0.0003/call via MCP, swappable endpoint (www.reddit.com) 20w

Most of my LLM cost was on the wrong tier of work. Classification, extraction, JSON formatting, summarization I'm going to review anyway.

↯ Ollama ↯ Vllm ↯ Mistral ↯ DeepSeek 4 mistral vllm ollama+3
Best RTX Pro 6000 vllm settings? (www.reddit.com) 2 20w

Just got myself (for my company) a RTX Pro 6000 Blackwell Workstation card. Managed to get really good TPS on qwen3 27b fp8.

↯ Vllm ↯ Qwen 3 vllm
thinking of gemma 4 26B vs 31B (www.reddit.com) 2 20w

I see a big difference in agentic coding between gemma-4-31B-it-Q5_K_M and gemma-4-26B-A4B-it-UD-Q8_K_XL. The 26B model is much faster because of A4B and generally works well, but there is a big difference in thinking.

↯ Llama ↯ Vllm ↯ Gemma ↯ Gemma 4 vllm gemma moe+2
Reasoning Guard: Stopping LLM Thinking Loops at the Proxy Layer (www.reddit.com) 20w

Reasoning Guard: Stopping LLM Thinking Loops at the Proxy Layer I’ve been running Qwen3.6 MoE behind a vLLM proxy and hit a specific reliability issue: occasional runaway reasoning loops. This isn’t a criticism of Qwen3.6.

↯ Vllm ↯ Qwen 3.6 vllm moe agentic
Only 120 tps on Qwen 35b on h200 (www.reddit.com) 9 20w

Just a sanity check, this is too slow and something is wrong, right? Like, this is setup with mtp, vllm with awq quants, I suspect that I did configure something wrongly.

↯ Vllm vllm qwen
locally uncensored v2.4.2 - chat, coding agent, image + video generation in one local app. plus remote access from your phone. one-click install (www.reddit.com) 5 20w

locally uncensored is a desktop app that combines four things most people run separately: chat, a coding agent, image generation, and video generation. all local, all on your hardware, no docker, no cloud account needed.

↯ Ollama ↯ Llama ↯ Vllm vllm ollama llama+3
How do you actually use Qwen3 72B Instruct locally? (www.reddit.com) 8 20w

I just got Qwen3 72B Instruct running on a high RAM setup and I’m kinda confused about the proper way to use it. What’s the correct workflow for running it smoothly (like best quant, tools, or runtime)?

↯ Ollama ↯ Llama ↯ Vllm ↯ Qwen 3 vllm ollama llama
Free book on building AI agent harnesses — 22 chapters, Python harness, written by AI (www.reddit.com) 3 21w

Claude Code drafted the prose. I did the research, direction, architecture, ran the code, caught the bugs, and reviewed every commit.

↯ Ollama ↯ Vllm vllm ollama mcp+3
Is there an alternative between vLLM and Ollama that handles token prefill? (Arc Pro B70) (www.reddit.com) 1 21w

I am using an Arc Pro B70 to do inference, and it's token generation speed is fine using Ollama, but it takes *forever* to do a prefill. vLLM absolutely tackles the prefill problem (nearly instant responses), but I can't run nearly as larg…

↯ Ollama ↯ Vllm ↯ Gemma 4 vllm ollama
Brand new dual 3090 PC - what should I install first for the best local agentic coding experience? (www.reddit.com) 6 21w

↯ Llama ↯ Vllm ↯ Qwen 3.6 vllm llama qwen+1
Qwen3-30B-A3B-Instruct-2507 is better than the new Qwen 3.6 for our tasks (www.reddit.com) 9 21w

↯ Vllm ↯ Gemma ↯ Qwen 3.6 vllm gemma moe+1
made a desktop app that puts ollama, comfyui and coding into one window (www.reddit.com) 1 22w

been using local AI for a while now but my workflow was a mess. ollama for chat, comfyui for images, different tools for video and coding.

↯ Ollama ↯ Vllm vllm ollama
M1 Pro 16GB users: what local LLM configs are actually usable day to day? (www.reddit.com) 2 22w

I'm trying to get past generic "best model" recommendations and collect real-world configs from people on similar hardware. My setup: MacBook M1 Pro, 10-core CPU, 14-core GPU, 16 GB unified memory.

↯ Ollama ↯ Llama ↯ Vllm vllm ollama llama
DGX Spark users: What's the easiest way to do multi-node vLLM clustering with a browser UI and training? (www.reddit.com) 3 22w

Hey r/LocalLLaMA, I've been running a small 4-node DGX Spark cluster on a 400µT fabric switch and got frustrated with the usual raw Ray/vLLM scripts and EXO basically ignoring pure NVIDIA paths. I started from the solid foundation in [eugr…

↯ Vllm ↯ Fine Tuning vllm fine-tuning openai
gemma4 e2b ore4b on rtx 5070 ti laptop 12GB not running on vLLM (www.reddit.com) 3 22w

I cant get gemma 4 e2b or gemma 4 e4b to run on my laptop. I am runnning it via docker as per vllm website and i get the error : Free memory on device cuda:0 (9.71/11.5 GiB) on startup is less than desired GPU memory utilization (0.9, 10.3…

↯ Vllm ↯ Gemma ↯ Gemma 4 vllm gemma
Lower inference speed of Gemma4 26BA4B on vllm. (www.reddit.com) 8 22w

For my earlier use case I used to host qwen 2.5 vl 7b gptq int4. Now I was looking to switch to Gemma4 26B A4B, as it would improve performance as well as improve latency considering only 4B parameters are active..

↯ Vllm ↯ Qwen 2.5 vllm qwen
Hardware needed for Gemma 26B MoE vs Qwen 14B for ~100–300 users (vLLM, single node?) (www.reddit.com) 16 22w

I'm trying to figure out what sort of hardware setup i will need to accomodate a userbase of 100 users (not necessarily concurrent). Does anyone have any idea what sort of setup i'd be looking at?

↯ Vllm ↯ Gemma ↯ Qwen 2.5 vllm gemma moe+1
What is the best way to deploy LLM on 3x3090? (www.reddit.com) 13 22w

Two questions: which model? In my mind, Qwen3.5 27b or Gemma 4 31b are top options.

↯ Llama ↯ Vllm ↯ Gemma ↯ Qwen 3.5 vllm gemma llama
Optimizing a WSL2-based Local AI Orchestration for Product Viz | RTX 3090 24GB VRAM & i7-14700KF (www.reddit.com) 7 22w

Hi everyone, I’m building a local AI pipeline on WSL2 (Ubuntu) specifically for Product Visualization. My goal is to orchestrate LLMs for scene generation and Stable Diffusion/ComfyUI for high-fidelity rendering, keeping my Windows host cl…

↯ Ollama ↯ Vllm vllm ollama
No GPU left behind: Unlocking Efficiency with Co-located vLLM in TRL (huggingface.co) 67w

↯ Vllm vllm
Introducing multi-backends (TRT-LLM, vLLM) support for Text Generation Inference (huggingface.co) 87w

↯ Vllm vllm

← all tags