Hallucinated AI & agentic coding news. Some of it is real.

event

Vllm

311 items · started 2025-01-16 · ongoing (last activity 2026-09-18)

  1. Keeping vLLM's Prefix Cache Warm Between Agent Turns From 55% to 95% cached 🔗 I’ve been playing with a few different ways to host Qwen3.8 locally. I’m aiming for something that can replace Claude Code for most of my tasks.

  2. Continual post-training of large multimodal models should add new capabilities while preserving those from pre-training, and the two goals pull in opposite directions. SFT gives explicit target supervision that learns a task from near-zero…

  3. 𝗥𝘂𝗻𝗻𝗶𝗻𝗴 @vllm_project 𝗶𝘀 𝗲𝗮𝘀𝘆. 𝗞𝗲𝗲𝗽𝗶𝗻𝗴 𝗹𝗮𝘁𝗲𝗻𝗰𝘆 𝗽𝗿𝗲𝗱𝗶𝗰𝘁𝗮𝗯𝗹𝗲 𝘄𝗵𝗲𝗻 𝟯𝟬𝟬 𝗰𝘂𝘀𝘁𝗼𝗺𝗲𝗿𝘀 𝗮𝗿𝗿𝗶𝘃𝗲 𝗮𝘁 𝗼𝗻𝗰𝗲 𝗶𝘀 𝘁𝗵𝗲 𝗵𝗮𝗿𝗱 𝗽𝗮𝗿𝘁.

  4. External cache transfers can succeed while a hybrid language model resumes from an inconsistent state. We examine the full 45-layer GLM-5.3-Flash model, using the RedHatAI/ GLM-5.3-Flash-NVFP4 quantized checkpoint with vLLM and LMCache und…

  5. Prefix caching can reduce the time to first token (TTFT) of long-context LLM requests by reusing previously computed key-value (KV) states, but for short prefixes or fast GPUs, recomputation can be faster than loading from an external cach…

  6. Performance Benchmarks Explore Optimized AI models recipes across GPUs, frameworks, and deployment configurations. BENCHMARKED MODELS REASONING vLLM · 8× NVIDIA H100-80GB Qwen3-235B-A22B +138.7% token/s +58% Cost Savings Qwen3-235B-A22B ·…

  7. LiteLLM in front of vLLM Context: the vLLM serving entry covers the engines. This is the front door — a LiteLLM proxy in front of them: one OpenAI-compatible endpoint, virtual keys per consumer, spend tracking, health-check routing between…

  8. Serving LLMs on Tenstorrent Hardware: Inside the vLLM TT Plugin Today we are introducing vLLM TT Plugin, which brings Tenstorrent accelerators to vLLM through the standard out-of-tree platform plugin mechanism. Install it alongside vLLM an…

  9. Exploring Speculative Decoding in vLLM on AMD GPUs TL;DR: Speculative decoding allows vLLM to verify multiple drafted tokens in a single target-model pass. In our experiments, its effect on output-token throughput varied across drafting me…

  10. MaskShift is a local-first coding agent harness. Features: - Works with models that have no tool-calling API.

  11. trie trie stands for trace replay inference evaluation and is a lightweight benchmarking harness that exercises OpenAI-compatible inference servers with synthetic workloads derived from production traces. It targets backends like vLLM, SGL…

  12. Hey HN. We built an openai-compatible API for running open-weight VLMs, OCR VLMs and ViT-based vision models.

  13. Hey HN, most agent systems default to server-side Python inside containers and chain frameworks. I wanted to see how far we could push agent in the browser with vanilla JavaScript https://buttercup.sh The reason this is interesting is beca…

  14. trimtab Change SGLang and vLLM scheduler settings while the server runs. No restart, weights never leave the GPU.

  15. Long story short, about a year ago, in spite of everybody bashing gpt-oss for broken tool calling and refusals, I thought there's something there worth exploring. Model hit a sweet spot for me in that it was the first time I could run full…

  16. Hi everyone, I’m one of the co-founders of axem. We recently open sourced Shaide, a project we’ve been working on to make running multiple LLMs on your own infrastructure less painful.

  17. Efficient Decode Context Parallelism with vLLM for Long Context Workloads 1. Introduction Long-context inference is becoming essential for agentic AI, where assistants may need to reason over large code repositories and long chat histories.

  18. 中文 ｜ English 🖥️ Official Website | 💬 GitHub Table of Contents Model Introduction A New Flagship Generation Built for Productivity Benchmark Appendix Known Limitations News Model Links Quickstart Deployment vLLM SGLang Finetuning Quantizati…

  19. I always run so many AI agents that I keep ending up with mystery processes, stray localhost ports, and no quick way to tell what started what. So I built Port Visualizer, a free open-source Windows app that shows which process owns each T…

  20. vLLM can drop or garble a tool call and still return 200 Four reproducible parser failures in vLLM 0.26.0, 0.27.1 and 0.28.0, run on CPU with no GPU or weights. As of 2026-08-27 one is fixed on main but not in any release; the rest are ope…

  21. I've forked https://github.com/tonyd2wild/GLM-5.3-Flash-NVFP4-2x-DGX-Spark and make it run on sm120. I'm using it right now - got 1,4M context (5,45 sessions 262k each) 3,7kt/s PP and 160 - 230t/s TG (MTP enabled) You can make vllm Docker…

  22. Hey all! I'm finally doing some cool stuff with my "thinking heater" (h/t u/-TV-Stand-).

  23. Blackwell vLLM Recipes: Why NVFP4 KV Matters I’ve released two vLLM recipes for Blackwell GPUs: MTP-3 + NVFP4 KV DFlash2 + NVFP4 KV Both are built around the same idea: Blackwell can do native FP4 well, so the KV cache should benefit from…

  24. Rig: 3945WX (12C, 2 CCDs, no AVX-512) · 8×32GB DDR4-3200 · 4× 5060 Ti 16GB · PCIe 4.0. Agentic workload (Hermes Agent).

  25. I hit my token limit three times a day on my max subscription - got sick of that and designed this MCP setup to shift some of the coding load to my local Qwen3.8-27B model. I've been iterating on it now for a bit, and thought I'd share it…

  26. Yet another vLLM fork thread here, but this time its for older INT8-centric hardware. This is a complete INT8 serving stack for Qwen3.8 27B based on vLLM, AITER, and a 27B GPTQ INT8 quant w/ DFlash2.

  27. Do I have something wrong in my settings? I keep having to restart vLLM to get Qwen3.8-27b-FP8 to stop outputting nonsense / garbage.

  28. Question: if your goal is to run inference, have a bunch of docker containers running your software and bunch of users using your product. What’s the most efficient route given both above options cost the same amount roughly.

  29. We're releasing a fully quantized NVFP4 version of Qwen3.8-27B. The checkpoint was trained using quantization-aware distillation (QAD) with QUASAR, our new QAT algorithm.

  30. Man I love the quality of 27B int8 (currently running lued/Qwen3.8-27B-INT8-W8A16-MTP). But man, it is not fast and I am seriously skating a line with memory consumption on my A40 at 256K context (KV at FP8).

  31. TL;DR: I improved multi-agent batching on iOS by implementing vLLM-style continuous batching in native Swift on MLX. I've written a lot about the tokenomics of local LLMs and agent swarms.

  32. Deploy LLMs on any GPU, anywhere Pick a model, pick a GPU, hit deploy. LLMPanel provisions the container, exposes an OpenAI-compatible endpoint, and streams every GPU metric back to one dashboard — on your own hardware or any cloud.

  33. Some data showing the value of tools on the professional intelligence of Qwen 3.8 27b. My workload with Qwen focuses on real estate, private equity, and adjacent finance stuff.

  34. Hey guys, I'm setting up a local workflow on a single 24GB RTX 3090 to handle project planning—specifically digesting massive (~128k context) requirements documents/PRDs and spitting out a ton of structured .md files to act like Jira ticke…

  35. I had read that I should not have mismatched PCIe configurations for tensor parallelism, so I upgraded my motherboard from MSI PRO-X670-P-WIFI to Asus ProArt X870E-Creator WIFI. With the old board I had gen4 x16 and gen4 x4.

  36. Free, gamified roadmaps for LLM engineering: an Inference Engineering path (KV caches, CUDA kernels, production vLLM serving) and a Model Training path (pretraining on a budget, scaling laws, SFT/DPO/GRPO) — 183 tasks with auto-verified mi…

  37. Has anyone successfully patched the drivers of a modded RTX 3080 20gb to get Rebar support? I already updated my 3090.

  38. I ran the Aider benchmark on Qwen 3.8 27B FP8 with FP8 KV cache 256K context vLLM. The score: 72.9 This matches Gemini 2.5 Pro from 2025-04-12 which also scored 72.9.

  39. Hey all, So I currently use 2x DGX Sparks with Deepseek v4 Flash 0731 and it works fantastic at 1M context, with 1.8M kv, dspark, vllm tp 2, etc. All good.

  40. Hi folks, with tp=2 I get the following logs out of vllm with official Qwen3.8-27b-FP8 with MTP3: [vllm] | (APIServer pid=1) INFO 08-24 05:47:40 [loggers.py:310] Engine 000: Avg prompt throughput: 198.5 tokens/s, Avg generation throughput:…

  41. Hi friends, I've just spent the last few hours to ultimately fail to run Qwen3.8 27B in a docker container. Would anyone happen to have a working docker file they could share the contents of?

  42. I was just curious how the new qwen3.8:27b does on a hard C to HTML porting job against Opus 5 in a default Claude Code. The job: my fun side project is a procedural shooter in a single C file.

  43. Ling Tiny has now replaced Gemma4-12B in my rig as an auxiliary model doing hindsight operations. This is on a 4060Ti, which is a reasonable GPU available out there, and the speed is phenomenal.

  44. How do you guys set it up , i constantly get the error : I tried increasing the contex to 142k and putting the contex size as 115k in DSH , it still did not compress correctly. I have 0 issues if i run it with llama.ccp , it can work for 2…

  45. **Setup:** 2× DGX Spark (GB10, 121 GiB unified each), TP=2 over 2×200GbE RoCE, vLLM 0.25.2.dev0, DeepSeek-V4-Flash-0731 FP8, `max_model_len=450000`, prefix caching on. KV pool = **1,686,693 tokens**.

  46. Hello, So I've been trying lots of combinations in that never-ending landscape of options and settings. I wanted a proper quant of 3.8 27B running as fast as possible on my 5090 at 400W, with vision and with as much KV-cache as possible an…

  47. What I ran: 8x B300 on Modal, $56.79 per hour, vLLM, tensor parallel 8, native MXFP4 Cold boot ~27 min (1.56 TB load, JIT, 51 CUDA graph captures) TTFT 0.92 to 1.02 s, decode 92 tok/s steady, 83 tok/s average over 4 prompts $190 per millio…

  48. This is amazing. All I did was switch from llamacpp on windows to vllm on linux.

  49. Qwen3.8-27B (W4A16-AutoRound) on vLLM 0.27.1, two RTX 3090s, TP=2, no NVLink. Agent: Pi, straight to the local endpoint.

  50. Local LLM Inference: A 96GB Blackwell Field Guide (2026) Running local LLMs on 96GB Blackwell: which models fit, vLLM against SGLang, the throughput you actually get, and the places the setup quietly falls over. The verdict.

  51. I have dual r9700 set up with Ubuntu. With llama.cpp I'm getting about 40 tokens per second for single instance.

  52. This is the Qwen3.8-27B setup I actually use every day on one RTX 5090. I wanted to write it down with enough detail that another 5090 owner can reproduce it instead of guessing which memory knobs I used.

  53. Good afternoon, everyone! I wanted to show the work I've been doing around porting Ninfer over to the CMP170HX (Github) So, first, I do want to call out the amazing work that Neroued, Sergiuszm and specifically Don-Chad have all done, to n…

  54. I took SergiioB's Intel Arc Pro B70 Inference Cookbook and made it into a Helm chart. It applies the patches onto the pinned vLLM version.

  55. Just make sure to leave MTP off. It's a decent upgrade from Qwen3.6 27B, and better than KAT Coder V2.5 Dev (which also requires their custom vLLM).

  56. For those who bought the CMP170HX and unlocked the memory, I have some questions… are you using any specific llama.cpp or vLLM forks? What are you using for cooling the card?

  57. I finally made the move from LM Studio to vLLM thanks to this post https://www.reddit.com/r/LocalLLaMA/s/NmS9CgHvqz. I may not know what it all means yet but I’m going to start diving into the docs to learn as much as I can.

  58. - loading / unloading models - ollama, lm-studio, vllm - optional security tokens and visibility and many more on - https://github.com/Chleba/ollamaMQ

  59. vllm-rocm-windows-RDNA2-oneclick []() []() []() []() []() Native vLLM + ROCm 7.15 (TheRock) for the whole AMD Radeon RDNA2 family on Windows — no WSL2, no NVIDIA, no compiler. One-click installer, everything prebuilt, and an OpenAI-compati…

  60. GenOffice (local-LLM fork) A free, open-source AI Office suite — this fork drops the cloud-account requirement and talks to any OpenAI-compatible endpoint instead: a local server (Ollama, LM Studio, vLLM, llama.cpp server, text-generation-…

  61. The rapid adoption of AI coding assistants and autonomous agentic development systems has coincided with major changes in the pace and structure of open-source software engineering. Yet empirical longitudinal evidence of these changes at t…

  62. kvcachescope A logical memory profiler and state inspector for PagedAttention inference engines (vLLM, SGLang). Standard GPU profilers (nvidia-smi, nsys, torch.cuda.memory_allocated()) observe physical VRAM allocations at the PyTorch tenso…

  63. The smallest async RL trainer I could write: one loop that runs REINFORCE on CartPole on a laptop and async GRPO on a cluster (e.g. 8xH100 trainer, 8 vLLM workers, ran as a [SkyPilot job group](https://docs.skypilot.ai/en/latest/examples/j…

  64. I believe vLLM has been the gold standard for inferencing but I'd like to learn how vLLM, langgraph, langfuse, other observability and business logic is meld into a single setup.

  65. Qwen3.8-2.4T-A95B [!Note] This repository contains model weights and configuration files for the post-trained model in the Hugging Face Transformers format. These artifacts are compatible with vLLM, SGLang, TokenSpeed, etc.

  66. Qwen3.8-27B [!Note] This repository contains model weights and configuration files for the post-trained model in the Hugging Face Transformers format. These artifacts are compatible with Hugging Face Transformers, vLLM, SGLang, TokenSpeed,…

  67. hey everyone, i decided to make a vLLM plugin that implements the Star-KV paper. the results are quiet promising with a decode kernel thats faster than FA2 in higher batch sizes.

  68. The @vllm_project maintainers at @inferact 🚀 are some of the most cracked engineers in the world. They’re building one of the inference engines that powers much of the world’s intelligence—and doing so with remarkable dedication, kindness,…

  69. Meta is back with Muse Glimmer: local, agentic, multimodal, and open source! To celebrate, we are shipping with Meta day-0 support in transformers , llama.cpp , vLLM , Inference Endpoints, and other libraries.

  70. A Better Knob Beats More Silicon: 12 vLLM Serving Experiments on H100s Doubling my GPU count raised throughput 61% — and still lost to a one-line config change. A roofline-guided tour of what actually moves production serving metrics.

  71. Inside vLLM: Anatomy of a High-Throughput LLM Inference System From paged attention, continuous batching, prefix caching, specdec, etc. to multi-GPU, multi-node dynamic serving at scale August 29, 2025 In this post, I'll gradually introduc…

  72. LettuceDetect v2 in Semantic Router: Generative Hallucination Detection as a vLLM Endpoint Semantic Router can now verify grounded responses with a generative span detector served by vLLM. The new endpoint detector backend runs LettuceDete…

  73. Same tokens as vLLM. Same throughput.

  74. DeepSeek-V4-Flash-0731-Latent-Reasoning. A self-contained model that does thinking in latent space, NVFP4-quantized, with a production vllm form for serving runtime.

  75. Benchmark scores are reported as properties of a model, yet the inference framework used to produce them, such as HuggingFace, vLLM, or Ollama, are considered non-influential and their names and versions are almost never disclosed. In this…

  76. Kimi K3 is here: a 2.8T-parameter MoE with a 1M-token context and native vision, running on vLLM from day 0. Here is our canonical deployment guide: architecture, kernels, recipes, and the flags to run it in production.

  77. Quantization became the default way to speed up inference because it is the easiest one. In vLLM it is one flag or one swapped checkpoint (docs.vllm.ai), while doing the real optimization work like writing better kernels takes months.

  78. Come help push the frontier of token speed across local models and hardware with your agents! Current frontier Laguna XS 2.1 · AMD R9700 (llama.cpp HIP) +31.14% 143.3 tok/s Laguna XS 2.1 · DGX Spark GB10 (vLLM NVFP4) +5.28% 37.3 tok/s Lagu…

  79. 📖 Documentation | 🚀 Quick Start | 📦 Installation | 💬 Slack Latest News 🔥 [2026/07] 🚧 v0.25.1 under development — Added Qwen3.5 / Qwen3.5-MoE, Gemma4 (text and multimodal), GLM MoE DSA, and DFlash speculative decoding [2026/02] ⚡ Performanc…

  80. Learn about Red Hat's approach to self-hosted inference with vLLM, addressing the reliability gap between open-weight models and hosted frontier models for agentic workloads.

  81. Kimi K3 Is Here: Efficient Day-0 Support on vLLM We're thrilled to announce efficient day-0 vLLM support for Kimi K3, one of the most powerful open-weight models ever released. Last week, we previewed the production-scale integration work…

  82. 📄 Production-Grade SLM-Powered OCR Course 📄 Build a self-scaling, event-driven OCR pipeline on Kubernetes (AKS / GKE) with Qwen 3.5 + the GLM-OCR SDK Table of Contents Table of Contents Course Overview Who is this course for? Course Breakd…

  83. 🛡️ AI Firewall — Security Gateway & Reverse Proxy for LLM Traffic AI Firewall is a security gateway that sits between your applications and LLM APIs (OpenAI, Anthropic, Gemini, Ollama, vLLM). It inspects prompts before they leave your peri…

  84. Hi HN, Thomas here, I'm a Machine Learning, and I've been deploying models in prod for the last 5 years and decided it was time to open-source some of my learnings! In Coloma you'll find my vLLM tuner, I use it to ensure models don't OOM a…

  85. # Veta Autonomous AI agent swarm for visual, functional, and accessibility testing of Android apps and mobile web — 100% of AI inference runs on AMD GPUs (Fireworks AI on AMD Instinct / self-hosted vLLM on ROCm), with containerized Android…

  86. vLLM x TileRT: Specialized Decode for Latency-Critical Serving Disaggregated serving, which separates the compute-bound prefill phase from the memory-bandwidth-bound decode phase, has become an increasingly standard pattern for serving lar…

  87. The multiplicative repetition penalty shipped across the LLM inference ecosystem (HuggingFace, vLLM, llama$.$cpp, and a dozen further engines) branches on the sign of each raw logit (divide positives by theta, multiply negatives). But the…

  88. Context-Stitcher 🪡 Zero-Copy Context Bridging Gateway for Multi-Agent GPU Inference. 💡 Core Value Proposition In multi-agent collaborative workflows, separate agents often process the same long text context sequentially.

  89. Large Language Models are reshaping how software is developed and maintained. They are typically deployed in production using inference engines such as vLLM, which can efficiently serve pre-trained, highly configurable models.

  90. Native-speed vLLM transformers modeling backend # Upgrade the vllm pip package uv pip install --upgrade vllm --torch-backend auto The transformers library has become the reference modeling library for Machine Learning. It supports 450+ arc…

  91. Hello everyone, After the Claude Code leak started floating around, I spent time studying how the workflow was put together and rebuilt the core experience into my own project. I’m calling it Super Grokie, because it started as a joke but…

  92. 中文 ｜ English 🖥️ Official Website | 💬 GitHub Table of Contents Model Introduction Stronger Agent Capabilities More Reliable Product Experiences Benchmark Appendix News Model Links Quickstart Deployment vLLM SGLang Finetuning Quantization Li…

  93. AKM-CLR — Tenant/Task Governance Layer for Shared LLM Serving AKM-CLR is a validated prototype governance/control layer for multi-tenant LLM infrastructure. It sits above OpenAI-compatible serving engines such as vLLM and helps decide whet…

  94. Hey all, So I tend to favor the Claude Desktop app in Code mode as the GUI does a great job of previewing code, MCP browser interactions/screenshot evals/etc. But I recall people saying they could get Claude Desktop to use a local API.

  95. Automatic Prefix Caching¶ Prefix caching kv-cache blocks is a popular optimization in LLM inference to avoid redundant prompt computations. The core idea is simple – we cache the kv-cache blocks of processed requests, and reuse these block…

  96. While Large Multimodal Models excel in comprehension, high-throughput inference engines lack native support for multimodal generation. This is severe in Speech Language Models, where generating multi-layered audio tokens via decoupled AR+N…

  97. Run a vLLM Server on HF Jobs in One Command It's the quickest way to stand up a model for tests, evals, or batch generation. (If you're after a managed, production-ready service instead, that's what Inference Endpoints are for — more on wh…

  98. I just released ParseHawk v0.1.0: Apache-2.0 licensed 100% local document AI platform that extracts JSON from PDFs, images etc. It builds on top of NuMind's NuExtract3 but additionally enforces a provided JSON schema with constrained decod…

  99. I read it with a little bit of effort The tiny model result is insane, theoretically this could make make a 0.5b on-par with a 2/3/4b ish class model in coding with no weights change*. And for large models it could maybe fix let's say 30-5…

  100. vLLM Recipes — Deploy any model on any hardware with vLLM Pick a model, adjust for your GPUs, copy the vllm serve line that runs. Community-maintained recipes for NVIDIA H100/H200/B200/B300, Grace-Blackwell, and AMD MI300X/MI325X/MI355X.

  101. The rapid growth of large language model (LLM) inference services has increased the demand for efficient multi-tenant GPU scheduling. While modern inference runtimes such as vLLM improve throughput through continuous batching and optimized…

  102. Hi, I currently work on a GenAI platform for one of the largest local industrial companies. My daily work mostly involves building inference infrastructure on top of a 48x H200 GPU, Kubernetes and vLLM.

  103. petite-vllm Part 2: KV Cache & Paged Attention Building an LLM Serving Engine from Scratch · 12 min readKV Cache In part 1 we implemented a simple autoregressive loop and LLM interface. There was no KV caching, which means that at each tok…

  104. Profile A physics-grounded, cost-aware optimization loop for vLLM inference servers. The Problem: vLLM exposes numerous configuration flags (e.g., --max-num-seqs, --enable-prefix-caching).

  105. Retrieval-Augmented Generation (RAG) improves factual grounding, but it also lengthens prompts and raises prefill cost. Prefix caching in serving engines such as vLLM reduces this cost only when requests share the same token prefix.

  106. Running GLM-5.2 5× faster than vLLM, on a runtime that doesn't support it I rented an 8×B200 and tried to run GLM-5.2 on TileRT, the runtime MiMo used to push a 1T model past 1000 tok/s. TileRT doesn't support GLM-5.2, so I reverse-enginee…

  107. IndexCache Accelerating Sparse Attention via Cross-Layer Index Reuse Tsinghua University & Z.ai This repository provides a patch for SGLang and vLLM that enables IndexCache inference acceleration for models using DeepSeek Sparse Attention…

  108. Sors - reorders prompts for LLM to maximize prefix cache hit. A minimal reverse proxy that reorders prompt content to maximize prefix cache hits in LLM inference engines (vLLM, SGLang, or any OpenAI-compatible backend with prefix caching e…

  109. Achieving Up to 67% Cost Savings with Prefill-Decode Disaggregation Using Ray + vLLM on AMD MI325X In LLM serving, the optimization objective is deceptively simple: given a set of latency SLA targets – time to first token (TTFT), time per…

  110. Monitor LLM Inference in Production (2026): Prometheus & Grafana for vLLM, TGI, llama.cpp Monitor LLM with Prometheus and Grafana LLM inference looks like “just another API” — until latency spikes, queues back up, and your GPUs sit at 95%…

  111. Just got 100 tps on generation, but in total time it around 45-60 t/s in case of prompt processing waiting. Available memory show: GPU KV cache size: 152,671 tokens Maximum concurrency for 131,072 tokens per request: 1.16x amd-smi monitor…

  112. Hi everybody, I spent the last two weeks building zerostack, a coding agent in Rust, focused on memory footprint, shipping with ollama and vLLM integrations. I managed to get it to run at ~16MB (with peaks of 24MB) of RAM usage, and no CPU…

  113. I'm trying to use Gemma 4 12B — the new encoder-free unified model (audio/vision/text in one) — for a one-pass audio → response voice assistant: feed the recorded WAV + system prompt and get the reply back as text directly, collapsing the…

  117. Loving the local AI world and been building out my own the last few weeks. One pesky recurring problem is what seems to be related to how the model produces JSON for its responses.

  118. You bought RDNA4 with the promise of go-fast, and it doesn't deliver in vLLM. I know the feeling, out of the box vllm is a complete dog on RDNA4...

  119. I needed a smarter model for my local Hermes Agent setup, so I moved to DeepSeek v4 Flash. First things first: Running 4 concurrent threads on vLLM, I can hit ~400 tok/s 400 x 60 x 60 x 24 x 30 is ~1B TOKENS per month!!!

  121. Hi all, this is a mere "see what others are doing post" rather than a solution to a problem. As newbie, I put together a 2x3090 box that I run vllm on.

  122. OpenEnv is a tool for creating an agentic execution environment like terminals, browsers, or anything an agent can interact with. And today, we’re excited to announce that OpenEnv is becoming even more open, to make the future of training…

  123. vllm-doctor reads metrics from a vLLM server's /metrics endpoint or a Prometheus instance and runs rule-based checks to find what is wrong. It detects queue pressure, high TTFT/TPOT, KV cache pressure, and other rules across pods.

  125. when running by using transformers it runs by using vllm some weird error come up plese can any body share the command of running it on vllm ?

  126. It’s finally here! Something many of us running dual RTX 3090 rigs have been anticipating.

  127. Im into both HPC and 3D reconstruction, so I built this as a side project. dvlt.cu is a single 5MB binary: - No python, torch, TF, ONNX, llama.cpp, vLLM, or huggingface runtime - Nearly no dependencies: only cuBLASLt (shipped with libcuda…

  128. Has anyone figured out how to activate MTP for Gemma4’s new QAT q4_0 GGUF for 31b? Or is this still not supported in llamacpp?

  129. Are there any quality voice cloning and speech generation models that already have support in Llama.cpp or, more likely, vLLM-Omni? It would be nice to swap them out like any other inference model and use a common API, rather making a sepa…

  130. NVIDIA DGX Spark GB10 — AI Models & Inference Guide Welcome to my repository and guide for running, optimizing, and benchmarking state-of-the-art AI models on the NVIDIA DGX Spark deskside supercomputer, powered by the cutting-edge NVIDIA…

  131. vLLM: An Efficient Inference Engine for Large Language Models by Woosuk Kwon A dissertation submitted in partial satisfaction of the requirements for the degree of Doctor of Philosophy in Computer Science in the Graduate Division of the Un…

  132. LLMhop is a tiny stateless proxy for LLM inference servers. It tackles an issue I faced when trying to serve more than one local LLM at once which is not natively supported by vLLM.

  133. Fast & Efficient LLM Inference with vLLM: A New Course with DeepLearning.AI We're excited to announce, with Red Hat and Andrew Ng's DeepLearning.AI, a hands-on course that walks through LLM fundamentals and the full optimize, deploy, and b…

  134. ⚡️ Built for agentic and long-context workloads. 💡 KVarN delivers 3-5x more KV-cache capacity and up to ~1.3x the throughput of FP16, so you fit far longer contexts and serve more concurrent requests, with FP16-level accuracy.

  135. Apply quantization to shrink a model's memory footprint, then measure the accuracy tradeoff. Fast & Efficient LLM Inference with vLLM Instructor: Cedric Clyburn Earn an accomplishment with PRO - Intermediate - 1h38m - 9 Video Lessons - 3 C…

  136. tiny-vllm You're going to build a high performance LLM inference engine with C++ and CUDA - tiny-vllm, a younger and smaller sibling of vLLM We will learn a lot along the way, make mistakes and derive the ideas and maths from scratch This…

  137. looks like you can run it on any potato (A1B)! https://huggingface.co/LiquidAI/LFM2.5-8B-A1B-GGUF from LiquidAI: LFM2.5 is a new family of hybrid models designed for on-device deployment.

  138. EDIT - IGNORE. I MADE A MISTAKE.

  139. I have been coming to this subreddit to understand what the optimal config is to run a model on a given hardware setup. I referred to specific benchmarks, but they are too generic and do not consider the underlying hardware.

  140. Context: I am testing and running local LLM on Linux for some months, first with llama.cpp and now with vLLM for better concurrent capabilities. I use llama-swap in front of either vLLM or llama.cpp in order to have thinking and non-thinki…

  141. Used the vllm version of https://github.com/noonghunna/club-3090 It worked fine for myabe 20 40k context, havent tried the new one. Anyone used the new llama.cpp patched one for single 3090?

  142. Okay fun time I got access to two Nvlinked A100s for some research project I benchmarked my work against the Gemma 4 31b-it available through Google, but my dataset is rather massive, so I need to run it on the "local" resources. Basically…

  143. Best I tried so far is https://huggingface.co/nsparks/DeepSeek-V4-Flash-FP4-FP8-GGUF with the custom llama.cpp fork, but it suffers from low quality and random incoherent output. VLLM wouldn't support anything other than H100s for DS4.

  144. I was given the great opportunity to borrow a H100 with 94GB VRAM at work until it is needed by a customer. (No idea how much system ram I will get, but I guess they are a bit flexible on this).

  145. Hi Reddit, I am planning on running Qwen 3.6 27b NVFP4 via vLLM on my 5090 but was wondering if something like 35b a3b at Q8 on Llama would produce better results for agentic coding and utilize the system memory. My research says no but if…

  146. https://github.com/user-attachments/assets/e4897391-c5a8-4391-93c3-9f8b76155f11 Setup your local LLM stack effortlessly. Starts fully configured Open WebUI and Ollama harbor up Now, Open WebUI can do Web RAG and TTS/STT harbor up searxng s…

  147. I'm usually not posting about Harbor releases out of the respect for the community here, but I think v0.4.19 might save a lot of people some time. Harbor can now launch your local agentic coding tools with local inference backends.

  148. Posted on other feeds last week and figured some of you out here might be interested as well; Someone commented asking if it supported OpenAI-compatible endpoints (LM Studio, vLLM, OpenRouter, Together, Groq, LocalAI…), so i have spent few…

  149. EAGLE 3.1: Advancing Speculative Decoding Through Collaboration Between the EAGLE Team, vLLM, and TorchSpec The EAGLE series — including EAGLE 1, EAGLE 2, and EAGLE 3 — has become one of the most widely adopted and practically deployed fam…

  150. Just released a blog on a side research project I have been doing for the past two months and would love for you all to check out and see how it is! It's about output length-constrained summarization using LLMs with GRPO.

  151. link: https://huggingface.co/JC1DA/Qwopus3.6-27B-v2-INT4-W4A16-Autoround Super surprised how good Jackrong's model is... It's taking so much time to evaluate the all the base qwen3.6-27B, Jackrong's version and other's quantized models but…

  152. Wondering what’s the best coding model that can fit on a RTX 3060 (12GB). Has anyone been able to do something useful with it?

  153. NuExtract3 is a unified 4B vision-language reasoning model for document understanding. It combines strong structured information extraction with high-quality image-to-Markdown conversion, making it suitable for extraction pipelines, OCR, a…

  154. Got a chance to play around with 2x RTX PRO 6000 setup so sharing some number for Qwen 3.6. All these were run using latest stable VLLM backend.

  155. My use-cases will be to test open-weight LLMs and work on harnesses, inference systems and possibly other non-ML workflows (CS-related) in the future. Fine-tuning would not be something I do locally because I can rent a B200 from RunPod fo…

  156. Everyone remembers that sneaky download of Gemini Nano earlier this month? and if you talk to it, it will happily tell you it’s a Gemma.

  157. Got tired of $160 Opus bills so I spent a weekend wiring up a routing layer on vLLM 0.8 (2xA100, enable_auto_tool_choice). Getting the tool call parser to cooperate took longer than the actual routing logic.

  158. Looked at my coding agent's bill last month: $340 for repo maintenance across three repos, each around 15k lines. Most of those tool calls were just grep and file reads.

  159. Run production LLMs on your own hardware A Kubernetes operator for self-hosted LLM inference. vLLM, llama.cpp, TGI, NVIDIA, Apple Silicon.

  160. nvidia-smi topo -m is showing the both GPU as PHB (i.e. via CPU) connected as expected but I cannot get NCCL all_reduce_perf to run at all, it always hangs after starting up.

  161. Edit: does this happen every time a newbie tries to post here. Getting roasted despite having valid results?

  162. I work on AxonFlow, a source-available (BSL 1.1) runtime for long-running agent workflows. We’ve been running it in front of Ollama-served models and OpenAI-compatible local endpoints (llama.cpp `--server`, vLLM, LM Studio).

  163. I'm daily driving dual Asus GX10 (spark) with vllm and it's fantastic. But I want to try model that is GGUF only and won't fit into single spark.

  164. This has turned out to be useful to many of my friends so I thought I'd share here as well. I created a tool and documentation page for most major open-souce project's adherence to 'OpenAI compatibility' after seeing inconsistencies betwee…

  165. I’m running Hermes Agent on a single NVIDIA DGX Spark using vLLM with: docker run --gpus all \ --name qwen36-aggressive \ --restart unless-stopped \ -p 8000:8000 \ --ipc=host \ --ulimit memlock=-1 \ --ulimit stack=67108864 \ --shm-size=32g…

  166. Hello guys, two days ago i ran the spark-arena for my Qwen 3.5 122B Recipe on a single DGX Spark and I got the highest score on speed for any context length and concurrency across all 3.5 122B Int4 Recipes. Just wanted to share if somebody…

  167. I posted recently about EvalShift, an OSS CLI for regression-testing LLM model changes. A few people pointed out that for LocalLLaMA, the more interesting use case may be quantization regression: Q8 -> Q4_K_M Same base model, same prompts,…

  168. Greetings from former TurboQuant's biggest defender, now middle-sized niche-aware TurboQuant defender. Today I'm presenting to you the results of me thoroughly exploring the world of PPL and KLD benchmarks with my single RTX 3090 using Bee…

  169. I posted earlier about RTX 5060 Ti local LLM testing, and I have cleaned the repo up quite a bit since then. The project is now a more structured benchmark/recipe repo rather than scattered notes.

  170. I'm on a quest to profile and benchmark different GPUs for PyTorch, vLLM, and llama.cpp. Cannot find the high-end AMD consumer cards for rent anywhere online and interested in the PyTorch ROCm performance of the 7900 XTX (if you want to co…

  171. Built this for myself after wanting to use local LLMs during work calls without the window showing up on screen share. Every existing tool was either cloud-only or a 200MB Electron app.

  172. TL;DR best setup I tested on a RTX 3090 24 GB: ik_llama.cpp + Qwen3.6-27B-MTP-IQ4_KS.gguf 156k context, q8_0/q8_0 KV, MTP, vision on CPU benchmark result on a ~5.9k prompt + 1k output: about 1261 tok/s prefill, 72.9 tok/s decode llama.cpp…

  173. I have been running some benchmarks on a heterogeneous 7-GPU cluster to see how different inference engines handle long context prefill using pipeline parallelism. My setup consists of a mix of Blackwell and Ada cards: one RTX PRO 6000 96G…

  174. Hi all, Not sure many people are aware so wanted to give a word about Fara-1.5 release. => this release will likely be the big sister of Fara-7B and built on top of Qwen3.5 Actual Fara-7B performs not bad at all but actually requires a pro…

  175. Hey all, While nvfp4 still seems to be a work in progress, the latest version of vllm 0.21 finally has mtp working for gemma. With all the talk of qwen being badass I thought I would revisit gemma.

  176. A briefing on the inference memory hierarchy: prompt layout, host-side shared KV, distributed lookup, RDMA transfer, encoder reuse, and evidence discipline. Covers vLLM × Mooncake, LMCache MP, LMCache CacheBlend, SGLang, NVIDIA Dynamo, and…

  177. Hello, I'm currently using Ollama / lm studio for things like code inference and proof reading emails, etc. Definitely not experienced in this space but looking to grow.

  178. I kept seeing inference-speed claims for these models and wanting an apples-to-apples comparison on the hardware I actually have. So I built a harness and a public page that dumps every run as YAML.

  179. Simple question: Has anyone run two or more of either of these on Ubuntu ? Intel Arc Pro B70 (32 GB) Intel Arc Pro B65 (32 GB) Running llama or vLLM etc., Any thoughts

  180. https://preview.redd.it/8o43bjhe9d1h1.png?width=5346&format=png&auto=webp&s=1c87c2ee8b8ffff43495f543266056b0e26d3947 In another post I had someone ask me about the power draw of the 4x 3090 setup so I'm sharing a a full test I conducted to…

  181. I put together a small public repo for RTX 5060 Ti 16GB local LLM setups: I took inspiration from the club-3090 repo, but this one is focused on documenting what we’ve actually tested on 5060 Ti hardware so the setup details are easier to…

  182. Ok, hear me out. This all started when I was trying to understand why this Qwen3.6 27B INT8 Autoround (https://huggingface.co/Minachist/Qwen3.6-27B-INT8-AutoRound/tree/main) recipe was performing so much better than any other Qwen3.6 27B q…

  183. Hi, i just wanted to share what im playing with for last couple weaks. I built my own AI harness: TinyHarness My main goal was low memory footprint, it is not written in Typescript/Javascript/Python, leaving as much memory as possible for…

  184. Shipped this for the AMD x lablab hackathon. Attached video is one of the actual reels the pipeline produced - one English sentence in, finished mp4 with characters, story, music, and voice-over out (fast demo video, not the best quality).

  185. TL;DR Results from the title are for single inference with 2 prompt of 1k and 15k tokens. So no MTP (as it’s slower for big prompt), no DFlash (working too but slower for big prompt), no quant used (full precision wanted) and the results a…

  186. https://preview.redd.it/74cj1xu9pw0h1.png?width=1229&format=png&auto=webp&s=3ae999cc3530ecb4eccf70e25f1a9eb2aa3f2d7b Sometimes qwen 3.6 just stops at the middle of a task, is there a way to avoid it? This is qwen-code CLI, but also happens…

  187. So, as most of us here are, I'm a llama.cpp loyalist. Easy to understand, great configuration, relatively stable, etc.

  188. EDIT: Edited to provide more clarity It occurred to me, that perhaps the same draft model used for speculative decoding would be completely adequate if we just used it's output as-is for reasoning, without validating the results against th…

  189. I would like to dedicate a budget of about 500 euros to upgrade my workstation and run inference on the qwen 3.6 27b and gemma 4 31b models. I currently have an RTX 5060Ti 16GB.

  190. I've been using the int4 Autoround quant from "Lorbus/Qwen3.6-27B-int4-AutoRound" and it has been pretty good! Great quality and performance on an RTX 5090 vllm.

  191. Benchmarked Gemma 4 MTP and z-lab's DFlash on a single H100 80GB using vLLM and NVIDIA's SPEED-Bench qualitative dataset. Setup: Hardware: 1x H100 80GB Runtime: vLLM Dataset: SPEED-Bench qualitative Prompts: 880 total, 80 prompts across ea…

  192. I’m running locally with 2 RTX 3099s and 128gb of RAM I run my workflows with Hermes/OWUI and use Comfy for media generation. My inference is with LM Studio.

  193. I am looking for recommendations on the best way to run local LLMs on NVIDIA DGX Spark. Which stack makes the most sense in practice: TensorRT-LLM, vLLM, or llama.cpp?

  194. TL;DR: DeepSeek-V4-Flash running at 85.52 tok/s @ 524k ctx and ~111 tok/s @ 128k single-stream on 2× RTX PRO 6000 Max-Q pasta-paul's DeepSeek-V4-Flash-W4A16-FP8 quant is great, but its MTP head silently gets stripped at load time (HF trans…

  195. I have been trying various NVFP4 based variations of Qwen 3.6 27B, and I am seeing this for the ones that look most interesting to run on my 2x 16GB VRAM with KV cache fp8. vllm | (Worker_TP0 pid=136) WARNING 05-09 13:49:27 [kv_cache.py:10…

  196. vLLM has the ability to run .safetensors LLMs before they are converted to GGUF and represents a new engine to explore. I personally had never tried it out until u/krishna2910-amd/ u/mikkoph and u/sa1sr1 made it as easy as running llama.cp…

  197. Past few days, its all been about MTPs. Somehow people missed out the fact that Z lab released the Dflash for Gemma4 26B a couple of days ago.

  198. I ran a benchmark to see how much DFlash speculative decoding actually helps in vLLM. Setup: GPU: RTX 5090, 32GB VRAM vLLM: 0.19.2rc1 Main model: cyankiwi/gemma-4-26B-A4B-it-AWQ-4bit Draft model: z-lab/gemma-4-26B-A4B-it-DFlash Workload: r…

  199. TL;DR On 4× RTX 3090 with NVLink bonded between GPU pairs (0↔2 and 1↔3), pinning TP=2 to a NVLinked pair gave +25% throughput at concurrency 1 and +53% at concurrency 4 vs running TP=2 over PCIe. Adding the other two GPUs to make it TP=4 m…

  200. I've been running a persistent local agent for about 2 months - hundreds of sessions, mix of local models (llama.cpp/vLLM/lmstudio) and paid (Claude). One of the things that has been driving me nuts with OpenClaw and Hermes is the way memo…

  201. Some of you saw our post a couple weeks back about hitting 102 tok/s stable on Qwen3.5-35B on a DGX Spark. A lot of you asked "cool, where's the code?" Today's the day: Github Atlas is open source.

  202. I frequently see (both here and on r/LocalLLM ) comments that multi-gpu setups are complex, problematic and typically bottlenecked by PCI-E bandwidth on consumer motherboards. I am running 2x RTX 5060 TI 16gb ( and about to add a third ),…

  203. Granite Switch is an open-source IBM Research project for composing several task-specific LoRA adapters into a single deployable Granite model checkpoint. The idea is to get the accuracy benefits of multiple fine-tuned models without havin…

  204. vLLM V0 to V1: Correctness Before Corrections in RL TL;DR. vLLM V1 matched our vLLM V0 reference after we fixed four things: processed rollout logprobs, V1-specific runtime defaults, the inflight weight-update path, and the fp32 lm_head us…

  205. Hiya, today I was trying to get a response from GPT-OSS-120B via vLLM - and failed miserably! Has anybody gotten it to work, i.e.

  206. Hey everyone, ​I'm running into a weird issue and hoping someone here might have a fix or some troubleshooting ideas. I'm currently trying to run the new Gemma 4 31b-it model using vLLM (v0.20.0-cu130) deployed via Helm chart (https://gith…

  207. Just tested Gemma 4 31B with the new official MTP Drafter on my H100 today and compared the approach with DFlash to help you decide which one to use. Without drafter: 13.7 tok/s.

  208. emotion-steering Extract and serve CAA-style emotion steering vectors for any HuggingFace causal LM, with a fast vLLM path for Qwen3. ┌────────────┐ ┌────────────────────┐ labeled │ extract │ vectors + AUC report │ serve │ contrasts ├─────…

  209. This project was born out of time I spent digging into a biologically inspired algorithm I was using to measure co-activation for placement of experts and ranks onto chips. The default scheduling that vllm provides can end up causing laten…

  210. Previously it was throwing a 'Not Implemented' error due to Mamba layers. Going to test it now!

  211. Last year researchers affiliated with NVIDIA, University of Warsaw, and University of Edinburgh published Dynamic Memory Sparsification (DMS), a KV-cache sparsification technique using learned per-head token eviction, reporting up to 8x KV…

  212. I have a GPU server (4 × RTX3090s) that I've been using for research and PoC in the past 2 years. Mostly running vLLM for Qwen, GPT-OSS, and Gemma.

  213. If one has enough vram, would Sglang be a superior choice than vLLM or llamacpp in terms of inference speed for serving a model dedicated to powering a personal (single user) agent harness like Hermes agent? Sglang has MTP for speculative…

  214. Hi all, I come across to relatively niche problem and could not find much useful posts or guides about it. I have a mini pc (Beelink Ser 8, 8745HS and 32GB 5600 DDR5 SODIMM) headless server for hosting some routing services, and I am wonde…

  215. I have been using llama.cpp to run some models recently. For example, I've been running GLM-4.7-Flash with this command .\llama-server.exe -hf unsloth/GLM-4.7-Flash-GGUF:Q6_K_XL --alias "GLM-4.7-Flash" --host 127.0.0.1 --port 10000 --ctx-s…

  216. Valkyr is a fresh take on LM Inference runtimes. It's quite different from llama.cpp, vLLM, or ZINC for example.

  217. Hi everyone, I'm considering building a home LLM inference rig around: - HUANANZHI H12D-8D + BMC - AMD EPYC 7002/7003 - 4x RTX 3090 24GB - DDR4 ECC RDIMM, 8-channel - Linux + vLLM / SGLang / llama.cpp - Open frame, PCIe 4.0 x16 risers The…

  218. I'm about to do a clean install of Ubuntu 26.04 on a desktop that has a 5060ti 16gb and a 4060ti 16gb. Can you help me work out the best local AI setup for my use cases?

  219. Hello everyone, I think Qwen 3.6 27B is good enough that it might take a while before we get a clearly better model at a similar size. I have a single headless RTX 3090 with a 300W power limit.

  220. Hey guys, A couple of weeks ago, I asked this sub for the hardest Vision use cases you were dealing with to test the newly dropped Qwen 3.6 against Gemma 4. I finally finished running the gauntlet side-by-side locally on vLLM (FP8 quants)…

  221. I run Qwen-3.6 27B FP8 on vllm for long-horizon agentic coding harness workloads with high context window and concurrent sub-agents. On two 3090s that aren’t used for anything else, it seems reasonable to expect a good balance between spee…

  222. I have been trying Mistral 3.5 on my 4x RTX 3090 rig with llama.cpp. Inference is slow (about 11 t/s) even without anything being offloaded to the CPU.

  223. The angle here is native Windows, no WSL. Simple installation, open source, no telemetry.

  224. I wanted to use vast.ai, but ollama doesnt have it, and when i used vLLM I didn't have success. I genuinely don't know what failed.

  225. Most of my LLM cost was on the wrong tier of work. Classification, extraction, JSON formatting, summarization I'm going to review anyway.

  226. I am currently running dual RTX 5060 ti 16gb (both of which are easy to sell or re-use in other PCs at home) and monitoring the used market for more of the same and alternatively RTX 3090. I couldn't help but notice that sometimes some qui…

  227. Just got myself (for my company) a RTX Pro 6000 Blackwell Workstation card. Managed to get really good TPS on qwen3 27b fp8.

  228. 32 MI50 32GB setup moonshotai/Kimi-K2.6 int4 @ 9.7 tok/s (output of 136 tok) and 263 tok/s (input of 14564 tok) on vllm-gfx906-mobydick Github link of vllm fork: https://github.com/ai-infos/vllm-gfx906-mobydick Power draw: ~640W (idle) / ~…

  229. Has anyone tried mixing 3090s with 3080 20G for vllm using tensor parallelism? I know vllm normally discourages mixing GPUs, but given how much 3090 is selling nowadays, the modded 20G 3080s with half the price feel like better deals.

  230. Following up on our previous post about running Qwen3.6-27B on a single RTX 3090 (~125K context, higher TPS). We’ve been pushing further on both context length and stability for tool-agent workloads.

  231. I see a big difference in agentic coding between gemma-4-31B-it-Q5_K_M and gemma-4-26B-A4B-it-UD-Q8_K_XL. The 26B model is much faster because of A4B and generally works well, but there is a big difference in thinking.

  232. Reasoning Guard: Stopping LLM Thinking Loops at the Proxy Layer I’ve been running Qwen3.6 MoE behind a vLLM proxy and hit a specific reliability issue: occasional runaway reasoning loops. This isn’t a criticism of Qwen3.6.

  233. Just a sanity check, this is too slow and something is wrong, right? Like, this is setup with mtp, vllm with awq quants, I suspect that I did configure something wrongly.

  234. Trying some if Bartowski's Q4 quants. Using Vulkan with the latest main branch as of a few hours ago.

  235. I see a lot of support for running LLMs on PCs with ollama to vLLM. Whats the current state for running on mobile?

  236. Okay, so I have this quad 5060ti setup and for forever I have had people nagging me to try vllm. I thought it was too complicated, like varsity golf or putting on both legs of pants at the same time.

  237. I’ve been testing Qwen3.6 27B on a pretty non-standard local setup and figured the numbers might be useful for anyone looking at the newer 16GB Blackwell cards. Hardware: 2x RTX 5060 Ti 16GB 32GB total VRAM Proxmox LXC 16 vCPU ~60GB RAM CU…

  238. vLLM-compile: Bringing Compiler Optimizations to LLM Inference Luka Govedič vLLM Committer Senior Machine Learning Engineer, Red Hat 1

  239. Has anyone got a reliable vLLM recipe for 3.6 27B that fixes the tool calling issues? I am getting "Not let me..." - then nothing.

  240. Disaggregated Serving for Hybrid SSM Models in vLLM Introduction Hybrid architectures that interleave Mamba-style SSM layers with standard full-attention (FA) layers — such as NVIDIA Nemotron-H — are gaining traction as a way to combine th…

  241. Anyone have experience with both? Luce is llama.cpp with custom dlflash and noonghunnas project is vllm with patches.

  242. Hello all, the docs for the vLLM production stack suggested autoscaling the vllm worker instances based on the number of waiting requests, but it seems like this would only help with new coming requests? We are having burst LLM calls which…

  243. Trying to find the sweet-spot to tradeoff between power and tg/s. 250W seems to be a sweet spot for Qwen3.6-27B.

  244. I have a 4 x R9700 system on Threadripper pro, but I have never been happy with the performance of my GPUs in vLLM. I have started benchmarking any new model I try out with llama-benchy so that I can get a better idea of how models of diff…

  245. Qwen3.6-27B vLLM Docker Docker-based vLLM serving for Qwen3.6-27B with Lorbus AutoRound INT4 quant and MTP speculative decoding. Model is downloaded at runtime and stored on a host volume so the container can be upgraded without redownload…

  246. I'm curious if any brave soul has upgraded their computer (especially if it's Strix Halo) from Ubuntu 24.04 -> 26.04 and seen a significant performance improvement for inference with VLLM, llama-server, and/or LM Studio.

  247. I'm a daily llama-cpp user and was hoping to try MTP on vLLM. Unfortunately, pipeline parallelism + MTP does not seem to work with this model in vLLM.

  248. In case anyone is interested, I decided to test out LLama.cpp's new OpenVino backend to see how it compares on Intel GPUs. At first glance, it stomps all over the previous best-case, SYCL, but lags behind LLM-Scaler (Intel's VLLM fork), li…

  249. locally uncensored is a desktop app that combines four things most people run separately: chat, a coding agent, image generation, and video generation. all local, all on your hardware, no docker, no cloud account needed.

  250. I'm running Qwen/Qwen3.6-27B-FP8 via vLLM using this command: vllm serve Qwen/Qwen3.6-27B-FP8 --tensor-parallel-size 4 --gpu-memory-utilization 0.95 --max-num-seqs 8 \ --enable-auto-tool-choice --tool-call-parser qwen3_xml \ --enable-prefi…

  251. I just got Qwen3 72B Instruct running on a high RAM setup and I’m kinda confused about the proper way to use it. What’s the correct workflow for running it smoothly (like best quant, tools, or runtime)?

  252. I've heard mostly bad opinions about multiple slots with llama.cpp (--parallel > 1). I guess comparing to vLLM it might be worse at this, but I recently tried vLLM on 4 slots and it indeed improved the overall speed significantly (150-170t…

  253. What it says in the title, I'm interested in hearing what you all have landed on as a workable / useful stack for you. Mine looks like this: back end inference servers - llama.cpp, vLLM | V hermes-agent - cron jobs + OpenAI compatible endp…

  254. Apologies in advance, if this is a newbie question. When running Qwen3.6-27B-FP8 using the below command on an Nvidia RTX PRO 5000, in opencode, I am seeing errors such as: "The issue is that the JS file is too long and causing JSON trunca…

  255. https://preview.redd.it/c76w57d1yexg1.png?width=1482&format=png&auto=webp&s=1164d8bc3e2e8a4157f26dd5583238a736474932 KLD for INTs and NVFP4s. AS ALWAYS - Use Case is important.

  256. Hi all, Looking for some advice with a GX10 I purchased about 4 months ago. I've been having all kind of issues trying to run local models on this device.

  257. Qwen3.6-27B is out for a few days and the NVFP4 with MTP is dropped earlier on HF: https://huggingface.co/sakamakismile/Qwen3.6-27B-Text-NVFP4-MTP Can follow the same recipe I used for Qwen3.5-27B to achieve ~80 tps on a single RTX 5090 at…

  258. I wanted to figure out which of the newer small and mid-size models are actually worth running on a single H100, so I put 8 of them through a proper vLLM benchmark and recorded what came out. The setup was simple.

  259. As far as i know the weight is of 160gb + 9.6gb needed for max 1 million token window + 5 gigs overhead = 175gb vram. But vllm and othere sources said "To use the full 1M context, you need 4x A100 80G" --> thats a 320gb vram ??

  260. I'm sure people have asked before for settings for these gpu's, but for me, no matter what I do, It doesn't work as good as 3.6 35B! I've tried VLLM and LLAMACPP .

  261. Easy, fast, and cheap LLM serving for everyone | Documentation | Blog | Paper | Twitter/X | User Forum | Developer Slack | 🔥 We have built a vLLM website to help you get started with vLLM. Please visit vllm.ai to learn more.

  262. Hey guys, I built a custom vLLM pipeline to run Gemma 4 (31B FP8) and Qwen 3.5 side-by-side locally to see how they actually perform in the wild with preprocessing of audio and images. But of course new model Qwen 3.6 27B came out just whe…

  263. I have tested Qwen3.6-27B-Uncensored-HauhauCS-Aggressive-Q5_K_P.gguf on my 4x3090 system (opencode) and find it really good and fast. However, I can't find any uncensored models for vllm (preferably as AWQ).

  264. DeepSeek V4 in vLLM: Efficient Long-context Attention We are excited to announce that vLLM now supports the DeepSeek V4 family of models (deepseek-ai/DeepSeek-V4-Pro and deepseek-ai/DeepSeek-V4-Flash ). These models feature an efficient lo…

  265. Got a new playground at work. Anything I cn help run (via vllm maybe) that you might be curious about.

  266. I am currently working on a system to translate books/web novels. I got a working prototype, but now I am looking into optimizing it.

  267. Hi guys, I’ve been running side-by-side experiments on Gemma 4 (31B FP8) and Qwen 3.5 Vision for the last few days using vLLM in Docker to see how they actually handle real-world images and video. A few things I found out: 1.

  268. Always been stuck with models that fit on my 16gb .... Going to have about a week for free with 4x rtx6000pro .

  269. Hi everyone, I’m building a small support chatbot in Symfony for a limited group of users (around 300 people). For the MVP, I’m running everything locally on an NVIDIA DGX Spark with the GB10 Grace Blackwell superchip, using vLLM.

  270. Claude Code drafted the prose. I did the research, direction, architecture, ran the code, caught the bugs, and reviewed every commit.

  272. I am using an Arc Pro B70 to do inference, and it's token generation speed is fine using Ollama, but it takes *forever* to do a prefill. vLLM absolutely tackles the prefill problem (nearly instant responses), but I can't run nearly as larg…

  281. Hi guys I managed to get a multi GPU setup going with a 3090 and three 3060 bringing my vram to 60gb along with 64gb ddr5. The objective is to run the largest coding model I can at a respectable token speed of over 20 tokens / second.

  282. Setup: 3x Mac Minis in a cluster running MLX. One node drives training, two push rollouts via vLLM.

  283. been using local AI for a while now but my workflow was a mess. ollama for chat, comfyui for images, different tools for video and coding.

  284. I'm trying to get past generic "best model" recommendations and collect real-world configs from people on similar hardware. My setup: MacBook M1 Pro, 10-core CPU, 14-core GPU, 16 GB unified memory.

  285. MLX https://github.com/Blaizzy/mlx-vlm?tab=readme-ov-file#turboquant-kv-cache vLLM https://github.com/vllm-project/vllm/pull/38479 MLX & vLLM users, please share your experience with benchmarks(t/s). Adding llama.cpp Links related to Turbo…

  286. Im running the raw version straight from the minimax release on hugging face (https://huggingface.co/MiniMaxAI/MiniMax-M2.7) on 3 rtx pro 6000's on vllm. So no quantization.

  287. Inference engine used (vllm fork): https://github.com/ai-infos/vllm-gfx906-mobydick/tree/main Huggingface Quants used: cyankiwi/MiniMax-M2.7-AWQ-4bit Relevant commands to run: docker run -it --name vllm-gfx906-mobydick-mixa3607 -v ~/llm/mo…

  288. Here is how to run the new Qwen3.6-35B-A3B > At full context on a 4090 - IQ4_XS gguf with llama cpp > At full context on a Spark - FP8 with a tweaked vLLM Here is the docker compose with llama cpp services: llamacpp: container_name: llamac…

  289. Just swapped Qwen 3.5 for the 3.6 variant (FP8, RTX 6000 Pro) using the same recommended generation settings. My stack is vLLM (v0.19.0) + Open WebUI (v0.8.12) in a RAG setup where the model has access to several document retrieval tools.

  290. Qwen3.6-35B-A3B [!Note] This repository contains model weights and configuration files for the post-trained model in the Hugging Face Transformers format. These artifacts are compatible with Hugging Face Transformers, vLLM, SGLang, KTransf…

  291. There have been many TurboQuant implementations recently in llama.cpp, mlx, vllm, and sglang, but a lot of the discussion and code around them feels pretty noisy and looks to be AI-generated. I’m trying to understand which claims from the…

  292. Hey r/LocalLLaMA, I've been running a small 4-node DGX Spark cluster on a 400µT fabric switch and got frustrated with the usual raw Ray/vLLM scripts and EXO basically ignoring pure NVIDIA paths. I started from the solid foundation in [eugr…

  293. I cant get gemma 4 e2b or gemma 4 e4b to run on my laptop. I am runnning it via docker as per vllm website and i get the error : Free memory on device cuda:0 (9.71/11.5 GiB) on startup is less than desired GPU memory utilization (0.9, 10.3…

  294. For my earlier use case I used to host qwen 2.5 vl 7b gptq int4. Now I was looking to switch to Gemma4 26B A4B, as it would improve performance as well as improve latency considering only 4B parameters are active..

  295. Hi all, first post here. I've started a project in OpenClaw a month ago, and it's been a very "intense" 4 weeks to say the least...

  296. So, yesterday run was a success and I did get an avg rollout length of about 64 tokens as attached in the image! This was with quality_reward + length_penalty (more info below!) Next, I'll be going with length penalty as the reward and wit…

  297. Just got a DGX Spark set up today and starting to configure it for local LLM inference. Plan is to run: • vLLM • PyTorch • Hugging Face models as a local API backend for an application I’m building (education / analytics use case, trying t…

  298. Deep Dive into Efficient LLM Inference with nano-vLLM A look inside a lightweight implementation of vLLM. KV cache, paged attention, tensor parallelism &multi-GPU support, etc.

  299. I am playing around with Intel Arc B70, still trying to decide whether I keep it or not. After some battle, I got it working with Radeon 5500 and B550M, now I am on to the fun part of getting software to work.

  300. I'm trying to figure out what sort of hardware setup i will need to accomodate a userbase of 100 users (not necessarily concurrent). Does anyone have any idea what sort of setup i'd be looking at?

  301. Fleet Watch Process governance for AI workloads on a single machine. The Problem You're running MLX, Ollama, vLLM, Candle/Cake, experiment runners, and AI coding agents on the same machine.

  302. Hi folks - is there any "standard" (acceptable) vllm way of implementing turboquant or a similar rotational quant for vllm's kvcache? I found https://github.com/mitkox/vllm-turboquant - but this seems inactive.

  303. I’m working on an LLM agent setup (using Qwen-style chat templates with tool calling), and I ran into a design trade-off that I’d like to get some insights on. In these templates, the full tool definitions (JSON schemas) are injected into…

  304. Two questions: which model? In my mind, Qwen3.5 27b or Gemma 4 31b are top options.

  305. So vLLM recently added the feature to use GGUF quants with the syntax author/model:quant format. I was just wondering if people were able to use the quants on older vLLM versions.

  306. my current build is just a 16GB 5060Ti running on a 3800X with 32GB DDR4. not really anything special, but I only really use it right now for Qwen3-VL-8B-Instruct at INT8 to do handwriting transcription (and it works great for that). someo…

  307. Hi everyone, I’m building a local AI pipeline on WSL2 (Ubuntu) specifically for Product Visualization. My goal is to orchestrate LLMs for scene generation and Stable Diffusion/ComfyUI for high-fidelity rendering, keeping my Windows host cl…

  308. I'm an ex crypto miner with remnant mining parts so I threw them together into a franken hydra case. I've been using claude oath previously, but they just shut that door last week or so.

← all threads