Serving GLM-5.3 at 607.8 tok/s across two Nvidia DGX stations with vLLM (developer.nvidia.com via hn) vllmglm 1d
topic
Vllm
-
Running a frontier-scale model locally means keeping all its weights in GPU memory; the largest models no longer fit on one machine. Two NVIDIA DGX Stations connected over ConnectX-8 RDMA serve one model through a single endpoint, with ten…
-
CyberGate – vLLM plugin to unlock cybersecurity without weight edits (github.com via hn) vllm 3d
CyberGate: vLLM Plugin for Cybersecurity Uncensoring Without Weight Editing English | 简体中文 -8b5cf6.svg) A drop-in vLLM 0.32 plugin that unlocks cybersecurity, CTF, and reverse-engineering capabilities per request - using original upstream…
-
Compress any LLM 3x in one click, near-lossless, vLLM successor (ecosistemastartup.com via hn) vllm 3d
Un motor que promete comprimir tu LLM 3x sin GPU NeuraCryption publicó en noviembre de 2026 la documentación técnica de SPECTRA, un motor que comprime modelos de lenguaje alrededor de 3 veces —el documento sitúa esa franja entre 2,80x y 3,…
-
Show HN: Tensward – Local AI inference optimisation (github.com via hn) vllmllama 4d
Tensward Measures how your LLM is served on your own machine, offers one change at a time, and keeps a change only if it is faster and the answers pass your check. You already run vLLM or llama.cpp.
-
Show HN: Warmctl – Fast cold starts for vLLM inference (github.com via hn) vllm 5d
warmctl is GPU checkpoint/restore for vLLM. It snapshots a warmed-up engine and restores it in seconds on any host with the same GPUs and NVIDIA driver branch, so a new replica skips weight loading, torch.compile, kernel autotuning and CUD…
-
Serving open-weight models at scale with vLLM (ngrok.com via hn) vllm 6d
Serving open-weight models at scale with vLLM LLMs might be fantastic at prediction, but they weren’t designed to handle bajillions of requests efficiently. After all, they’re models, not servers.
-
Hours of YouTube and 85 handtyped pages of notes on LLM Inference – Start here (github.com via hn) vllm 7d
samir-vllm Teaching LLM inference from first principles by building a small engine in C++ and CUDA. The series follows what happens between a prompt and a generated token, from loading model weights to running each part of the transformer…
-
Show HN: Flide: Make Jev-like decision models up to 9.5x faster (github.com via hn) vllm 7d
-
Kinference's routing benchmark is a reminder to check cache reuse before changing models (www.reddit.com via reddit) vllmqwen 8d
Buoyant's Kinference sits in front of vLLM workers and routes requests using signals including cached prompt prefixes and queue size. In its own benchmark, median time to first token on shared-prefix traffic fell from 6.88s with plain Kube…
-
Show HN: Jacq, a DSL that compiles model and kernel descriptions to CUDA (github.com via hn) vllm 8d
I was working (with agents) on testing out various attention algorithm ideas I had, and I found that the existing infrastructure/libraries made it more difficult than it should be. So I created Jacq, a DSL and compiler.
-
Distributed Inference Benchmarks: [K]Inference vs. Plain Kubernetes (www.buoyant.io via hn) vllm 10d
Distributed Inference Benchmarks: Kinference vs. Plain Kubernetes Kinference (“kay-inference") is an inference control plane for Kubernetes and vLLM.
-
Kinference: Open-source inference control plane for vLLM (vllm.ai via hn) vllm 11d
Introducing Kinference: An Open Source Inference Control Plane for vLLM and Kubernetes We’re excited to announce Kinference [kay-inference], a new open source project for building scalable inference platforms with vLLM. We created Kinferen…
-
Show HN: GPUWorks – A 40 MB GPU cluster manager for vLLM/SGLang, no K8s needed (github.com via hn) vllm 11d
A 40 MB MaaS platform: one cluster across public and private clouds, on any mix of accelerators. English | 简体中文 GPUWorks is an ultra-lightweight GPU management and Model-as-a-Service (MaaS) platform.
-
vLLM Hook v1: A Plug-in for Programming Model Internals on vLLM (arxiv.org) vllm 12d
Modern artificial intelligence models are deployed on inference engines to optimize runtime efficiency and resource allocation, particularly for transformer-based large language models. The vLLM project is a major open-source library to su…
-
A team serving a 9B-20B open-weight language model on vLLM or SGLang must choose a quantization recipe, and needs two numbers: the speedup over BF16 and the accuracy given up. We assemble every public measurement of these numbers that we c…
-
Show HN: Deploy vLLM on RunPod with confirmation gates on every billable step (github.com via hn) vllmopenai 12d
RunPod vLLM Deployer A safe, tested Python CLI for deploying and tearing down OpenAI-compatible vLLM servers on RunPod GPUs. GPU Pods are billable resources.
-
Show HN: SDCGuard – Inspect vLLM KV-checksum coverage per request (github.com via hn) vllm 13d
sdcguard Release tests for LLM inference servers. An inference server can send back a wrong answer with a normal HTTP 200.
-
Show HN: 13M opinions on 48k products extracted from Reddit and HN (www.trustmebro.reviews via hn) vllmgemmamoe 2w
I made this a couple of months ago. Extractions were done with Gemma 4 26B MoE running on pairs of NVLinked 3090 cards with vLLM, in my basement.
-
Ask HN: What inference engine and KV offload engine do you use and why? (news.ycombinator.com) vllm 2w
I have the question to the guys in inference world, I am building some stuff connected to the inference in my work, and I started wondering, what do actually big guys use in their inference work? Do you prefer standard (and what is standar…
-
Attention Matching (AM) compacts a KV cache by keeping some keys per head, refitting their values and adding a bias to each kept key's logit. FlashAttention and vLLM's paged caches have no input for it.
-
Fixing Tool-Call Accuracy in vLLM (www.ubicloud.com via hn) vllm 3w
When I use an open-weight model, I see results that are nowhere close to what the quality benchmarks indicate. Tool call accuracy is an important part of serving open-weight models.
-
Bringing Flame Serving Close to Standalone vLLM (xflops.io via hn) vllm 3w
Bringing Flame Serving Close to Standalone vLLM An assistant answering several questions about one document repeatedly processes the same prefix. Keeping that prefix on one GPU can avoid rebuilding its attention keys and values.
-
Humming: Fast, Flexible Kernels for Quantized Inference (vllm.ai via hn) vllm 3w
Announcing Humming: Fast, Flexible Kernels for Quantized Inference Ant Group and the vLLM community are open-sourcing Humming, a general-purpose, high-performance kernel library for quantized inference. Humming offers the following feature…
-
vLLM TorchTPU: Zero-copy KV cache offloading and disaggregated serving (vllm.ai via hn) vllm 3w
Introducing vLLM TorchTPU Today we're releasing vLLM TorchTPU (vllm-torchtpu ), a new backend for running vLLM on Google Cloud TPUs. It's built on Google's TorchTPU stack and runs the model definitions that already live in vLLM, so most mo…
-
A new Speculative decoding drafter for vLLM and SGLang called EAGLE-4 (github.com via hn) vllm 3w
EAGLE-4 One Flexible Drafter for All Batch Sizes and Hardware Paper (EAGLE-4) · EAGLE-4 draft for Qwen3.8-27B · Documentation Figure 1(a) of the EAGLE-4 paper: Qwen3.8-27B on the SPEED-Bench qualitative split, 4× NVIDIA B300, one request,…
-
Cosmos3-Edge in under 8 GB: VLM out now, video gen and robot policy next. (www.reddit.com via reddit) vllm 3w
Hey guys, posted about optimized Cosmos Reason 2 model here a few months ago. Wanted to share a build of the latest Cosmos3 model, with even more optimizations!
-
Hi everyone! I'm the developer of llampart, an open-source Web UI for chatting with LLMs.
-
OvisOCR3: a 0.8B OCR model with structured output and cross-page table reconstruction (www.reddit.com via reddit) vllm 3w
OvisOCR3 is out, still based on Qwen3.5-0.8B and available under Apache 2.0, with vLLM inference examples. Compared with OvisOCR2’s Markdown parsing, it adds a few useful options: Structured JSON output with bounding boxes, element categor…
-
Push-based KV transfer to improve TTFT, TPOT for disaggregated inference in vLLM (aws.amazon.com via hn) vllm 3w
Artificial Intelligence Push-based KV transfer to improve TTFT and TPOT for disaggregated inference in vLLM vLLM is an open-source inference and serving engine for large language models (LLMs). LLM inference has two distinct phases: prefil…
-
PyTorch-native TPU back end for vLLM (github.com via hn) vllm 3w
| Documentation | Blog | User Forum | Developer Slack (#sig-tpu) | About vLLM TorchTPU is the PyTorch-native TPU backend for vLLM. It is an out-of-tree vLLM platform plugin, packaged as vllm-torchtpu, that runs vLLM's standard PyTorch mode…
-
vLLM Agentic API (vllm-project.github.io via hn) vllmcodexagentic+1 4w
Everything you need for agentic inference: run Codex and Claude Code on vLLM with conversation state, built-in tools, and multi-turn execution.
-
Hyper Optimizing Kimi K3 in vLLM (vllm.ai via hn) vllmoperator 4w
Hyper Optimizing Kimi K3 in vLLM: Moving Fast and Responsibly Coding agents can now help with some of the hardest parts of model optimization: writing complex GPU kernels, performing cross-operator fusions, overlapping computation with com…
-
Outlines – Structured Outputs for LLMs (github.com via hn) vllm 4w
🗒️ Structured outputs for LLMs 🗒️ Made with ❤👷️ by the team at .txt Trusted by NVIDIA, Cohere, HuggingFace, vLLM, etc. [![PyPI Version][pypi-version-badge]][pypi] [![Downloads][downloads-badge]][pypistats] [![Stars][stars-badge]][stars] [!…
-
The first disclosure from our Mythos harness (www.chainguard.dev via hn) vllmmythos 4w
The first disclosure from our Mythos harness - View all articles Chainguard Team Chainguard Team Our harness's first public finding is in vLLM: one client could force the output of another client's request. The bug had been sitting in vLLM…
-
Show HN: Modelship – an open source ray backed model cluster (github.com via hn) vllmopenai 5w
Hey all, just wanted to share how my spare time has been spent for quite some time now. Ok, this one started oddly.
-
2x 3090, 3x 3060 potential (smart and informed) future upgrade paths (send help) (www.reddit.com via reddit) vllm 5w
I'm trying to figure out the best long-term upgrade path for my setup. Right now my GPUs are spread across several consumer PCs: Main: 5900X, 32GB DDR4, 2× RTX 3090 24GB.
-
[vllm] Is it possible to run Qwen3.8-27B-FP8 on a 5090? (www.reddit.com via reddit) vllm 5w
Based on my experiments and feedback from claude, it seems extremely unlikely and even if I get it working, not practical because of minimum context. But just asking here as a last effort.
-
Java vLLM-like framework claim 90% perfomance of llamacpp inference on Nvidia HW (www.reddit.com via hn) vllm 5w
could not extract summary
-
Best open-weight coding model + harness for an on-prem multi-agent setup? (80-200+ GB VRAM, 2-3 concurrent users) Setup: GPU box with 1x H200-class card now, can allocate more (up to several H200s) if it clearly buys better results.
-
Native memory for local LLMs, (www.reddit.com via reddit) vllmqwen 5w
TL;DR: Native consumption of memory at the LLM level, no context. It's generally applicable to transformer-based models as well as Mamba and similar architectures.
-
I know you run LLMs. Will it boot? (mondegreens.github.io via hn) vllm 6w
Fourteen open models, from one RTX 4090 to eight H200s: Apron predicted their memory before renting the GPU, then booted each one on vLLM and measured.
-
Show HN: InferPilot – the vLLM doctor that tells you when fp8 won't help (github.com via hn) vllm 6w
InferPilot The vLLM doctor — know if a config change will actually help, before you ship it. Point InferPilot at a running vLLM: it reads the live Prometheus /metrics and tells you your real bottleneck — and whether a change like fp8 KV ca…
-
Hey HN, Anders and Tom here. We're building Magnitude, an inference engine for agents that optimizes itself to run as fast as possible on your hardware.
-
640tok/s on Gemma 4 with a megakernel (withhopper.com via hn) vllmgemmamoe 6w
A worklog on megakernels for Gemma 4 on TPU v5e: 640 tok/s per user on the 26B-A4B MoE model versus 173 on an H100 with vLLM, why the dense 31B model gains only 1.2x, and when a megakernel is worth building.
-
ContextMemory – Markdown memory for your llama.cpp/vLLM server (github.com via hn) vllmllamaopenai 6w
Try it (3 commands) · Engines · Docs · vs Mem0 / Zep / Letta Self-hosted memory gateway for your llama.cpp / vLLM server. Put one OpenAI-compatible /v1 URL in front of your engine.
-
Run Decision Models on vLLM and Red Hat AI Using DiffusionGemma (developers.redhat.com via hn) vllm 6w
Why Jev went viral, what "System One" models are good for, and how the vLLM community turned DiffusionGemma, already a Red Hat AI validated model, into an open, self-hostable decision engine. A new shape of model Over the last couple of we…
-
How to build a Jev-style classifier with DiffusionGemma and vLLM (medium.com via hn) vllm 7w
7 min read 5 days ago Press enter or click to view image in full size For years, diffusion was how AI generated images, not language. When open-weights text diffusion models like DiffusionGemma arrived, watching a paragraph materialize out…
-
Custom Models in Oh My Pi: vLLM, Llama.cpp, SGLang and More (doug.sh via hn) vllmllama 7w
Custom Models in Oh My Pi: vLLM, llama.cpp, SGLang and More Custom models let you run Oh My Pi (omp) on local models, or on models from providers it doesn’t support out of the box. omp reads them from ~/.omp/agent/models.yml .
-
What one vLLM replica on an L40 can carry and how it fails (percentes.ai via hn) vllm 7w
What one vLLM replica on an L40 can carry and how it fails The experiment Percentes runs takes one of two serving replicas away under sustained load and measures what the survivor does. Each replica is vLLM, an engine that loads a model an…
-
IronWarden – A microsecond PII firewall for streaming LLMs in safe Rust (github.com via hn) vllmollamaopenai+1 7w
🏰 IronWarden High-Performance Sovereign AI Reverse Proxy & Privacy Firewall IronWarden is a sovereign, ultra-low-latency AI security reverse proxy and PII firewall written in bare-metal Rust. Point any OpenAI, Anthropic, or Ollama/vLLM SDK…
-
Watermarking in vLLM (vllm.ai via hn) vllm 7w
Watermarking in vLLM What is text provenance, and why is it challenging? Establishing the provenance of digital content is essential to build trust and accountability in the information we share.
-
Hardware-Agnostic Models in vLLM (pytorch.org via hn) vllm 7w
could not extract summary
-
I built Kev, an Apache-2.0 open-source System One-style decision engine, using Claude Code during development. The project is free to try and can be run locally with your own model.
-
-
vLLM Architecture, Memory and Benchmarks Deep Dive (www.g-ftech.com via hn) vllm 7w
If you have ever stared at nvidia-smi during a production inference run and felt your heart sink seeing 12% GPU compute utilization while users complained about sluggish generation, you have run headfirst into the central reality of modern…
-
Self-hosted inference orchestrators compared: LocalAI, exo, GPUStack, vLLM (www.nexlab.net via hn) vllm 8w
A reference comparison of the self-hosted AI orchestrators in 2026: modalities, multi-machine support, auto-discovery, cache-aware routing, ops console, cloud burst, non-LLM fan-out, training, Kubernetes, platforms and signed images — with…
-
Keeping vLLM's Prefix Cache Warm Between Agent Turns (doug.sh via hn) vllmclaude-code 8w
Keeping vLLM's Prefix Cache Warm Between Agent Turns From 55% to 95% cached 🔗 I’ve been playing with a few different ways to host Qwen3.8 locally. I’m aiming for something that can replace Claude Code for most of my tasks.
-
Continual post-training of large multimodal models should add new capabilities while preserving those from pre-training, and the two goals pull in opposite directions. SFT gives explicit target supervision that learns a task from near-zero…
-
Show HN: Self-adjusting vLLM at production scale (twitter.com via hn) vllm 8w
𝗥𝘂𝗻𝗻𝗶𝗻𝗴 @vllm_project 𝗶𝘀 𝗲𝗮𝘀𝘆. 𝗞𝗲𝗲𝗽𝗶𝗻𝗴 𝗹𝗮𝘁𝗲𝗻𝗰𝘆 𝗽𝗿𝗲𝗱𝗶𝗰𝘁𝗮𝗯𝗹𝗲 𝘄𝗵𝗲𝗻 𝟯𝟬𝟬 𝗰𝘂𝘀𝘁𝗼𝗺𝗲𝗿𝘀 𝗮𝗿𝗿𝗶𝘃𝗲 𝗮𝘁 𝗼𝗻𝗰𝗲 𝗶𝘀 𝘁𝗵𝗲 𝗵𝗮𝗿𝗱 𝗽𝗮𝗿𝘁.
-
External cache transfers can succeed while a hybrid language model resumes from an inconsistent state. We examine the full 45-layer GLM-5.3-Flash model, using the RedHatAI/ GLM-5.3-Flash-NVFP4 quantized checkpoint with vLLM and LMCache und…
-
Prefix caching can reduce the time to first token (TTFT) of long-context LLM requests by reusing previously computed key-value (KV) states, but for short prefixes or fast GPUs, recomputation can be faster than loading from an external cach…
-
Neural Nova – GPU optimization benchmarks for LLM workloads (www.neural-nova.com via hn) vllm 9w
Performance Benchmarks Explore Optimized AI models recipes across GPUs, frameworks, and deployment configurations. BENCHMARKED MODELS REASONING vLLM · 8× NVIDIA H100-80GB Qwen3-235B-A22B +138.7% token/s +58% Cost Savings Qwen3-235B-A22B ·…
-
Public vs. Private LLM (wirt.ee via hn) vllmopenai 9w
LiteLLM in front of vLLM Context: the vLLM serving entry covers the engines. This is the front door — a LiteLLM proxy in front of them: one OpenAI-compatible endpoint, virtual keys per consumer, spend tracking, health-check routing between…
-
Serving LLMs on Tenstorrent Hardware: Inside the vLLM TT Plugin Today we are introducing vLLM TT Plugin, which brings Tenstorrent accelerators to vLLM through the standard out-of-tree platform plugin mechanism. Install it alongside vLLM an…
-
Speculative Decoding in vLLM on AMD GPUs (vllm.ai via hn) vllm 10w
Exploring Speculative Decoding in vLLM on AMD GPUs TL;DR: Speculative decoding allows vLLM to verify multiple drafted tokens in a single target-model pass. In our experiments, its effect on output-token throughput varied across drafting me…
-
Show HN: MaskShift – a maximalist coding agent with zero NPM dependencies (github.com via hn) tool-callingvllmollama+6 10w
MaskShift is a local-first coding agent harness. Features: - Works with models that have no tool-calling API.
-
trie stands for trace replay inference evaluation (github.com via hn) vllmopenai 10w
trie trie stands for trace replay inference evaluation and is a lightweight benchmarking harness that exercises OpenAI-compatible inference servers with synthetic workloads derived from production traces. It targets backends like vLLM, SGL…
-
Show HN: Run open-weight OCR, VLM and vision models behind one API (www.vlmrun.com via hn) vllmopenai 10w
Hey HN. We built an openai-compatible API for running open-weight VLMs, OCR VLMs and ViT-based vision models.
-
Show HN: Building AI agents client-side JavaScript (news.ycombinator.com) function-callingvllmollama 10w
Hey HN, most agent systems default to server-side Python inside containers and chain frameworks. I wanted to see how far we could push agent in the browser with vanilla JavaScript https://buttercup.sh The reason this is interesting is beca…
-
Hot reload vLLM and sglang configs (github.com via hn) vllm 10w
trimtab Change SGLang and vLLM scheduler settings while the server runs. No restart, weights never leave the GPU.
-
Long story short, about a year ago, in spite of everybody bashing gpt-oss for broken tool calling and refusals, I thought there's something there worth exploring. Model hit a sweet spot for me in that it was the first time I could run full…
-
Show HN: OSS, K8s-native AI platform for distributed multi-model inference (github.com via hn) vllm 10w
Hi everyone, I’m one of the co-founders of axem. We recently open sourced Shaide, a project we’ve been working on to make running multiple LLMs on your own infrastructure less painful.
-
Efficient Decode Context Parallelism with vLLM for Long Context Workloads 1. Introduction Long-context inference is becoming essential for agentic AI, where assistants may need to reason over large code repositories and long chat histories.
-
Tencent Hy4 Preview LLM (github.com via hn) vllm 11w
中文 | English 🖥️ Official Website | 💬 GitHub Table of Contents Model Introduction A New Flagship Generation Built for Productivity Benchmark Appendix Known Limitations News Model Links Quickstart Deployment vLLM SGLang Finetuning Quantizati…
-
I always run so many AI agents that I keep ending up with mystery processes, stray localhost ports, and no quick way to tell what started what. So I built Port Visualizer, a free open-source Windows app that shows which process owns each T…
-
Four reproducible vLLM parser failures that return 200 with the wrong tool call (ingot.tools via hn) vllm 11w
vLLM can drop or garble a tool call and still return 200 Four reproducible parser failures in vLLM 0.26.0, 0.27.1 and 0.28.0, run on CPU with no GPU or weights. As of 2026-08-27 one is fixed on main but not in any release; the rest are ope…
-
GLM-5.3-Flash (FP8) on 4 x RTX6000 Pro (www.reddit.com via reddit) vllmglm 11w
I've forked https://github.com/tonyd2wild/GLM-5.3-Flash-NVFP4-2x-DGX-Spark and make it run on sm120. I'm using it right now - got 1,4M context (5,45 sessions 262k each) 3,7kt/s PP and 160 - 230t/s TG (MTP enabled) You can make vllm Docker…
-
GLM-5.3-Flash @ DGX Station GB300: ~206 tok/s (single stream), 1M context (www.reddit.com via reddit) vllmglm 11w
Hey all! I'm finally doing some cool stuff with my "thinking heater" (h/t u/-TV-Stand-).
-
Two vLLM Recipes for Blackwell: More Context, More Concurrency, Less VRAM Waste (www.reddit.com via reddit) vllm 11w
Blackwell vLLM Recipes: Why NVFP4 KV Matters I’ve released two vLLM recipes for Blackwell GPUs: MTP-3 + NVFP4 KV DFlash2 + NVFP4 KV Both are built around the same idea: Blackwell can do native FP4 well, so the KV cache should benefit from…
-
Questions on optimism speed/intelligence on this rig (www.reddit.com via reddit) vllmdeepseekagentic 11w
Rig: 3945WX (12C, 2 CCDs, no AVX-512) · 8×32GB DDR4-3200 · 4× 5060 Ti 16GB · PCIe 4.0. Agentic workload (Hermes Agent).
-
Running local LLM's as agents in Claude Code (www.reddit.com via reddit) vllmllamamcp+2 11w
I hit my token limit three times a day on my max subscription - got sick of that and designed this MCP setup to shift some of the coding load to my local Qwen3.8-27B model. I've been iterating on it now for a bit, and thought I'd share it…
-
Qwen3.8 27B C8 at 972 TG / 5,680 PP on 4x MI100 rig ($6.5k) using my new INT8 vLLM fork (www.reddit.com via reddit) vllm 11w
Yet another vLLM fork thread here, but this time its for older INT8-centric hardware. This is a complete INT8 serving stack for Qwen3.8 27B based on vLLM, AITER, and a 27B GPTQ INT8 quant w/ DFlash2.
-
Qwen3.8 Garbage Outputs after a few hours of use (www.reddit.com via reddit) vllmqwenopenai 11w
Do I have something wrong in my settings? I keep having to restart vLLM to get Qwen3.8-27b-FP8 to stop outputting nonsense / garbage.
-
Fully quantized NVFP4 Qwen3.8-27B with QUASAR QAD (huggingface.co via reddit) vllm 11w
We're releasing a fully quantized NVFP4 version of Qwen3.8-27B. The checkpoint was trained using quantization-aware distillation (QAD) with QUASAR, our new QAT algorithm.
-
Anyone directly compare 3.8 27B at INT4 vs INT8? (www.reddit.com via reddit) vllm 11w
Man I love the quality of 27B int8 (currently running lued/Qwen3.8-27B-INT8-W8A16-MTP). But man, it is not fast and I am seriously skating a line with memory consumption on my A40 at 256K context (KV at FP8).
-
vLLM-iOS: 88% Faster Multi-Agent Inference on iOS (jonready.com via hn) vllm 11w
TL;DR: I improved multi-agent batching on iOS by implementing vLLM-style continuous batching in native Swift on MLX. I've written a lot about the tokenomics of local LLMs and agent swarms.
-
LLMPanel Deploy vLLM to RunPod or Vast.ai Without Kubernetes (llmpanel.io via hn) vllmopenai 11w
Deploy LLMs on any GPU, anywhere Pick a model, pick a GPU, hit deploy. LLMPanel provisions the container, exposes an OpenAI-compatible endpoint, and streams every GPU metric back to one dashboard — on your own hardware or any cloud.
-
Qwen 3.8 27b with tools and directed search on a non-coding professional suite (www.reddit.com via reddit) vllmqwen 11w
Some data showing the value of tools on the professional intelligence of Qwen 3.8 27b. My workload with Qwen focuses on real estate, private equity, and adjacent finance stuff.
-
Best 64k-128k models/fine-tunes on a single 3090 for PRD planning & ticket creation? (www.reddit.com via reddit) vllmgemmamoe+1 11w
Hey guys, I'm setting up a local workflow on a single 24GB RTX 3090 to handle project planning—specifically digesting massive (~128k context) requirements documents/PRDs and spitting out a ton of structured .md files to act like Jira ticke…
-
2xR9700 - Switching to gen5 x8/x8 from gen4 x16+x4 - I see about 18-32% more performance in vLLM (www.reddit.com via reddit) vllm 11w
I had read that I should not have mismatched PCIe configurations for tensor parallelism, so I upgraded my motherboard from MSI PRO-X670-P-WIFI to Asus ProArt X870E-Creator WIFI. With the old board I had gen4 x16 and gen4 x4.
-
Show HN: Free Inference Engineer and Model Training Roadmap (inferquest.org via hn) dpovllm 11w
Free, gamified roadmaps for LLM engineering: an Inference Engineering path (KV caches, CUDA kernels, production vLLM serving) and a Model Training path (pretraining on a budget, scaling laws, SFT/DPO/GRPO) — 183 tasks with auto-verified mi…
-
ReBar support for 20gb RTX 3080? (www.reddit.com via reddit) vllmllama 12w
Has anyone successfully patched the drivers of a modded RTX 3080 20gb to get Rebar support? I already updated my 3090.
-
Qwen 3.8 27B Aider score (www.reddit.com via reddit) aidervllmdeepseek+3 12w
I ran the Aider benchmark on Qwen 3.8 27B FP8 with FP8 KV cache 256K context vLLM. The score: 72.9 This matches Gemini 2.5 Pro from 2025-04-12 which also scored 72.9.
-
Is anybody using Deepseek v4F 0731 with a vision encoder and have had any success? (www.reddit.com via reddit) vllmdeepseekmcp 12w
Hey all, So I currently use 2x DGX Sparks with Deepseek v4 Flash 0731 and it works fantastic at 1M context, with 1.8M kv, dspark, vllm tp 2, etc. All good.
-
R9700 AI Pro TP=2 Qwen3.8-27B-FP8 low speed? Need Advice. (www.reddit.com via reddit) vllm 12w
Hi folks, with tp=2 I get the following logs out of vllm with official Qwen3.8-27b-FP8 with MTP3: [vllm] | (APIServer pid=1) INFO 08-24 05:47:40 [loggers.py:310] Engine 000: Avg prompt throughput: 198.5 tokens/s, Avg generation throughput:…
-
Qwen3.8 27B VLLM or SGLang Support (www.reddit.com via reddit) vllm 12w
Hi friends, I've just spent the last few hours to ultimately fail to run Qwen3.8 27B in a docker container. Would anyone happen to have a working docker file they could share the contents of?
-
New qwen3.8:27b on a 39k line C to single-file HTML / three.js port (www.reddit.comhttps) vllmopusclaude-code 12w
I was just curious how the new qwen3.8:27b does on a hard C to HTML porting job against Opus 5 in a default Claude Code. The job: my fun side project is a procedural shooter in a single C file.
-
Ling Tiny, King of Speed (www.reddit.comhttps) vllm 12w
Ling Tiny has now replaced Gemma4-12B in my rig as an auxiliary model doing hindsight operations. This is on a 4060Ti, which is a reasonable GPU available out there, and the speed is phenomenal.
-
vLLM + Deepseek harness or hermes? qwen3.8 (www.reddit.com via reddit) vllmllamadeepseek 12w
How do you guys set it up , i constantly get the error : I tried increasing the contex to 142k and putting the contex size as 115k in DSH , it still did not compress correctly. I have 0 issues if i run it with llama.ccp , it can work for 2…
-
**Setup:** 2× DGX Spark (GB10, 121 GiB unified each), TP=2 over 2×200GbE RoCE, vLLM 0.25.2.dev0, DeepSeek-V4-Flash-0731 FP8, `max_model_len=450000`, prefix caching on. KV pool = **1,686,693 tokens**.
-
Hello, So I've been trying lots of combinations in that never-ending landscape of options and settings. I wanted a proper quant of 3.8 27B running as fast as possible on my 5090 at 400W, with vision and with as much KV-cache as possible an…
-
I hosted Kimi K3 (2.8T parameters) using 8 B300s. 92 tok/s, $190 per million tokens (www.reddit.comhttps) vllmllama 12w
What I ran: 8x B300 on Modal, $56.79 per hour, vLLM, tensor parallel 8, native MXFP4 Cold boot ~27 min (1.56 TB load, JIT, 51 CUDA graph captures) TTFT 0.92 to 1.02 s, decode 92 tok/s steady, 83 tok/s average over 4 prompts $190 per millio…
-
i finally switched from windows to linux and got a 30-50% boost in speed. (www.reddit.com via reddit) vllm 12w
This is amazing. All I did was switch from llamacpp on windows to vllm on linux.
-
~3,200 tok/min on Two 3090s, No NVLink — Local Qwen Coding Agent at Full 262K Context (www.reddit.com via reddit) vllmqwen 12w
Qwen3.8-27B (W4A16-AutoRound) on vLLM 0.27.1, two RTX 3090s, TP=2, no NVLink. Agent: Pi, straight to the local endpoint.
-
Local LLM Inference: A 96GB Blackwell Field Guide (2026) (kitdev.ai via reddit) vllm 12w
Local LLM Inference: A 96GB Blackwell Field Guide (2026) Running local LLMs on 96GB Blackwell: which models fit, vLLM against SGLang, the throughput you actually get, and the places the setup quietly falls over. The verdict.
-
Question for folks with r9700 (www.reddit.com via reddit) vllmllamaagentic 12w
I have dual r9700 set up with Ubuntu. With llama.cpp I'm getting about 40 tokens per second for single instance.
-
This is the Qwen3.8-27B setup I actually use every day on one RTX 5090. I wanted to write it down with enough detail that another 5090 owner can reproduce it instead of guessing which memory knobs I used.
-
Good afternoon, everyone! I wanted to show the work I've been doing around porting Ninfer over to the CMP170HX (Github) So, first, I do want to call out the amazing work that Neroued, Sergiuszm and specifically Don-Chad have all done, to n…
-
Helm chart for Qwen 3.8 for B70 users (www.reddit.com via reddit) vllmqwen 12w
I took SergiioB's Intel Arc Pro B70 Inference Cookbook and made it into a Helm chart. It applies the patches onto the pinned vLLM version.
-
If you want to upgrade from Qwen3.6 27B, but dislike 3.8, give Ornith-1.5-35B-A3B a try. (www.reddit.com via reddit) vllmmoe 12w
Just make sure to leave MTP off. It's a decent upgrade from Qwen3.6 27B, and better than KAT Coder V2.5 Dev (which also requires their custom vLLM).
-
For those who bought the CMP170HX and unlocked the memory, I have some questions… are you using any specific llama.cpp or vLLM forks? What are you using for cooling the card?
-
I feel like I finally graduated. (www.reddit.com via reddit) vllmqwen 12w
I finally made the move from LM Studio to vLLM thanks to this post https://www.reddit.com/r/LocalLLaMA/s/NmS9CgHvqz. I may not know what it all means yet but I’m going to start diving into the docs to learn as much as I can.
-
New OllamaMQ v0.3.0 (news.ycombinator.com) vllmollama 12w
- loading / unloading models - ollama, lm-studio, vllm - optional security tokens and visibility and many more on - https://github.com/Chleba/ollamaMQ
-
Native vLLM and ROCm 7.15 for RX 6000 (RDNA2) on Windows 11 – 26 Tflops FP16 (github.com via hn) vllmopenai 12w
vllm-rocm-windows-RDNA2-oneclick []() []() []() []() []() Native vLLM + ROCm 7.15 (TheRock) for the whole AMD Radeon RDNA2 family on Windows — no WSL2, no NVIDIA, no compiler. One-click installer, everything prebuilt, and an OpenAI-compati…
-
GenOffice fork that works with any local LLM instead of a cloud account (github.com via hn) vllmollamallama+1 12w
GenOffice (local-LLM fork) A free, open-source AI Office suite — this fork drops the cloud-account requirement and talks to any OpenAI-compatible endpoint instead: a local server (Ollama, LM Studio, vLLM, llama.cpp server, text-generation-…
-
The rapid adoption of AI coding assistants and autonomous agentic development systems has coincided with major changes in the pace and structure of open-source software engineering. Yet empirical longitudinal evidence of these changes at t…
-
Show HN: Kvcachescope – Why Nvidia-smi is blind to vLLM KV cache leaks (github.com via hn) vllm 13w
kvcachescope A logical memory profiler and state inspector for PagedAttention inference engines (vLLM, SGLang). Standard GPU profilers (nvidia-smi, nsys, torch.cuda.memory_allocated()) observe physical VRAM allocations at the PyTorch tenso…
-
Show HN: NanoRL – RL training for LLMs in ~1,800 lines (github.com via hn) vllm 13w
The smallest async RL trainer I could write: one loop that runs REINFORCE on CartPole on a laptop and async GRPO on a cluster (e.g. 8xH100 trainer, 8 vLLM workers, ran as a [SkyPilot job group](https://docs.skypilot.ai/en/latest/examples/j…
-
Ask HN: Are there any production LLM pipeline setups to learn from? (news.ycombinator.com) vllm 13w
I believe vLLM has been the gold standard for inferencing but I'd like to learn how vLLM, langgraph, langfuse, other observability and business logic is meld into a single setup.
-
Qwen/Qwen3.8-2.4T-A95B (huggingface.co via hn) vllmqwen 13w
Qwen3.8-2.4T-A95B [!Note] This repository contains model weights and configuration files for the post-trained model in the Hugging Face Transformers format. These artifacts are compatible with vLLM, SGLang, TokenSpeed, etc.
-
Qwen 3.8-27B goes openweight in 2 days (huggingface.co via hn) vllmqwen 13w
Qwen3.8-27B [!Note] This repository contains model weights and configuration files for the post-trained model in the Hugging Face Transformers format. These artifacts are compatible with Hugging Face Transformers, vLLM, SGLang, TokenSpeed,…
-
Show HN: Proxima serves 4x more requests with no hardware change on vLLM (github.com via hn) vllm 13w
hey everyone, i decided to make a vLLM plugin that implements the Star-KV paper. the results are quiet promising with a decode kernel thats faster than FA2 in higher batch sizes.
-
Inferact vLLM creators are hiring (twitter.com via hn) vllm 13w
The @vllm_project maintainers at @inferact 🚀 are some of the most cracked engineers in the world. They’re building one of the inference engines that powers much of the world’s intelligence—and doing so with remarkable dedication, kindness,…
-
Meta is back with Muse Glimmer: local, agentic, multimodal, and open source! To celebrate, we are shipping with Meta day-0 support in transformers , llama.cpp , vLLM , Inference Endpoints, and other libraries.
-
vLLM Serving Experiments on H100s – config beats the baseline on p95 TTFT,ITL (efficientagent.substack.com via hn) vllm 14w
A Better Knob Beats More Silicon: 12 vLLM Serving Experiments on H100s Doubling my GPU count raised throughput 61% — and still lost to a one-line config change. A roofline-guided tour of what actually moves production serving metrics.
-
Inside vLLM: Anatomy of a High-Throughput LLM Inference System (2025) (www.aleksagordic.com via hn) vllm 14w
Inside vLLM: Anatomy of a High-Throughput LLM Inference System From paged attention, continuous batching, prefix caching, specdec, etc. to multi-GPU, multi-node dynamic serving at scale August 29, 2025 In this post, I'll gradually introduc…
-
LettuceDetect v2 in Semantic Router: Generative Hallucination Detection as a vLLM Endpoint Semantic Router can now verify grounded responses with a generative span detector served by vLLM. The new endpoint detector backend runs LettuceDete…
-
C++ Version of vLLM (github.com via hn) vllm 14w
Same tokens as vLLM. Same throughput.
-
DeepSeek-V4-Flash-0731-Latent-Reasoning. A model thinking in latent space (blog.n.ichol.ai via hn) vllmdeepseek 14w
DeepSeek-V4-Flash-0731-Latent-Reasoning. A self-contained model that does thinking in latent space, NVFP4-quantized, with a production vllm form for serving runtime.
-
Benchmark scores are reported as properties of a model, yet the inference framework used to produce them, such as HuggingFace, vLLM, or Ollama, are considered non-influential and their names and versions are almost never disclosed. In this…
-
The Inference Engine Guide for K3 Deployment (twitter.com via hn) vllmmoe 14w
Kimi K3 is here: a 2.8T-parameter MoE with a 1M-token context and native vision, running on vLLM from day 0. Here is our canonical deployment guide: architecture, kernels, recipes, and the flags to run it in production.
-
Lossless Inference (runinfra.ai via hn) vllm 14w
Quantization became the default way to speed up inference because it is the easiest one. In vLLM it is one flag or one swapped checkpoint (docs.vllm.ai), while doing the real optimization work like writing better kernels takes months.
-
Show HN: Gainz.fast – Local Inference, Faster (gainz.fast via hn) vllmllama 14w
Come help push the frontier of token speed across local models and hardware with your agents! Current frontier Laguna XS 2.1 · AMD R9700 (llama.cpp HIP) +31.14% 143.3 tok/s Laguna XS 2.1 · DGX Spark GB10 (vLLM NVFP4) +5.28% 37.3 tok/s Lagu…
-
vLLM for Baidu Kunlun (github.com) vllmglmmoe 15w
📖 Documentation | 🚀 Quick Start | 📦 Installation | 💬 Slack Latest News 🔥 [2026/07] 🚧 v0.25.1 under development — Added Qwen3.5 / Qwen3.5-MoE, Gemma4 (text and multimodal), GLM MoE DSA, and DFlash speculative decoding [2026/02] ⚡ Performanc…
-
Why self-hosted inference is essential (www.redhat.com via hn) vllmagentic 15w
Learn about Red Hat's approach to self-hosted inference with vLLM, addressing the reliability gap between open-weight models and hosted frontier models for agentic workloads.
-
Kimi K3 on vLLM: Up to 370 Tokens/sec (vllm.ai via hn) vllm 15w
Kimi K3 Is Here: Efficient Day-0 Support on vLLM We're thrilled to announce efficient day-0 vLLM support for Kimi K3, one of the most powerful open-weight models ever released. Last week, we previewed the production-scale integration work…
-
A production-grade OCR pipeline on Kubernetes with vLLM and Rust (github.com via hn) vllmglmqwen 16w
📄 Production-Grade SLM-Powered OCR Course 📄 Build a self-scaling, event-driven OCR pipeline on Kubernetes (AKS / GKE) with Qwen 3.5 + the GLM-OCR SDK Table of Contents Table of Contents Course Overview Who is this course for? Course Breakd…
-
AI Firewall – Security Gateway and Reverse Proxy for LLM Traffic (github.com via hn) vllmollamagemini+2 16w
🛡️ AI Firewall — Security Gateway & Reverse Proxy for LLM Traffic AI Firewall is a security gateway that sits between your applications and LLM APIs (OpenAI, Anthropic, Gemini, Ollama, vLLM). It inspects prompts before they leave your peri…
-
Show HN: Open-source verification and tuning layer for self-hosted LLMs (github.com via hn) vllm 17w
Hi HN, Thomas here, I'm a Machine Learning, and I've been deploying models in prod for the last 5 years and decided it was time to open-source some of my learnings! In Coloma you'll find my vLLM tuner, I use it to ensure models don't OOM a…
-
Veta: AI agent that QA-tests Android apps (github.com via hn) vllm 17w
# Veta Autonomous AI agent swarm for visual, functional, and accessibility testing of Android apps and mobile web — 100% of AI inference runs on AMD GPUs (Fireworks AI on AMD Instinct / self-hosted vLLM on ROCm), with containerized Android…
-
vLLM prefill paired with TileRT decode (vllm.ai via hn) vllm 17w
vLLM x TileRT: Specialized Decode for Latency-Critical Serving Disaggregated serving, which separates the compute-bound prefill phase from the memory-bandwidth-bound decode phase, has become an increasingly standard pattern for serving lar…
-
The multiplicative repetition penalty shipped across the LLM inference ecosystem (HuggingFace, vLLM, llama$.$cpp, and a dozen further engines) branches on the sign of each raw logit (divide positives by theta, multiply negatives). But the…
-
Show HN: MemStitch – Zero-copy context bridging for vLLM (25x TTFT speedup) (github.com via hn) vllm 17w
Context-Stitcher 🪡 Zero-Copy Context Bridging Gateway for Multi-Agent GPU Inference. 💡 Core Value Proposition In multi-agent collaborative workflows, separate agents often process the same long text context sequentially.
-
Large Language Models are reshaping how software is developed and maintained. They are typically deployed in production using inference engines such as vLLM, which can efficiently serve pre-trained, highly configurable models.
-
Native-speed vLLM transformers modeling backend (huggingface.co) vllm 18w
Native-speed vLLM transformers modeling backend # Upgrade the vllm pip package uv pip install --upgrade vllm --torch-backend auto The transformers library has become the reference modeling library for Machine Learning. It supports 450+ arc…
-
I rebuilt the Claude Code-style terminal workflow as a hackable multi-provider coding agent (www.reddit.com via reddit) vllmollamagrok+4 18w
Hello everyone, After the Claude Code leak started floating around, I spent time studying how the workflow was put together and rebuilt the core experience into my own project. I’m calling it Super Grokie, because it started as a joke but…
-
Hy3 (295B A21B), a reasoning and agent model with great cost efficiency (github.com via hn) vllm 18w
中文 | English 🖥️ Official Website | 💬 GitHub Table of Contents Model Introduction Stronger Agent Capabilities More Reliable Product Experiences Benchmark Appendix News Model Links Quickstart Deployment vLLM SGLang Finetuning Quantization Li…
-
AKM-CLR – pre-inference governance for vLLM-style LLM serving (github.com via hn) vllmopenai 19w
AKM-CLR — Tenant/Task Governance Layer for Shared LLM Serving AKM-CLR is a validated prototype governance/control layer for multi-tenant LLM infrastructure. It sits above OpenAI-compatible serving engines such as vLLM and helps decide whet…
-
Can you use Claude Desktop (code) but use local API? (www.reddit.com via reddit) vllmollamallama+1 19w
Hey all, So I tend to favor the Claude Desktop app in Code mode as the GUI does a great job of previewing code, MCP browser interactions/screenshot evals/etc. But I recall people saying they could get Claude Desktop to use a local API.
-
Automatic Prefix Caching – vLLM (docs.vllm.ai via hn) vllm 19w
Automatic Prefix Caching¶ Prefix caching kv-cache blocks is a popular optimization in LLM inference to avoid redundant prompt computations. The core idea is simple – we cache the kv-cache blocks of processed requests, and reuse these block…
-
While Large Multimodal Models excel in comprehension, high-throughput inference engines lack native support for multimodal generation. This is severe in Speech Language Models, where generating multi-layered audio tokens via decoupled AR+N…
-
Run a vLLM Server on HF Jobs in One Command (huggingface.co) vllm 20w
Run a vLLM Server on HF Jobs in One Command It's the quickest way to stand up a model for tests, evals, or batch generation. (If you're after a managed, production-ready service instead, that's what Inference Endpoints are for — more on wh…
-
Show HN: ParseHawk – 100% Local Document AI with API, CLI, and Web UI (github.com via hn) vllm 20w
I just released ParseHawk v0.1.0: Apache-2.0 licensed 100% local document AI platform that extracts JSON from PDFs, images etc. It builds on top of NuMind's NuExtract3 but additionally enforces a provided JSON schema with constrained decod…
-
I read it with a little bit of effort The tiny model result is insane, theoretically this could make make a 0.5b on-par with a 2/3/4b ish class model in coding with no weights change*. And for large models it could maybe fix let's say 30-5…
-
vLLM Recipes (recipes.vllm.ai via hn) vllm 20w
vLLM Recipes — Deploy any model on any hardware with vLLM Pick a model, adjust for your GPUs, copy the vllm serve line that runs. Community-maintained recipes for NVIDIA H100/H200/B200/B300, Grace-Blackwell, and AMD MI300X/MI325X/MI355X.
-
The rapid growth of large language model (LLM) inference services has increased the demand for efficient multi-tenant GPU scheduling. While modern inference runtimes such as vLLM improve throughput through continuous batching and optimized…
-
How to become an AI infrastructure engineer? (news.ycombinator.com) vllm 21w
Hi, I currently work on a GenAI platform for one of the largest local industrial companies. My daily work mostly involves building inference infrastructure on top of a 48x H200 GPU, Kubernetes and vLLM.
-
petite-vllm Part 2: KV Cache & Paged Attention (kristenmcintosh.dev via hn) vllm 21w
petite-vllm Part 2: KV Cache & Paged Attention Building an LLM Serving Engine from Scratch · 12 min readKV Cache In part 1 we implemented a simple autoregressive loop and LLM interface. There was no KV caching, which means that at each tok…
-
Profile(v2.1.4) physics-aware optimizer for vLLM (31→470 tok/s on A100) (github.com via hn) vllm 21w
Profile A physics-grounded, cost-aware optimization loop for vLLM inference servers. The Problem: vLLM exposes numerous configuration flags (e.g., --max-num-seqs, --enable-prefix-caching).
-
Retrieval-Augmented Generation (RAG) improves factual grounding, but it also lengthens prompts and raises prefill cost. Prefix caching in serving engines such as vLLM reduces this cost only when requests share the same token prefix.
-
Running GLM-5.2 5x faster at 500tps with limitation (abhishek.it via hn) vllmglm 21w
Running GLM-5.2 5× faster than vLLM, on a runtime that doesn't support it I rented an 8×B200 and tried to run GLM-5.2 on TileRT, the runtime MiMo used to push a 1T model past 1000 tok/s. TileRT doesn't support GLM-5.2, so I reverse-enginee…
-
IndexCache: Accelerating Sparse Attention via Cross-Layer Index Reuse (github.com via hn) vllmdeepseek 21w
IndexCache Accelerating Sparse Attention via Cross-Layer Index Reuse Tsinghua University & Z.ai This repository provides a patch for SGLang and vLLM that enables IndexCache inference acceleration for models using DeepSeek Sparse Attention…
-
Sors: a Rust proxy that reorders prompts to maximize vLLM prefix cache hits (github.com via hn) vllmopenai 21w
Sors - reorders prompts for LLM to maximize prefix cache hit. A minimal reverse proxy that reorders prompt content to maximize prefix cache hits in LLM inference engines (vLLM, SGLang, or any OpenAI-compatible backend with prefix caching e…
-
67% Cost Savings with PD Disaggregation Using Ray and vLLM on AMD MI325X (www.anyscale.com via hn) vllm 21w
Achieving Up to 67% Cost Savings with Prefill-Decode Disaggregation Using Ray + vLLM on AMD MI325X In LLM serving, the optimization objective is deceptively simple: given a set of latency SLA targets – time to first token (TTFT), time per…
-
Monitoring LLM Inference with Prometheus and Grafana (vLLM, TGI, Llama.cpp) (www.glukhov.org via hn) vllmllama 22w
Monitor LLM Inference in Production (2026): Prometheus & Grafana for vLLM, TGI, llama.cpp Monitor LLM with Prometheus and Grafana LLM inference looks like “just another API” — until latency spikes, queues back up, and your GPUs sit at 95%…
-
DifussionGemma 4 on 4x7900xtx (www.reddit.com via reddit) vllm 22w
Just got 100 tps on generation, but in total time it around 45-60 t/s in case of prompt processing waiting. Available memory show: GPU KV cache size: 152,671 tokens Maximum concurrency for 131,072 tokens per request: 1.16x amd-smi monitor…
-
Built a minimalist coding agent optimized for memory footprint and speed (www.reddit.com via reddit) mistralvllmollama 22w
Hi everybody, I spent the last two weeks building zerostack, a coding agent in Rust, focused on memory footprint, shipping with ollama and vLLM integrations. I managed to get it to run at ~16MB (with peaks of 24MB) of RAM usage, and no CPU…
-
I'm trying to use Gemma 4 12B — the new encoder-free unified model (audio/vision/text in one) — for a one-pass audio → response voice assistant: feed the recorded WAV + system prompt and get the reply back as text directly, collapsing the…
-
-
Releasing Cohere North Mini Code (www.reddit.comhttps) vllm 22w
-
I have 4x 128 GB VRAM now , what should i do. (www.reddit.com via reddit) vllmqwen 22w
-
Fixing single missing quote errors. (www.reddit.com via reddit) vllm 22w
Loving the local AI world and been building out my own the last few weeks. One pesky recurring problem is what seems to be related to how the model produces JSON for its responses.
-
RDNA4 Specific Docker Image vLLM (www.reddit.com via reddit) vllm 22w
You bought RDNA4 with the promise of go-fast, and it doesn't deliver in vLLM. I know the feeling, out of the box vllm is a complete dog on RDNA4...
-
Here are some tips on hitting nearly 200 tok/s for DeepSeek v4 Flash on Hopper (dnhkng.github.io via reddit) vllmdeepseek 22w
I needed a smarter model for my local Hermes Agent setup, so I moved to DeepSeek v4 Flash. First things first: Running 4 concurrent threads on vLLM, I can hit ~400 tok/s 400 x 60 x 60 x 24 x 30 is ~1B TOKENS per month!!!
-
5070 Ti + 5060 Ti on vLLM hangs on GDN with Qwen3.6 (www.reddit.com via reddit) vllm 22w
-
[2x3090]: SymmMemCommunicator: Device capability 8.6 not supported, communicator is not available. (www.reddit.com via reddit) vllm 22w
Hi all, this is a mere "see what others are doing post" rather than a solution to a problem. As newbie, I put together a 2x3090 box that I run vllm on.
-
OpenEnv is a tool for creating an agentic execution environment like terminals, browsers, or anything an agent can interact with. And today, we’re excited to announce that OpenEnv is becoming even more open, to make the future of training…
-
vllm-doctor — a CLI tool to diagnose and monitor vLLM inference servers (www.reddit.com via reddit) vllm 23w
vllm-doctor reads metrics from a vLLM server's /metrics endpoint or a Prometheus instance and runs rule-based checks to find what is wrong. It detects queue pressure, high TTFT/TPOT, KV cache pressure, and other rules across pods.
-
Breaking the Ice: Analyzing Cold Start Latency in vLLM (arxiv.org) discussed ↗ vllm 23w
-
how to run gemma-4-12b-it-qat-w4a16-ct in vllm or any version quantized of the model (www.reddit.com via reddit) vllmgemma 23w
when running by using transformers it runs by using vllm some weird error come up plese can any body share the command of running it on vllm ?
-
club-3090 adds experimental FP8 support for Qwen3.6-27B! (www.reddit.com via reddit) vllmqwen 23w
It’s finally here! Something many of us running dual RTX 3090 rigs have been anticipating.
-
dvlt.cu: inference engine written from scratch in CUDA/C++ for NVIDIA's DVLT 3D transformer model (www.reddit.comhttps) vllmllama 23w
Im into both HPC and 3D reconstruction, so I built this as a side project. dvlt.cu is a single 5MB binary: - No python, torch, TF, ONNX, llama.cpp, vLLM, or huggingface runtime - Nearly no dependencies: only cuBLASLt (shipped with libcuda…
-
Activating MTP for QATGemma4 31b q4_0? (www.reddit.com via reddit) vllm 23w
Has anyone figured out how to activate MTP for Gemma4’s new QAT q4_0 GGUF for 31b? Or is this still not supported in llamacpp?
-
Serving TTS/cloning models on llama.cpp? (www.reddit.com via reddit) vllmllama 23w
Are there any quality voice cloning and speech generation models that already have support in Llama.cpp or, more likely, vLLM-Omni? It would be nice to swap them out like any other inference model and use a common API, rather making a sepa…
-
Nvidia DGX Spark GB10 – AI Models and Guide with vLLM and Autonomous Script (github.com via hn) vllm 23w
NVIDIA DGX Spark GB10 — AI Models & Inference Guide Welcome to my repository and guide for running, optimizing, and benchmarking state-of-the-art AI models on the NVIDIA DGX Spark deskside supercomputer, powered by the cutting-edge NVIDIA…
-
vLLM: An Efficient Inference Engine for Large Language Models [pdf] (www2.eecs.berkeley.edu via hn) vllm 23w
vLLM: An Efficient Inference Engine for Large Language Models by Woosuk Kwon A dissertation submitted in partial satisfaction of the requirements for the degree of Doctor of Philosophy in Computer Science in the Graduate Division of the Un…
-
Show HN: LLMhop – A tiny, stateless router for LLMs with a NixOS module (github.com via hn) vllmllama 23w
LLMhop is a tiny stateless proxy for LLM inference servers. It tackles an issue I faced when trying to serve more than one local LLM at once which is not natively supported by vLLM.
-
Fast & Efficient LLM Inference with vLLM: A New Course with DeepLearning.AI We're excited to announce, with Red Hat and Andrew Ng's DeepLearning.AI, a hands-on course that walks through LLM fundamentals and the full optimize, deploy, and b…
-
KVarN: Native vLLM KV-cache quantization back end by Huawei (github.com via hn) vllmagentic 23w
⚡️ Built for agentic and long-context workloads. 💡 KVarN delivers 3-5x more KV-cache capacity and up to ~1.3x the throughput of FP16, so you fit far longer contexts and serve more concurrent requests, with FP16-level accuracy.
-
Free vLLM Course: Inference, Compression, Benchmarks (www.deeplearning.ai via hn) vllm 23w
Apply quantization to shrink a model's memory footprint, then measure the accuracy tradeoff. Fast & Efficient LLM Inference with vLLM Instructor: Cedric Clyburn Earn an accomplishment with PRO - Intermediate - 1h38m - 9 Video Lessons - 3 C…
-
Show HN: Tiny-vLLM – high performance LLM inference engine in C++ and CUDA (github.com via hn) vllm 24w
tiny-vllm You're going to build a high performance LLM inference engine with C++ and CUDA - tiny-vllm, a younger and smaller sibling of vLLM We will learn a lot along the way, make mistakes and derive the ideas and maths from scratch This…
-
LiquidAI/LFM2.5-8B-A1B · Hugging Face (huggingface.co via reddit) vllmmoellama+1 24w
looks like you can run it on any potato (A1B)! https://huggingface.co/LiquidAI/LFM2.5-8B-A1B-GGUF from LiquidAI: LFM2.5 is a new family of hybrid models designed for on-device deployment.
-
EDIT - IGNORE. I MADE A MISTAKE.
-
I have been coming to this subreddit to understand what the optimal config is to run a model on a given hardware setup. I referred to specific benchmarks, but they are too generic and do not consider the underlying hardware.
-
Local run for multi users: which software set? (www.reddit.com) vllmllama 24w
Context: I am testing and running local LLM on Linux for some months, first with llama.cpp and now with vLLM for better concurrent capabilities. I use llama-swap in front of either vLLM or llama.cpp in order to have thinking and non-thinki…
-
Used the vllm version of https://github.com/noonghunna/club-3090 It worked fine for myabe 20 40k context, havent tried the new one. Anyone used the new llama.cpp patched one for single 3090?
-
Okay fun time I got access to two Nvlinked A100s for some research project I benchmarked my work against the Gemma 4 31b-it available through Google, but my dataset is rather massive, so I need to run it on the "local" resources. Basically…
-
Looking for a working Deepseek-v4-Flash quant (www.reddit.com) vllmllamadeepseek 24w
Best I tried so far is https://huggingface.co/nsparks/DeepSeek-V4-Flash-FP4-FP8-GGUF with the custom llama.cpp fork, but it suffers from low quality and random incoherent output. VLLM wouldn't support anything other than H100s for DS4.
-
I was given the great opportunity to borrow a H100 with 94GB VRAM at work until it is needed by a customer. (No idea how much system ram I will get, but I guess they are a bit flexible on this).
-
Looking for Suggestions — Single 5090 & 64gb DDR5 (www.reddit.com) vllmllamaqwen+1 24w
Hi Reddit, I am planning on running Qwen 3.6 27b NVFP4 via vLLM on my 5090 but was wondering if something like 35b a3b at Q8 on Llama would produce better results for agentic coding and utilize the system memory. My research says no but if…
-
Show HN: Harbor v0.4.19 – harbor launch –back end vLLM –web codex (github.com via hn) vllmollamarag+1 24w
https://github.com/user-attachments/assets/e4897391-c5a8-4391-93c3-9f8b76155f11 Setup your local LLM stack effortlessly. Starts fully configured Open WebUI and Ollama harbor up Now, Open WebUI can do Web RAG and TTS/STT harbor up searxng s…
-
Harbor v0.4.19 - vllm/sglang/llama.cpp launch codex/claude/pi/opencode (www.reddit.com) vllmllamacodex+1 24w
I'm usually not posting about Harbor releases out of the respect for the community here, but I think v0.4.19 might save a lot of people some time. Harbor can now launch your local agentic coding tools with local inference backends.
-
Posted on other feeds last week and figured some of you out here might be interested as well; Someone commented asking if it supported OpenAI-compatible endpoints (LM Studio, vLLM, OpenRouter, Together, Groq, LocalAI…), so i have spent few…
-
EAGLE 3.1: Advancing Speculative Decoding Through Collaboration Between the EAGLE Team, vLLM, and TorchSpec The EAGLE series — including EAGLE 1, EAGLE 2, and EAGLE 3 — has become one of the most widely adopted and practically deployed fam…
-
Output Length Constrained Summarization using GRPO on tiny LLMs | smolcluster (www.reddit.com) vllmqwen 24w
Just released a blog on a side research project I have been doing for the past two months and would love for you all to check out and see how it is! It's about output length-constrained summarization using LLMs with GRPO.
-
Sharing INT4-W4A16 version of Jackrong/Qwopus3.6-27B-v2 for VLLM/SGLang users (www.reddit.com) vllm 24w
link: https://huggingface.co/JC1DA/Qwopus3.6-27B-v2-INT4-W4A16-Autoround Super surprised how good Jackrong's model is... It's taking so much time to evaluate the all the base qwen3.6-27B, Jackrong's version and other's quantized models but…
-
Best coding model on RTX 3060 (www.reddit.com) vllmllama 24w
Wondering what’s the best coding model that can fit on a RTX 3060 (12GB). Has anyone been able to do something useful with it?
-
numind/NuExtract3 · Hugging Face (huggingface.co via reddit) vllmrag 25w
NuExtract3 is a unified 4B vision-language reasoning model for document understanding. It combines strong structured information extraction with high-quality image-to-Markdown conversion, making it suitable for extraction pipelines, OCR, a…
-
Qwen 3.6 benchmarks on 2x RTX PRO 6000 (www.reddit.com) vllmqwen 25w
Got a chance to play around with 2x RTX PRO 6000 setup so sharing some number for Qwen 3.6. All these were run using latest stable VLLM backend.
-
What workstation to get for ~13k EUR? (www.reddit.com) vllmminimaxllama+2 25w
My use-cases will be to test open-weight LLMs and work on harnesses, inference systems and possibly other non-ML workflows (CS-related) in the future. Fine-tuning would not be something I do locally because I can rent a B200 from RunPod fo…
-
Run Chrome’s tiny Gemma4 (aka Gemini Nano) directly on PC without GPU (www.reddit.com) vllmgemmallama+1 25w
Everyone remembers that sneaky download of Gemini Nano earlier this month? and if you talk to it, it will happily tell you it’s a Gemma.
-
$16 refactor, 400 steps, 95% routed to open MoE (www.reddit.com) vllmmoedeepseek+1 25w
Got tired of $160 Opus bills so I spent a weekend wiring up a routing layer on vLLM 0.8 (2xA100, enable_auto_tool_choice). Getting the tool call parser to cooperate took longer than the actual routing logic.
-
$340 opus bill made me rethink how I route agent tool calls (www.reddit.com) vllmhaikudeepseek+1 25w
Looked at my coding agent's bill last month: $340 for repo maintenance across three repos, each around 15k lines. Most of those tool calls were just grep and file reads.
-
LLMKube – A Kubernetes operator for local LLMs across Nvidia and Mac fleets (llmkube.com via hn) vllmoperatorllama 25w
Run production LLMs on your own hardware A Kubernetes operator for self-hosted LLM inference. vLLM, llama.cpp, TGI, NVIDIA, Apple Silicon.
-
Cannot get NCCL test to run in docker with 2 x 6000 Pro connected x8 to AM4 CPU (www.reddit.com) vllm 25w
nvidia-smi topo -m is showing the both GPU as PHB (i.e. via CPU) connected as expected but I cannot get NCCL all_reduce_perf to run at all, it always hangs after starting up.
-
Edit: does this happen every time a newbie tries to post here. Getting roasted despite having valid results?
-
I work on AxonFlow, a source-available (BSL 1.1) runtime for long-running agent workflows. We’ve been running it in front of Ollama-served models and OpenAI-compatible local endpoints (llama.cpp `--server`, vLLM, LM Studio).
-
This has turned out to be useful to many of my friends so I thought I'd share here as well. I created a tool and documentation page for most major open-souce project's adherence to 'OpenAI compatibility' after seeing inconsistencies betwee…
-
I’m running Hermes Agent on a single NVIDIA DGX Spark using vLLM with: docker run --gpus all \ --name qwen36-aggressive \ --restart unless-stopped \ -p 8000:8000 \ --ipc=host \ --ulimit memlock=-1 \ --ulimit stack=67108864 \ --shm-size=32g…
-
Hello guys, two days ago i ran the spark-arena for my Qwen 3.5 122B Recipe on a single DGX Spark and I got the highest score on speed for any context length and concurrency across all 3.5 122B Int4 Recipes. Just wanted to share if somebody…
-
Do smaller quants silently break tool calls / JSON output? (www.reddit.com) vllmollamallama 25w
I posted recently about EvalShift, an OSS CLI for regression-testing LLM model changes. A few people pointed out that for LocalLLaMA, the more interesting use case may be quantization regression: Q8 -> Q4_K_M Same base model, same prompts,…
-
Greetings from former TurboQuant's biggest defender, now middle-sized niche-aware TurboQuant defender. Today I'm presenting to you the results of me thoroughly exploring the world of PPL and KLD benchmarks with my single RTX 3090 using Bee…
-
I posted earlier about RTX 5060 Ti local LLM testing, and I have cleaned the repo up quite a bit since then. The project is now a more structured benchmark/recipe repo rather than scattered notes.
-
Need help getting 7900 XTX PyTorch performance metrics (www.reddit.com) vllmllama 25w
I'm on a quest to profile and benchmark different GPUs for PyTorch, vLLM, and llama.cpp. Cannot find the high-end AMD consumer cards for rent anywhere online and interested in the PyTorch ROCm performance of the 7900 XTX (if you want to co…
-
Built this for myself after wanting to use local LLMs during work calls without the window showing up on screen share. Every existing tool was either cloud-only or a 200MB Electron app.
-
TL;DR best setup I tested on a RTX 3090 24 GB: ik_llama.cpp + Qwen3.6-27B-MTP-IQ4_KS.gguf 156k context, q8_0/q8_0 KV, MTP, vision on CPU benchmark result on a ~5.9k prompt + 1k output: about 1261 tok/s prefill, 72.9 tok/s decode llama.cpp…
-
Benchmarking vLLM vs SGLang vs llama.cpp on a mixed Blackwell/Ada cluster (www.reddit.com) vllmllama 26w
I have been running some benchmarks on a heterogeneous 7-GPU cluster to see how different inference engines handle long context prefill using pipeline parallelism. My setup consists of a mix of Blackwell and Ada cards: one RTX PRO 6000 96G…
-
While waiting for Fara-1.5 for my coding harness (www.reddit.com) vllmllama 26w
Hi all, Not sure many people are aware so wanted to give a word about Fara-1.5 release. => this release will likely be the big sister of Fara-7B and built on top of Qwen3.5 Actual Fara-7B performs not bad at all but actually requires a pro…
-
5060ti chads -> gemma-4-31b-it-nvfp4 + vllm + mtp (www.reddit.com) vllmgemmaqwen 26w
Hey all, While nvfp4 still seems to be a work in progress, the latest version of vllm 0.21 finally has mtp working for gemma. With all the talk of qwen being badass I thought I would revisit gemma.
-
KV Cache Is Becoming the Memory Hierarchy of Inference (touchdown-labs.com via hn) vllm 26w
A briefing on the inference memory hierarchy: prompt layout, host-side shared KV, distributed lookup, RDMA transfer, encoder reuse, and evidence discipline. Covers vLLM × Mooncake, LMCache MP, LMCache CacheBlend, SGLang, NVIDIA Dynamo, and…
-
Looking to migrate off of Ollama and LMStudio (www.reddit.com) vllmollamagemma+3 26w
Hello, I'm currently using Ollama / lm studio for things like code inference and proof reading emails, etc. Definitely not experienced in this space but looking to grow.
-
I kept seeing inference-speed claims for these models and wanting an apples-to-apples comparison on the hardware I actually have. So I built a harness and a public page that dumps every run as YAML.
-
Using Intel Arc Pro series, any thoughts ? (www.reddit.com) vllmllama 26w
Simple question: Has anyone run two or more of either of these on Ubuntu ? Intel Arc Pro B70 (32 GB) Intel Arc Pro B65 (32 GB) Running llama or vLLM etc., Any thoughts
-
club-5060ti: practical RTX 5060 Ti local LLM notes and configs (github.com via reddit) vllmllamaopenai 26w
I put together a small public repo for RTX 5060 Ti 16GB local LLM setups: I took inspiration from the club-3090 repo, but this one is focused on documenting what we’ve actually tested on 5060 Ti hardware so the setup details are easier to…
-
Ok, hear me out. This all started when I was trying to understand why this Qwen3.6 27B INT8 Autoround (https://huggingface.co/Minachist/Qwen3.6-27B-INT8-AutoRound/tree/main) recipe was performing so much better than any other Qwen3.6 27B q…
-
My own local first ai harness (www.reddit.com) vllmollamallama 26w
Hi, i just wanted to share what im playing with for last couple weaks. I built my own AI harness: TinyHarness My main goal was low memory footprint, it is not written in Typescript/Javascript/Python, leaving as much memory as possible for…
-
Shipped this for the AMD x lablab hackathon. Attached video is one of the actual reels the pipeline produced - one English sentence in, finished mp4 with characters, story, music, and voice-over out (fast demo video, not the best quality).
-
MI50s Qwen 3.6 27B @52.8 tps TG @1569 tps PP (no MTP, no Quant) (www.reddit.com) vllmqwenagentic+1 26w
TL;DR Results from the title are for single inference with 2 prompt of 1k and 15k tokens. So no MTP (as it’s slower for big prompt), no DFlash (working too but slower for big prompt), no quant used (full precision wanted) and the results a…
-
qwen3.6 just stops (www.reddit.com) vllmqwenopenai 26w
https://preview.redd.it/74cj1xu9pw0h1.png?width=1229&format=png&auto=webp&s=3ae999cc3530ecb4eccf70e25f1a9eb2aa3f2d7b Sometimes qwen 3.6 just stops at the middle of a task, is there a way to avoid it? This is qwen-code CLI, but also happens…
-
So, as most of us here are, I'm a llama.cpp loyalist. Easy to understand, great configuration, relatively stable, etc.
-
EDIT: Edited to provide more clarity It occurred to me, that perhaps the same draft model used for speculative decoding would be completely adequate if we just used it's output as-is for reasoning, without validating the results against th…
-
RTX 5060Ti 16GB or RTX 3080 20GB? (www.reddit.com) vllmgemmacopilot+2 26w
I would like to dedicate a budget of about 500 euros to upgrade my workstation and run inference on the qwen 3.6 27b and gemma 4 31b models. I currently have an RTX 5060Ti 16GB.
-
New Qwen3.6 27b Autoround Quant (int4) Best Recipe (www.reddit.com) vllm 26w
I've been using the int4 Autoround quant from "Lorbus/Qwen3.6-27B-int4-AutoRound" and it has been pretty good! Great quality and performance on an RTX 5090 vllm.
-
Gemma 4 MTP vs DFlash on 1x H100: dense vs MoE results (www.reddit.com) vllmgemmamoe 26w
Benchmarked Gemma 4 MTP and z-lab's DFlash on a single H100 80GB using vLLM and NVIDIA's SPEED-Bench qualitative dataset. Setup: Hardware: 1x H100 80GB Runtime: vLLM Dataset: SPEED-Bench qualitative Prompts: 880 total, 80 prompts across ea…
-
Is this a crazy idea? (www.reddit.com) vllm 26w
I’m running locally with 2 RTX 3099s and 128gb of RAM I run my workflows with Hermes/OWUI and use Comfy for media generation. My inference is with LM Studio.
-
TensorRT-LLM vs vLLM vs llama.cpp on NVIDIA DGX Spark? (www.reddit.com) vllmllama 26w
I am looking for recommendations on the best way to run local LLMs on NVIDIA DGX Spark. Which stack makes the most sense in practice: TensorRT-LLM, vLLM, or llama.cpp?
-
TL;DR: DeepSeek-V4-Flash running at 85.52 tok/s @ 524k ctx and ~111 tok/s @ 128k single-stream on 2× RTX PRO 6000 Max-Q pasta-paul's DeepSeek-V4-Flash-W4A16-FP8 quant is great, but its MTP head silently gets stripped at load time (HF trans…
-
vLLM + NVFP4 + Qwen3.6 27B: "Checkpoint does not provide a q scaling factor"? (www.reddit.com) vllmqwen 27w
I have been trying various NVFP4 based variations of Qwen 3.6 27B, and I am seeing this for the ones that look most interesting to run on my 2x 16GB VRAM with KV cache fp8. vllm | (Worker_TP0 pid=136) WARNING 05-09 13:49:27 [kv_cache.py:10…
-
vLLM ROCm has been added to Lemonade as an experimental backend (www.reddit.com) vllmllama 27w
vLLM has the ability to run .safetensors LLMs before they are converted to GGUF and represents a new engine to explore. I personally had never tried it out until u/krishna2910-amd/ u/mikkoph and u/sa1sr1 made it as easy as running llama.cp…
-
z-lab released gemma-4-26B-A4B-it-DFlash. Anybody tried it yet? (huggingface.co via reddit) vllmgemmaqwen 27w
Past few days, its all been about MTPs. Somehow people missed out the fact that Z lab released the Dflash for Gemma4 26B a couple of days ago.
-
Gemma 4 26B Hits 600 Tok/s on One RTX 5090 (www.reddit.com) vllmgemma 27w
I ran a benchmark to see how much DFlash speculative decoding actually helps in vLLM. Setup: GPU: RTX 5090, 32GB VRAM vLLM: 0.19.2rc1 Main model: cyankiwi/gemma-4-26B-A4B-it-AWQ-4bit Draft model: z-lab/gemma-4-26B-A4B-it-DFlash Workload: r…
-
Benchmark Qwen 3.6 27B MTP on 2x3090 NVLINK (www.reddit.com) vllmqwen 27w
TL;DR On 4× RTX 3090 with NVLink bonded between GPU pairs (0↔2 and 1↔3), pinning TP=2 to a NVLinked pair gave +25% throughput at concurrency 1 and +53% at concurrency 4 vs running TP=2 over PCIe. Adding the other two GPUs to make it TP=4 m…
-
I've been running a persistent local agent for about 2 months - hundreds of sessions, mix of local models (llama.cpp/vLLM/lmstudio) and paid (Claude). One of the things that has been driving me nuts with OpenClaw and Hermes is the way memo…
-
Some of you saw our post a couple weeks back about hitting 102 tok/s stable on Qwen3.5-35B on a DGX Spark. A lot of you asked "cool, where's the code?" Today's the day: Github Atlas is open source.
-
Exaggerated PCI-E bandwidth concerns? (www.reddit.com) vllm 27w
I frequently see (both here and on r/LocalLLM ) comments that multi-gpu setups are complex, problematic and typically bottlenecked by PCI-E bandwidth on consumer motherboards. I am running 2x RTX 5060 TI 16gb ( and about to add a third ),…
-
Show HN: Granite Switch - compose multiple LoRA adapters to one deployable model (github.com via hn) vllm 27w
Granite Switch is an open-source IBM Research project for composing several task-specific LoRA adapters into a single deployable Granite model checkpoint. The idea is to get the accuracy benefits of multiple fine-tuned models without havin…
-
vLLM V0 to V1: Correctness Before Corrections in RL (huggingface.co) vllm 27w
vLLM V0 to V1: Correctness Before Corrections in RL TL;DR. vLLM V1 matched our vLLM V0 reference after we fixed four things: processed rollout logprobs, V1-specific runtime defaults, the inflight weight-update path, and the fp32 lm_head us…
-
Help with GPT-OSS-120B on vLLM (www.reddit.com) vllm 27w
Hiya, today I was trying to get a response from GPT-OSS-120B via vLLM - and failed miserably! Has anybody gotten it to work, i.e.
-
Getting unexpected output with Gemma 4 31b-it on vLLM (www.reddit.com) vllmgemmaopenai 27w
Hey everyone, I'm running into a weird issue and hoping someone here might have a fix or some troubleshooting ideas. I'm currently trying to run the new Gemma 4 31b-it model using vLLM (v0.20.0-cu130) deployed via Helm chart (https://gith…
-
Gemma 4 31B MTP Drafter on H100 -- Real Benchmarks + DFlash Comparison (www.reddit.com) vllmgemma 27w
Just tested Gemma 4 31B with the new official MTP Drafter on my H100 today and compared the approach with DFlash to help you decide which one to use. Without drafter: 13.7 tok/s.
-
Show HN: Per-request emotion steering for vLLM, with batching preserved (github.com via hn) vllm 27w
emotion-steering Extract and serve CAA-style emotion steering vectors for any HuggingFace causal LM, with a fast vLLM path for Qwen3. ┌────────────┐ ┌────────────────────┐ labeled │ extract │ vectors + AUC report │ serve │ contrasts ├─────…
-
A plug-n-play open-source pruning tool that is workload-aware (www.reddit.com) vllmollamallama 27w
This project was born out of time I spent digging into a biologically inspired algorithm I was using to measure co-activation for placement of experts and ranks onto chips. The default scheduling that vllm provides can end up causing laten…
-
vLLM Just Merged TurboQuant Fix for Qwen 3.5+ (www.reddit.com) vllmqwen 27w
Previously it was throwing a 'Not Implemented' error due to Mamba layers. Going to test it now!
-
FastDMS: 6.4X KV-cache compression running faster than vLLM BF16/FP8 (www.reddit.com) vllmllama 27w
Last year researchers affiliated with NVIDIA, University of Warsaw, and University of Edinburgh published Dynamic Memory Sparsification (DMS), a KV-cache sparsification technique using learned per-head token eviction, reporting up to 8x KV…
-
Should I sell my RTX3090s? (www.reddit.com) vllmgemmaqwen 27w
I have a GPU server (4 × RTX3090s) that I've been using for research and PoC in the past 2 years. Mostly running vLLM for Qwen, GPT-OSS, and Gemma.
-
Sglang is better for serving a model for a personal agent harness? (www.reddit.com) vllm 28w
If one has enough vram, would Sglang be a superior choice than vLLM or llamacpp in terms of inference speed for serving a model dedicated to powering a personal (single user) agent harness like Hermes agent? Sglang has MTP for speculative…
-
Advice needed on eGPU and Mini PC (www.reddit.com) vllmmoeqwen 28w
Hi all, I come across to relatively niche problem and could not find much useful posts or guides about it. I have a mini pc (Beelink Ser 8, 8745HS and 32GB 5600 DDR5 SODIMM) headless server for hosting some routing services, and I am wonde…
-
I have been using llama.cpp to run some models recently. For example, I've been running GLM-4.7-Flash with this command .\llama-server.exe -hf unsloth/GLM-4.7-Flash-GGUF:Q6_K_XL --alias "GLM-4.7-Flash" --host 127.0.0.1 --port 10000 --ctx-s…
-
Show HN: Valkyr LM Inference with Realtime Guarantees (github.com via hn) vllmllamaopenai 28w
Valkyr is a fresh take on LM Inference runtimes. It's quite different from llama.cpp, vLLM, or ZINC for example.
-
Anyone running HUANANZHI H12D-8D + BMC with 4x RTX 3090 for LLM inference? (www.reddit.com) vllmllama 28w
Hi everyone, I'm considering building a home LLM inference rig around: - HUANANZHI H12D-8D + BMC - AMD EPYC 7002/7003 - 4x RTX 3090 24GB - DDR4 ECC RDIMM, 8-channel - Linux + vLLM / SGLang / llama.cpp - Open frame, PCIe 4.0 x16 risers The…
-
Requesting advice on local AI setup for academic use (www.reddit.com) vllmllama 28w
I'm about to do a clean install of Ubuntu 26.04 on a desktop that has a 5060ti 16gb and a 4060ti 16gb. Can you help me work out the best local AI setup for my use cases?
-
Need advice on Qwen 3.6 27B INT4 quantization (www.reddit.com) vllmllamaqwen+1 28w
Hello everyone, I think Qwen 3.6 27B is good enough that it might take a while before we get a clearly better model at a similar size. I have a single headless RTX 3090 with a 300W power limit.
-
Hey guys, A couple of weeks ago, I asked this sub for the hardest Vision use cases you were dealing with to test the newly dropped Qwen 3.6 against Gemma 4. I finally finished running the gauntlet side-by-side locally on vLLM (FP8 quants)…
-
Kv cache quantization: ignorance, or malice? (www.reddit.com) vllmqwenagentic 28w
I run Qwen-3.6 27B FP8 on vllm for long-horizon agentic coding harness workloads with high context window and concurrent sub-agents. On two 3090s that aren’t used for anything else, it seems reasonable to expect a good balance between spee…
-
[Help] Running big dense models faster (www.reddit.com) mistralvllmllama+1 28w
I have been trying Mistral 3.5 on my 4x RTX 3090 rig with llama.cpp. Inference is slow (about 11 t/s) even without anything being offloaded to the CPU.
-
The angle here is native Windows, no WSL. Simple installation, open source, no telemetry.
-
How can I locally run Deepseekv4 1.6T? I can use a VPS. (www.reddit.com) vllmollama 28w
I wanted to use vast.ai, but ollama doesnt have it, and when i used vLLM I didn't have success. I genuinely don't know what failed.
-
Most of my LLM cost was on the wrong tier of work. Classification, extraction, JSON formatting, summarization I'm going to review anyway.
-
Whats the latest status on 7900xtx multi-GPU setups? (www.reddit.com) vllm 28w
I am currently running dual RTX 5060 ti 16gb (both of which are easy to sell or re-use in other PCs at home) and monitoring the used market for more of the same and alternatively RTX 3090. I couldn't help but notice that sometimes some qui…
-
Best RTX Pro 6000 vllm settings? (www.reddit.com) vllm 28w
Just got myself (for my company) a RTX Pro 6000 Blackwell Workstation card. Managed to get really good TPS on qwen3 27b fp8.
-
32 MI50 32GB setup moonshotai/Kimi-K2.6 int4 @ 9.7 tok/s (output of 136 tok) and 263 tok/s (input of 14564 tok) on vllm-gfx906-mobydick Github link of vllm fork: https://github.com/ai-infos/vllm-gfx906-mobydick Power draw: ~640W (idle) / ~…
-
Mixing 3090 with 3080 20G (modded) for vllm (www.reddit.com) vllm 28w
Has anyone tried mixing 3090s with 3080 20G for vllm using tensor parallelism? I know vllm normally discourages mixing GPUs, but given how much 3090 is selling nowadays, the modded 20G 3080s with half the price feel like better deals.
-
Following up on our previous post about running Qwen3.6-27B on a single RTX 3090 (~125K context, higher TPS). We’ve been pushing further on both context length and stability for tool-agent workloads.
-
thinking of gemma 4 26B vs 31B (www.reddit.com) vllmgemmamoe+2 28w
I see a big difference in agentic coding between gemma-4-31B-it-Q5_K_M and gemma-4-26B-A4B-it-UD-Q8_K_XL. The 26B model is much faster because of A4B and generally works well, but there is a big difference in thinking.
-
Reasoning Guard: Stopping LLM Thinking Loops at the Proxy Layer (www.reddit.com) vllmmoeagentic 28w
Reasoning Guard: Stopping LLM Thinking Loops at the Proxy Layer I’ve been running Qwen3.6 MoE behind a vLLM proxy and hit a specific reliability issue: occasional runaway reasoning loops. This isn’t a criticism of Qwen3.6.
-
Only 120 tps on Qwen 35b on h200 (www.reddit.com) vllmqwen 28w
Just a sanity check, this is too slow and something is wrong, right? Like, this is setup with mtp, vllm with awq quants, I suspect that I did configure something wrongly.
-
Is Mistral-3.5-Medium-128B broken in Llama CPP? (www.reddit.com) swe-benchmistralvllm+1 28w
Trying some if Bartowski's Q4 quants. Using Vulkan with the latest main branch as of a few hours ago.
-
What’s up with mobile LLMs? (www.reddit.com) vllmollama 28w
I see a lot of support for running LLMs on PCs with ollama to vLLM. Whats the current state for running on mobile?
-
5060ti quad-chads - vllm (the reluctant arc) - pp and tg talk (www.reddit.com) vllm 28w
Okay, so I have this quad 5060ti setup and for forever I have had people nagging me to try vllm. I thought it was too complicated, like varsity golf or putting on both legs of pants at the same time.
-
Qwen3.6 27B on dual RTX 5060 Ti 16GB with vLLM: ~60 tok/s, 204k context working (www.reddit.com) vllm 28w
I’ve been testing Qwen3.6 27B on a pretty non-standard local setup and figured the numbers might be useful for anyone looking at the newer 16GB Blackwell cards. Hardware: 2x RTX 5060 Ti 16GB 32GB total VRAM Proxmox LXC 16 vCPU ~60GB RAM CU…
-
vLLM-Compile: Bringing Compiler Optimizations to LLM Inference (docs.google.com via hn) vllm 28w
vLLM-compile: Bringing Compiler Optimizations to LLM Inference Luka Govedič vLLM Committer Senior Machine Learning Engineer, Red Hat 1
-
3.6 27B Tool Calling Issues (vLLM) (www.reddit.com) vllmqwen 28w
Has anyone got a reliable vLLM recipe for 3.6 27B that fixes the tool calling issues? I am getting "Not let me..." - then nothing.
-
Disaggregated Serving for Hybrid SSM Models in vLLM (vllm-website-lx4pji0mz-inferact-inc.vercel.app via hn) vllm 28w
Disaggregated Serving for Hybrid SSM Models in vLLM Introduction Hybrid architectures that interleave Mamba-style SSM layers with standard full-attention (FA) layers — such as NVIDIA Nemotron-H — are gaining traction as a way to combine th…
-
Anyone have experience with both? Luce is llama.cpp with custom dlflash and noonghunnas project is vllm with patches.
-
Load balancer for vLLM server instances? (www.reddit.com) vllm 28w
Hello all, the docs for the vLLM production stack suggested autoscaling the vllm worker instances based on the number of waiting requests, but it seems like this would only help with new coming requests? We are having burst LLM calls which…
-
Power-limit vs TG/s for 2x3090 (www.reddit.com) vllmopenai 28w
Trying to find the sweet-spot to tradeoff between power and tg/s. 250W seems to be a sweet spot for Qwen3.6-27B.
-
I have a 4 x R9700 system on Threadripper pro, but I have never been happy with the performance of my GPUs in vLLM. I have started benchmarking any new model I try out with llama-benchy so that I can get a better idea of how models of diff…
-
Qwen3.6-27B vLLM Docker Docker-based vLLM serving for Qwen3.6-27B with Lorbus AutoRound INT4 quant and MTP speculative decoding. Model is downloaded at runtime and stored on a host volume so the container can be upgraded without redownload…
-
Ubuntu 26.04 vs 24.04 speed improvements for inference? (www.reddit.com) vllmllama 29w
I'm curious if any brave soul has upgraded their computer (especially if it's Strix Halo) from Ubuntu 24.04 -> 26.04 and seen a significant performance improvement for inference with VLLM, llama-server, and/or LM Studio.
-
I'm a daily llama-cpp user and was hoping to try MTP on vLLM. Unfortunately, pipeline parallelism + MTP does not seem to work with this model in vLLM.
-
Intel B70: LLama.ccp SYCL vs LLama.cpp OpenVino vs LLM-Scaler (www.reddit.com) vllmllama 29w
In case anyone is interested, I decided to test out LLama.cpp's new OpenVino backend to see how it compares on Intel GPUs. At first glance, it stomps all over the previous best-case, SYCL, but lags behind LLM-Scaler (Intel's VLLM fork), li…
-
locally uncensored is a desktop app that combines four things most people run separately: chat, a coding agent, image generation, and video generation. all local, all on your hardware, no docker, no cloud account needed.
-
I'm running Qwen/Qwen3.6-27B-FP8 via vLLM using this command: vllm serve Qwen/Qwen3.6-27B-FP8 --tensor-parallel-size 4 --gpu-memory-utilization 0.95 --max-num-seqs 8 \ --enable-auto-tool-choice --tool-call-parser qwen3_xml \ --enable-prefi…
-
How do you actually use Qwen3 72B Instruct locally? (www.reddit.com) vllmollamallama 29w
I just got Qwen3 72B Instruct running on a high RAM setup and I’m kinda confused about the proper way to use it. What’s the correct workflow for running it smoothly (like best quant, tools, or runtime)?
-
Will llama.cpp multislot improve speed? (www.reddit.com) vllmllama 29w
I've heard mostly bad opinions about multiple slots with llama.cpp (--parallel > 1). I guess comparing to vLLM it might be worse at this, but I recently tried vLLM on 4 slots and it indeed improved the overall speed significantly (150-170t…
-
your daily driver stack, what's it look like? and why? (www.reddit.com) vllmllamacowork+2 29w
What it says in the title, I'm interested in hearing what you all have landed on as a workable / useful stack for you. Mine looks like this: back end inference servers - llama.cpp, vLLM | V hermes-agent - cron jobs + OpenAI compatible endp…
-
Qwen3.6-27B-FP8 - JS file is too long and causing JSON truncation (www.reddit.com) vllmqwen 29w
Apologies in advance, if this is a newbie question. When running Qwen3.6-27B-FP8 using the below command on an Nvidia RTX PRO 5000, in opencode, I am seeing errors such as: "The issue is that the JS file is too long and causing JSON trunca…
-
Qwen3.6-35B-A3B KLDs - INTs and NVFPs (www.reddit.com) vllm 29w
https://preview.redd.it/c76w57d1yexg1.png?width=1482&format=png&auto=webp&s=1164d8bc3e2e8a4157f26dd5583238a736474932 KLD for INTs and NVFP4s. AS ALWAYS - Use Case is important.
-
ASUS Ascent GX10 - Having tons of issues (www.reddit.com) vllmollama 29w
Hi all, Looking for some advice with a GX10 I purchased about 4 months ago. I've been having all kind of issues trying to run local models on this device.
-
Qwen3.6-27B at ~80 tps with 218k context window on 1x RTX 5090 served by vllm 0.19 (www.reddit.com) vllm 29w
Qwen3.6-27B is out for a few days and the NVFP4 with MTP is dropped earlier on HF: https://huggingface.co/sakamakismile/Qwen3.6-27B-Text-NVFP4-MTP Can follow the same recipe I used for Qwen3.5-27B to achieve ~80 tps on a single RTX 5090 at…
-
I wanted to figure out which of the newer small and mid-size models are actually worth running on a single H100, so I put 8 of them through a proper vLLM benchmark and recorded what came out. The setup was simple.
-
To run deepseek v4 flash how much max vram we need? 175 gb or 320gb? (www.reddit.com) vllmdeepseek 29w
As far as i know the weight is of 160gb + 9.6gb needed for max 1 million token window + 5 gigs overhead = 175gb vram. But vllm and othere sources said "To use the full 1M context, you need 4x A100 80G" --> thats a 320gb vram ??
-
I'm sure people have asked before for settings for these gpu's, but for me, no matter what I do, It doesn't work as good as 3.6 35B! I've tried VLLM and LLAMACPP .
-
VLLM PR : New MoE model from Cohere soon (github.com via reddit) vllmmoe 29w
Easy, fast, and cheap LLM serving for everyone | Documentation | Blog | Paper | Twitter/X | User Forum | Developer Slack | 🔥 We have built a vLLM website to help you get started with vLLM. Please visit vllm.ai to learn more.
-
Hey guys, I built a custom vLLM pipeline to run Gemma 4 (31B FP8) and Qwen 3.5 side-by-side locally to see how they actually perform in the wild with preprocessing of audio and images. But of course new model Qwen 3.6 27B came out just whe…
-
Qwen3.6 uncensored AWQ (www.reddit.com) vllm 29w
I have tested Qwen3.6-27B-Uncensored-HauhauCS-Aggressive-Q5_K_P.gguf on my 4x3090 system (opencode) and find it really good and fast. However, I can't find any uncensored models for vllm (preferably as AWQ).
-
DeepSeek V4 in vLLM: Efficient Long-Context Attention (vllm-website-pdzeaspbm-inferact-inc.vercel.app via hn) vllmdeepseek 29w
DeepSeek V4 in vLLM: Efficient Long-context Attention We are excited to announce that vLLM now supports the DeepSeek V4 family of models (deepseek-ai/DeepSeek-V4-Pro and deepseek-ai/DeepSeek-V4-Flash ). These models feature an efficient lo…
-
What are your favorite LLMs for translation/docuement work? (www.reddit.com) vllmgemma 29w
I am currently working on a system to translate books/web novels. I got a working prototype, but now I am looking into optimizing it.
-
Hi guys, I’ve been running side-by-side experiments on Gemma 4 (31B FP8) and Qwen 3.5 Vision for the last few days using vLLM in Docker to see how they actually handle real-world images and video. A few things I found out: 1.
-
Always been stuck with models that fit on my 16gb .... Going to have about a week for free with 4x rtx6000pro .
-
Self-hosted AI: What is the way to go? (www.reddit.com) vllmopenai 29w
Hi everyone, I’m building a small support chatbot in Symfony for a limited group of users (around 300 people). For the MVP, I’m running everything locally on an NVIDIA DGX Spark with the GB10 Grace Blackwell superchip, using vLLM.
-
Claude Code drafted the prose. I did the research, direction, architecture, ran the code, caught the bugs, and reviewed every commit.
-
LLM performance benchmarking update (www.reddit.com) vllm 29w
-
I am using an Arc Pro B70 to do inference, and it's token generation speed is fine using Ollama, but it takes *forever* to do a prefill. vLLM absolutely tackles the prefill problem (nearly instant responses), but I can't run nearly as larg…
-
Show HN: Aide – A customizable Android assistant (voice, choose your provider) (aideassistant.com via hn) vllmollamachatgpt+1 29w
-
-
-
-
-
-
-
Qwen3-30B-A3B-Instruct-2507 is better than the new Qwen 3.6 for our tasks (www.reddit.com) vllmgemmamoe+1 30w
-
Multi GPU setup help (www.reddit.com) vllmmoe 30w
Hi guys I managed to get a multi GPU setup going with a 3090 and three 3060 bringing my vram to 60gb along with 64gb ddr5. The objective is to run the largest coding model I can at a respectable token speed of over 20 tokens / second.
-
Setup: 3x Mac Minis in a cluster running MLX. One node drives training, two push rollouts via vLLM.
-
made a desktop app that puts ollama, comfyui and coding into one window (www.reddit.com) vllmollama 30w
been using local AI for a while now but my workflow was a mess. ollama for chat, comfyui for images, different tools for video and coding.
-
M1 Pro 16GB users: what local LLM configs are actually usable day to day? (www.reddit.com) vllmollamallama 30w
I'm trying to get past generic "best model" recommendations and collect real-world configs from people on similar hardware. My setup: MacBook M1 Pro, 10-core CPU, 14-core GPU, 16 GB unified memory.
-
TurboQuant on MLX & vLLM (www.reddit.com) vllmllama 30w
MLX https://github.com/Blaizzy/mlx-vlm?tab=readme-ov-file#turboquant-kv-cache vLLM https://github.com/vllm-project/vllm/pull/38479 MLX & vLLM users, please share your experience with benchmarks(t/s). Adding llama.cpp Links related to Turbo…
-
Those of you running minimax 2.7 locally, how are you feeling about it? (www.reddit.com) vllmminimax 30w
Im running the raw version straight from the minimax release on hugging face (https://huggingface.co/MiniMaxAI/MiniMax-M2.7) on 3 rtx pro 6000's on vllm. So no quantization.
-
Bench 8xMI50 MiniMax M2.7 AWQ @ 64 tok/s peak (vllm-gfx906-mobydick) (www.reddit.com) vllmminimax 30w
Inference engine used (vllm fork): https://github.com/ai-infos/vllm-gfx906-mobydick/tree/main Huggingface Quants used: cyankiwi/MiniMax-M2.7-AWQ-4bit Relevant commands to run: docker run -it --name vllm-gfx906-mobydick-mixa3607 -v ~/llm/mo…
-
Here is how to run the new Qwen3.6-35B-A3B > At full context on a 4090 - IQ4_XS gguf with llama cpp > At full context on a Spark - FP8 with a tweaked vLLM Here is the docker compose with llama cpp services: llamacpp: container_name: llamac…
-
Qwen 3.6: worse adherence? (www.reddit.com) vllmqwenrag 30w
Just swapped Qwen 3.5 for the 3.6 variant (FP8, RTX 6000 Pro) using the same recommended generation settings. My stack is vLLM (v0.19.0) + Open WebUI (v0.8.12) in a RAG setup where the model has access to several document retrieval tools.
-
Alibaba open-sources Qwen3.6-35B-A3B, a 35B MoE model with 3B active parameters (huggingface.co via hn) vllmmoe 30w
Qwen3.6-35B-A3B [!Note] This repository contains model weights and configuration files for the post-trained model in the Hugging Face Transformers format. These artifacts are compatible with Hugging Face Transformers, vLLM, SGLang, KTransf…
-
Reproduction of TurboQuant (www.reddit.com) vllmllama 30w
There have been many TurboQuant implementations recently in llama.cpp, mlx, vllm, and sglang, but a lot of the discussion and code around them feels pretty noisy and looks to be AI-generated. I’m trying to understand which claims from the…
-
Hey r/LocalLLaMA, I've been running a small 4-node DGX Spark cluster on a 400µT fabric switch and got frustrated with the usual raw Ray/vLLM scripts and EXO basically ignoring pure NVIDIA paths. I started from the solid foundation in [eugr…
-
gemma4 e2b ore4b on rtx 5070 ti laptop 12GB not running on vLLM (www.reddit.com) vllmgemma 30w
I cant get gemma 4 e2b or gemma 4 e4b to run on my laptop. I am runnning it via docker as per vllm website and i get the error : Free memory on device cuda:0 (9.71/11.5 GiB) on startup is less than desired GPU memory utilization (0.9, 10.3…
-
Lower inference speed of Gemma4 26BA4B on vllm. (www.reddit.com) vllmqwen 30w
For my earlier use case I used to host qwen 2.5 vl 7b gptq int4. Now I was looking to switch to Gemma4 26B A4B, as it would improve performance as well as improve latency considering only 4B parameters are active..
-
Please help me pick the right Qwen3.5-27B format/quant for RTX5090 (www.reddit.com) vllmopenclawsonnet+2 30w
Hi all, first post here. I've started a project in OpenClaw a month ago, and it's been a very "intense" 4 weeks to say the least...
-
So, yesterday run was a success and I did get an avg rollout length of about 64 tokens as attached in the image! This was with quality_reward + length_penalty (more info below!) Next, I'll be going with length penalty as the reward and wit…
-
DGX Spark just arrived — planning to run vLLM + local models, looking for advice (www.reddit.com) vllm 30w
Just got a DGX Spark set up today and starting to configure it for local LLM inference. Plan is to run: • vLLM • PyTorch • Hugging Face models as a local API backend for an application I’m building (education / analytics use case, trying t…
-
Deep Dive into Efficient LLM Inference with Nano-vLLM (cefboud.com via hn) vllm 30w
Deep Dive into Efficient LLM Inference with nano-vLLM A look inside a lightweight implementation of vLLM. KV cache, paged attention, tensor parallelism &multi-GPU support, etc.
-
I am playing around with Intel Arc B70, still trying to decide whether I keep it or not. After some battle, I got it working with Radeon 5500 and B550M, now I am on to the fun part of getting software to work.
-
I'm trying to figure out what sort of hardware setup i will need to accomodate a userbase of 100 users (not necessarily concurrent). Does anyone have any idea what sort of setup i'd be looking at?
-
Show HN: Fleet Watch – preflight guard for local AI inference on Apple Silicon (github.com via hn) vllmollama 30w
Fleet Watch Process governance for AI workloads on a single machine. The Problem You're running MLX, Ollama, vLLM, Candle/Cake, experiment runners, and AI coding agents on the same machine.
-
Turboquant in vllm kv cache - how to implement ? (or any other rotational kv cache) (www.reddit.com) vllm 30w
Hi folks - is there any "standard" (acceptable) vllm way of implementing turboquant or a similar rotational quant for vllm's kvcache? I found https://github.com/mitkox/vllm-turboquant - but this seems inactive.
-
I’m working on an LLM agent setup (using Qwen-style chat templates with tool calling), and I ran into a design trade-off that I’d like to get some insights on. In these templates, the full tool definitions (JSON schemas) are injected into…
-
What is the best way to deploy LLM on 3x3090? (www.reddit.com) vllmgemmallama 30w
Two questions: which model? In my mind, Qwen3.5 27b or Gemma 4 31b are top options.
-
Using older vLLM version via Docker -- how do you use GGUF quants? (www.reddit.com) vllm 30w
So vLLM recently added the feature to use GGUF quants with the syntax author/model:quant format. I was just wondering if people were able to use the quants on older vLLM versions.
-
current: 1x 16GB 5060Ti. worth a 2nd for OpenCode? (www.reddit.com) vllmllama 30w
my current build is just a 16GB 5060Ti running on a 3800X with 32GB DDR4. not really anything special, but I only really use it right now for Qwen3-VL-8B-Instruct at INT8 to do handwriting transcription (and it works great for that). someo…
-
Hi everyone, I’m building a local AI pipeline on WSL2 (Ubuntu) specifically for Product Visualization. My goal is to orchestrate LLMs for scene generation and Stable Diffusion/ComfyUI for high-fidelity rendering, keeping my Windows host cl…
-
Going local with old GPUs (www.reddit.com) vllmollama 31w
I'm an ex crypto miner with remnant mining parts so I threw them together into a franken hydra case. I've been using claude oath previously, but they just shut that door last week or so.
-
No GPU left behind: Unlocking Efficiency with Co-located vLLM in TRL (huggingface.co) vllm 75w
-
Introducing multi-backends (TRT-LLM, vLLM) support for Text Generation Inference (huggingface.co) vllm 95w