Hallucinated AI & agentic coding news. Some of it is real.

event

Gemma

533 items · started 2024-02-21 · ongoing (last activity 2026-09-17)

  1. openjev Disclaimer: This project was inspired by vinnylarouge/jevlike. It is an independent reimplementation and is not affiliated with or endorsed by the original author.

  2. Who the judge is can affect an LLM-as-judge result, but measuring that effect without confusing it with candidate quality is difficult. We study four open-weight families (Llama 3.1, Qwen 2.5, Gemma 2, and Yi 1.5) in a fully crossed pairwi…

  3. TL;DR Three facts and one command. Bench — Mac mini M4 · 24 GB unified memory · macOS 26.5 · Ollama 0.34.0 · gemma4:26b Q4_K_M (MoE, 25.2B total / ~4B active) · measured 2026-09-13 ollama ps )sudo sysctl iogpu.wired_limit_mb=20480 # raise…

  4. Open-weight models, fully deterministic. The same answer every time, byte for byte, at the floor price.

  5. About Press Copyright Contact us Creators Advertise Developers Terms Privacy Policy & Safety How YouTube works Test new features NFL Sunday Ticket © 2026 Google LLC

  6. Gemma 4 E4B LLM Wiki Turn your notes into a living Karpathy LLM wiki. Chat with your vault, connect ideas, find gaps, and generate quizzes & flashcards — with Gemma 4 E4B running locally inside Obsidian.

  7. Sparse autoencoder (SAE) features are increasingly used to explain and steer language-model behavior, but it remains unclear whether a feature found in one language context plays the same causal role when processing prompts in another lang…

  8. About two weeks ago I started an experiment: instead of prompting a local model to play a character, give it a folder and let it become one. The idea is simple.

  9. Hi, I'm not a fan of chat as the main UI for interacting with AI, so I'm trying something where you interact with LLMs by handling regular files and folders. Agentray is an invisible AI agent.

  10. PicoLM is an LLM inference engine written in C99. It currently supports llama-2, GPT-2, Qwen 3.6/3.8(+MoE) and Gemma-3n models.

  11. hi, im desperate. at this point i dont know whats happening.

  12. This paper proposes a layer bit allocation method for Gemma-3-1B, formulating the problem as performance maximization (latency decrease) given a degradation budget constraint (allowable level of generation quality loss). This approach is d…

  13. I’ve been working on a small project called gemma4.c. The idea is pretty simple: you can download a modern language model, compile one 700-line C file, and have it generate text on an ordinary CPU.

  14. gemma4.c Gemma 4 E2B CPU inference in 700 lines of pure C. An educational project made to understand how LLM inference works.

  15. specs hardware: M4 Max 128GB Studio inference engine: oMLX & lllama.cpp insights it still very early, so had to disable oMLX K/V caching, qwen4_exp architectureis not yet supported + the obvious n-grams with which the whole 4 bit quant tak…

  16. I heard qwen3.8 27b is only good for coding really. Is that true I could do qwen 3.5 27b but qwen3.6 27b doesn’t fit on my Vram at IQ_XS I feel Gemma 4 31b might be good but it’s kinda not fitting in vram unless I go iQ3xxs and the qat wit…

  17. Hi all, I'm looking for the best model for a hobby project and trying to make sense of the various data I came across. I know benchmarks do not often translate to the real world, especially to your particular use case (whatever it may be).

  18. So I was testing this technique of runtime steering on tiny versions of Qwen 3.5 and Gemma 4 (2B and 4B). Basically, without changing the weights (like with Heretic/ablation, for example), we steer the model in the opposite direction of a…

  19. I ran 11 uncensored variants of Gemma 4 12B that I grabbed from huggingface, sorting by downloads. 10 full abliterations plus 2 LoRA adapters which were requested to be added in the comparison, against the official base.

  20. I spent the last several months building a photo journal that runs a full multimodal LLM — Gemma 4 E4B — entirely on an iPhone. No cloud inference, no accounts, no analytics, full privacy; the App Store privacy label is "Data Not Collected…

  21. Hey guys, I'm setting up a local workflow on a single 24GB RTX 3090 to handle project planning—specifically digesting massive (~128k context) requirements documents/PRDs and spitting out a ton of structured .md files to act like Jira ticke…

  22. Exciting news: Qwen3.8-27B by @Alibaba_Qwen just landed in Code Arena: WebDev at #9 overall with 1595 pts. It is the only model in its size class in the top 10, and also reshapes the Pareto Frontier!

  23. As per title. I'm not affiliated with the team behind this model in any way, shape or form.

  24. Hi everybody! Every now and then these days, we’re seeing really huge open-weight models popping up.

  25. I have been benchmarking local LLMs on a Mac M4 Pro 24 GB RAM using LM Studio. I've tested mostly with 4-bit quantization, both MLX and GGUF, from 4b to 35b models, with speeds of 3 to 40 tokens/second.

  26. Gemma 12B is obviously a very well trained model, I always thought the fine tuning they did on it wasn't really cut out for agentic coding. From my own experiences it struggles to use the tools it's given from Github Copilot and is also ve…

  27. Hello, I have a few questions that I can't seem to find a clear answer to. Does it make sense to make your own GGUF?

  28. I was finally able to replicate tensor level allocation outside the Gemma family. https://huggingface.co/ByteOtter/Qwen3.5-4B-CADA-IQ2_XS After the Gemma 4 12b, e4b and gemma 3 4b results, I attempted to expand into qwen and ran into a few…

  29. I'm here to show some benchmarks while using llama cpp with an AMD V620 on Windows 11 via Vulkan & ROCM. The benchmarks were written out by AI, but are verified by myself to be correct.

  30. Gemma is Google DeepMind's family of lightweight, state-of-the-art open models. Contents Start Here Models Core Models Variants Inference Local Hosted Fine-Tune Tutorials Demos and Applications Gemma 4 Good Challenge Gemma in Space Researc…

  31. blog Gemini 3.7 Flash offered for free Gemini 3.7 Flash is now included with every pad. It joins Gemma 4 31B and Gemini 3.6 Flash: three models in OpenCode, no provider account, no key to paste, usage included with the pad.

  32. Apertura A from-scratch Objective-C++ / MLX rebuild of Google's Gemma-4 for Apple Silicon — built to be inspected, observed, and experimented with. Most language models are run behind glass: you send a prompt and get an answer, with no way…

  33. blog Free coding models on every pad Every reachpad now includes two models you can use through OpenCode without bringing an API key: Gemma 4 31B and Gemini 3.6 Flash. Model usage is included with the pad.

  34. Gemma 4b vs Gemini Flash: You Don't Need Frontier Models For Tool Calling Workflows What our Acebench analysis showed about the models Most of what makes an AI assistant useful isn’t prose — it’s calling tools: booking the meeting, adding…

  35. So i had been building a side project screenmind ,sort of local ai desktop assistant, and hit a confusing multimodel failure... texts worked,but vision and image failed no crash just returning <unsused49>.

  36. Imagine building a fully autonomous robot that can see, hear, and react to its environment in real time, completely offline on a single compact device like Raspberry Pi. Edge AI unlocks this exact autonomy.

  37. I (like thousands of other engineers) got tired of trying to understand Claude's writing. ASD-STE100 instructions didn't change much, so I decided to approach the problem in the 2026 style: use an LLM.

  38. About Press Copyright Contact us Creators Advertise Developers Terms Privacy Policy & Safety How YouTube works Test new features NFL Sunday Ticket © 2026 Google LLC

  39. the-veil-12b The first official model for the nl-veil harness. A Gemma 4 12B derivative, fine-tuned on nl-veil's own tool surface.

  40. I started thinking over why doesn't fireworks support voice models. There are really good opensource models available now, like parakeet, kokoro, Qwen ASR etc but no way to use it without managing a bunch of GPUs yourself.

  41. Put together an extensive open source test suite for voice assistants. You can view it including current leaderboard at: https://git.cicero.sh/aquila/ha-voice-test-suite/ Tests are reproduceable, with clear instructions on how to run them…

  42. Hi HN, I built a specialized inference engine for running 4-bit Gemma 4 26B-A4B-IT on any M-series Mac using about 2 GB of RAM. It is called TurboFieldfare and is written in Swift and Metal.

  43. I wanted to share a use case for Noema Overfit, a model-paging system available today in the Noema app. For disclosure, I founded Noema and I am part of the team that has helped develop the system.

  44. The integration of large language models (LLMs) into the nuclear power industry requires outputs grounded in domain-specific knowledge. This study evaluates a 31-billion-parameter open-weight multimodal model (Gemma 4 31B-IT) on its capaci…

  45. Hey HN, we’re the developers of OpenLake, an open source storage engine for offloading LLM KV caches from GPU memory into a shared tier of RAM and NVMe. We built OpenLake because KV caches are outgrowing GPU memory.

  46. I'm the founder of Scalattice, this is my second company, third total product. I'm a 2x founder building some challenging software, some easy software, and some curiosity based tools that I've just always wanted to be a part of!

  47. This paper tests whether holonomy concentrates on active sparse-autoencoder (SAE) feature planes in Gemma 2 2B, a concrete operationalization of the broader semantic-concentration prediction. Holonomy is measured at the final-token layer-1…

  48. Open models behind one OpenAI-compatible endpoint. No account yet?

  49. Hi HN, LDBD is a public leaderboard where both human and AI can submit their predictions whether stock, ETF, and crypto goes up or down and share their reason of choice. This service starts from one question: does anyone or any AI can real…

  50. Hey HN, Henry & Roman here from Cactus. A small, on-device model is fast and private, but sometimes wrong, but frontier models are getting expensive pretty fast.

  51. We're excited to announce AI text and image detection in Bouncer! We've been seeing a lot of Bouncer users try to filter out AI content from their feeds by adding "AI slop", "AI video", or "AI written" as filter terms.

  52. LLM Red Team Lab A hands-on kit for educational, authorized red teaming of any locally-run LLM. It works with any OpenAI-compatible model — Llama, Mistral, Qwen, Gemma, DeepSeek R1, and more — and covers the two ways an LLM system gets exp…

  53. This study examines the cross-prompt generalization and first-language (L1) scoring effects of a LoRA-adapted open-weight large language model (Gemma-3-27B-it) applied to automated essay scoring. Using the identical model and inference con…

  54. I'm making Vox (a voice-to-text app for macOS and Windows that I built with Claude) free for everyone, and the repo is now open. You hold a hotkey, talk, release, and the transcribed + polished text lands in your clipboard (or gets pasted…

  55. I asked Codex 5.6 Sol Medium to one-shot a tree-view version of The Odyssey because I wanted to familiarize myself with it before I see the film. Think of it as an interactive Cliffs Notes where you can stay high level or drill down to the…

  56. June 8, 2026 · By Ryan Findley Running Gemma 4 26B at 5 tokens/sec on a 13-year-old Xeon with no GPU There’s a server in my basement that has no business running a modern language model. It’s a repurposed HP StoreVirtual storage box, rough…

  57. I've built fully on-device (macOS) meeting transcription/summaries/action-items app. It uses parakeet and gemma 4 or OpenClaw or Hermes Agent to drive main functionalities.

  58. Experimental Godot 4.7 project running Gemma 4 inference entirely in GDScript and Vulkan compute shaders.

  59. Hi HN, Roman and Henry here from Cactus (https://github.com/cactus-compute/cactus). We just shipped the biggest upgrade to our on-device inference platform: - Built-in model confidence-based routing to hand off inference runs to the cloud…

  60. Today we’re releasing ZML/LLMD. It’s a self-contained inference server that runs LLaMa, Gemma, Qwen and Mistral LLMs transparently on 5 architectures: NVIDIA CUDA, AMD ROCm, Google TPU, Intel oneAPI and Apple Metal.

  61. mulot [-4285F4?logo=googlechrome&logoColor=white)]() Agentic AI web pentester that drives a browser. An open-weights LLM (GLM-5.2, Gemma or Qwen) drives a real headless Chromium through a Burp-style toolkit and works a target the way a hum…

  62. We introduce Gemma 4, a new generation of open-weight, natively multimodal language models in the Gemma model family. Designed to advance compute efficiency and reasoning, the Gemma 4 model suite features dense and Mixture-of-Experts archi…

  63. Deploying Large Language Models (LLMs) in high-stakes clinical settings remains limited by structural hallucinations, weak deterministic reasoning over tabular patient data, and omissions in vector retrieval. This paper presents the archit…

  64. Hi C.J. here, I made a free app that does on device translation using Gemma Translate 4B AI model repackaged for iOS.

  65. I set out to find an answer to a completely different question: Does a model, when attempting to solve a cyber CTF (find the vulnerability in this app, and then Capture The Flag) while knowing how many steps it has left, perform differentl…

  66. MetalChat - LLM inference for Apple Silicon MetalChat is a Metal-accelerated C++ framework and command line interpreter for inference of Meta Llama and Google Gemma models. [!IMPORTANT] The library API and CLI are under active development,…

  67. If you have an Apple Silicon Mac you can run Claude Code completely locally (and free) by pointing it at a local server. Here's how: Setup (2 minutes) brew install mlx-serve mlx-serve run gemma-4-e4b-it # downloads + starts the server Then…

  68. 1. Coding 2.

  69. I was auditing my codes using the now back fable 5 and it kept failing to run runtime tests and this the error i got. so is anthropic now using gemma-4-12b-agentic-fable5-composer2.5-v2-3.5x-tau2 to run tests?

  70. Gemma 4 is now significantly faster in Ollama 0.31 on Apple Silicon via multi-token prediction (MTP), powered by MLX. Performance is now up to 90% faster when used with coding agents, as measured using the Aider polyglot benchmark.

  71. HF Realtime Voice Voice chat over WebSocket against a HF speech-to-speech The result is a speech-to-speech experience that feels dramatically more natural. Instead of waiting for an AI to respond, conversations flow with the responsiveness…

  72. Gemma 4 on Cerebras—The Fastest Inference is Now Multimodal Gemma 4 31B is now running at over 1,800 tokens per second on Cerebras Inference. This multimodal model unlocks an entirely new class of applications, from computer use to image-d…

  73. How do people measure power usage of GPUs at large (32x) self-hosted setups or small multi-rack setups? I've seen some PDUs which collect and transmit data, but I'm unsure of the processes and if/how people do this on small builds.

  74. One thing I miss when using local models is the artifact experience from Claude. With Claude, if you ask for a dashboard, chart, diagram, or landing page, you actually get the thing rendered in the chat.

  75. Hi everyone, I just released Gemma-4-12B-Uncensored-Opus4.7-CoT. To remove the safety filters without destroying the model's reasoning, I combined a precise ablation method with a CoT (Chain-of-Thought) data fine-tune to fully recover the…

  76. Requirements: iPhone with A17 Pro or newer (8 GB RAM floor for the model), iOS 26+. TestFlight beta is open to anyone with a compatible device.

  77. A from-scratch build book Small Kernels Learn the Zig programming language and how large language models actually work — by writing every piece yourself, from a single dot product to GPT-2, a modern Gemma, and the doorway to Whisper. I ori…

  78. A text that asks for nothing still changes the model's answer — and the shift is invisible at both the input and the output TL;DR: Gave Gemma a neutral-topic text to read before asking it about NATO. It refused.

  79. Update to my original post: https://www.reddit.com/r/ClaudeAI/comments/1u8nagi/connected_a_robinhood_account_to_claude_code_and/ I'm building a fully autonomous daily stock-trading desk in a Robinhood "Agentic" account. Opus is the CEO/PM,…

  80. I got tired of sending every text I translate to Google/DeepL. Even with all the opt-out options and privacy policies, it never felt right especially for some work documents, personal writing, or anything sensitive.

  81. Before Fable 5 was shut down, it pushed Gemma 4 to 255 tok/s on WebGPU. Some didn't believe it was real.

  82. Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference

  83. About Press Copyright Contact us Creators Advertise Developers Terms Privacy Policy & Safety How YouTube works Test new features NFL Sunday Ticket © 2026 Google LLC

  84. 9 min read 23 hours ago Building a phone agent on a multimodal LLM: dropping faster-whisper and letting Gemma 4 hear the caller directly — a response-time and reply-accuracy benchmark across English, French, and Mandarin Press enter or cli…

  85. How to Setup a Local Coding Agent on macOS Running Gemma 4 26B-A4B and Qwen3.6 35B-A3B locally with llama.cpp, MTP speculative decoding, multimodal support, and PI as a coding agent. I'd had my internet fail a few times recently leaving me…

  86. On your phone A small open model, like Gemma 4, runs on the device. Completely offline.

  87. I was not sure myself, seeing a lot of statements here and around like "you need XXX VRAM / Unified Memory to run this model". So today I finally tested it.

  88. Hello everyone, i had a 3080ti 12gb and added a 3080 20gb, so it has a bit less speed but more memory than my main card. I could finally get some speed with the usual suspects (i am testing gemma 4 31b/26b-a4b and qwen 3.6 27b/35b-a3b), BU…

  89. Currently recompiling my llama.cpp with support for diffusion Gemma, but I know on my hardware it won't likely be all that viable. I feel like if the goal was to take better advantage of consume GPUs for fast, intelligent generation, build…

  90. I've been experimenting a bit today with letting models reason for creative tasks, rationale being that it might help with keeping track of details and prompt adherence. And predictably, the wall I'm running into is that they all want to d…

  91. So after a few hours of reserach, im torn between getting either a radeon vii or 2 p100 (both options for roughly 240€). The Radeon would give me 32gb of vram and fast inferference, while the 2 p100 would give me a total of 48gb, but rough…

  92. Looking at older Nvidia cards under £200 for Gemma/Qwen MOE coding. Is there any reason to avoid older Titan 12GB cards other than being power hungry?

  93. I recently had a big family photo digitalization done for photos up to 130 years old. There are tons of people that I don't know or I don't recognize as young people in a soft lens.

  94. If Gemma 4 is better, does anyone have a link for the latest fixed template? Using LMstudio.

  95. Model Overview Description: DiffusionGemma 26B A4B IT is an open-weights multimodal generative model developed by Google DeepMind that processes text, image, and video inputs to produce text output via discrete diffusion. Built on the Gemm…

  96. TLDR: I just added an MCP to the Observer framework making it 10x easier to use, so you can create micro-agents that monitor your screen autonomously, literally one sentence and you're done! So just typing "Monitor my Steam download and se…

  97. I've spent the last six months trying to build a fully local, agentic pipeline for a text_processing and extraction tool I use daily. ​Because I’m running everything on a single consumer GPU setup, my choices are limited to smaller, quanti…

  98. https://huggingface.co/mradermacher/gemma-4-31B-it-qat-q4_0-unquantized-i1-GGUF/tree/main https://huggingface.co/mradermacher/gemma-4-31B-it-qat-q4_0-unquantized-GGUF/tree/main I waited a bit before asking this. I have 3060 12GB and 32GB d…

  99. Maybe a niche question, but most of the time, Q8_0 and Q4_0 is a reference to the weights themselves. The activations themselves are available in BF16 format.

  100. Pretty exciting, wonder what it will take from llama.cpp to get it working locally

  101. 26B 4B active parameters with crazy TPS Claims of 700+ TPS on a 5090 Introducing DiffusionGemma

  102. Your data stays on your phone. No bank sync track, no AI advice.

  103. Gemma-4 31B at 256K Context on a $1,400 AMD GPU — TurboQuant KV Cache on RDNA4 Running **Gemma-4-31B-it with a TurboQuant KV cache and HIP graphs together on AMD RDNA4 (gfx1201) — a combination that crashes out of the box and, to our knowl…

  104. I'm trying to use Gemma 4 12B — the new encoder-free unified model (audio/vision/text in one) — for a one-pass audio → response voice assistant: feed the recorded WAV + system prompt and get the reply back as text directly, collapsing the…

  105. I have been curious about how will having a infrastructure that provides agents the capability to explore code bases as relations, rather than text will change the performance of the AI agents So, for the last few weeks, I have been buildi…

  106. Hey everyone. I'm brand new to running LLMs in general, even more new to running them locally, and the sheer number of tools available is absolutely overwhelming.

  107. We have summaries annotated by real humans that we benchmark various models, using an LLM as a judge, we found that in the 30B params range, Qwen 3 tops it out, followed by Gemma 4. It feels like newer Qwens are optimized to perform agenti…

  109. Have a crap IGPU 64 gb AMD. THis model works pretty good.

  111. Unsloth Gemma 4 QAT MTP assistant models now available They're both available as q8_0 models named mtp-gemma-4-*.gguf on the root of the directory and in both q8 and larger quants within an MTP folder. https://huggingface.co/unsloth/gemma-…

  112. I never used to think it was all that important! But I’m using it for Svelte 5 and it ACTUALLY knows runes out of the box.

  113. Introducing Gemma 4 12B: a unified, encoder-free multimodal model Today, we are introducing Gemma 4 12B, our latest model designed to bring agentic multimodal intelligence directly to laptops. Bridging the gap between our edge-friendly E4B…

  114. I had a huge LLM server, and now I have a tiny one! I had a Jetson Orin NX gathering dust from a long dead robotics project, from back in the Llama-7B days.

  115. I'm just getting started using local LLMs for code. I'm not interested vibe coding, but I am hoping to increase my productivity in the publish or perish world of academia.

  116. Hi everyone, I am comparing the standard (non-QAT) iq4_xs and q3_k_m quants with this QAT q4_k_xl model. (All of them are Unsloth versions)(gemma-4-26B-A4B-it-GGUF via lmstudio).

  117. I'm trying to find out if anyone has done any benchmarking comparing the Gemma 4 4-bit QAT models (via Unsloth) against standard 8-bit non-QAT quants. I know QAT is supposed to retain a ton of accuracy compared to the baseline BF16, but I'…

  118. Hopefully this isn't too low effort of a post. I just finished the benchmarks and I figured I'd post them online because they certainly were insightful for me.

  119. Original post: https://www.reddit.com/r/LocalLLaMA/comments/1txwff3/comment/oq1e0jt/?context=3 TL;DR: Migrated to WSL2 to test Linux (several people suggested it). Embedded MTP on the UD model: 25.8 tok/s.

  120. Did anyone manage to launch that in LMStudio? I am on the most recent update with the most recent llama.cpp available in LMStudio.

  121. Hi all Been loving the QAT models but honestly what is up with the assistant models, any ggufs and ways to make em work with vanilla llamacpp and if this way of MTP is different than the one am17an developed for llamacpp. Followup question…

  122. the MLX version of the QAT 4bit is like 27gb but the none QAT version is 17gb and the regular 4bit MLX version is also 17gb… anyone know why?

  124. Ran a small, focused eval on three on-device models and the result was backwards from what I expected, so sharing the method and numbers. The task: tell the model "my dog is named Pablo," then add N turns of unrelated filler (shuffled gene…

  125. These last few weeks have been godsend for 24GB (and below) gpu poor peeps. Killer models released (Gemma 4 / Qwen 3.6) Free intelligence via QAT Bonus speed via MTP We're at the tipping point where GPU poor (24gb and below) people are act…

  127. I spent the last few days trying to get consistent tool calling out of the new Gemma 4 12b QAT model and had to give up. When the model actually works, it works great, but for my specific use case and workflows it is just not for me.

  128. Not talking about 31b. In terms of creative tasks, writing, chatting, not necessarily coding but can still be included, Does Gemma 12b outperform in any way?

  129. I wanted to try new QATs and opened two collections on HF (which HF found for me): https://huggingface.co/collections/google/gemma-4-qat-q4-0 https://huggingface.co/collections/unsloth/gemma-4-qat One strange thing caught my attention, for…

  132. when running by using transformers it runs by using vllm some weird error come up plese can any body share the command of running it on vllm ?

  133. How are you using it? Quantized?

  134. Running into something annoying with llama-server in router mode (`--models-preset`) and I can't tell if I'm missing a flag or if this is just how it works. My rig is 2x 3090, 2x 4060 Ti (one's unplugged at the moment, riser got repurposed…

  135. Gemma 4 E4B 6bit is now the local model of my choice and loaded 24/7 on my Mac (using @lmstudio), replacing Qwen3, 3.5 4B after ~9 months of usage What an insane model, congrats @GoogleDeepMind 🤠 The new setup replaces his nine-month daily…

  136. I am using llama.cpp version b9549 with this arguments as recommended: llama-server --temp 1.0 --top-p 0.95 --top-k 64 -hf ... Here is what I got on chessboard svg test https://www.reddit.com/r/LocalLLaMA/comments/1t53dhp/quality_compariso…

  137. Tried Gemma, Qwen and a few others. Need vision and larger context windows for an application I am working on.

  138. Hate to ask a simple question, but I’ve looked around and I see plenty of smaller models that *can* tool call, but none of them seem to do so appropriately or agentically. Referring to this.

  139. I’m trying to run: unsloth/gemma-4-31B-it-qat-GGUF gemma-4-31B-it-qat-UD-Q4_K_XL.gguf on an RTX 5090 32GB using llama.cpp Gemma 4 MTP PR branch. Main model loads.

  140. I just came across the following post, where a user found some confusing divergence results between Q4 quants of the original and QAT models with a Q8/unquantized reference of the original model. https://www.reddit.com/r/LocalLLaMA/comment…

  141. I've been running LLMs on my old potato i5-8500 with 32GB of RAM and *no GPU* for awhile now, running up to 12B dense models which run slow but perfectly useable. But this Gemma-4-26B-A4B simply flies on this CPU - only machine using Kobol…

  142. Just kidding. Are there any distills that actually improve a model's quality?

  143. I'll be upfront: I vibe-benched and vibe-reported this with Claude Sonnet 4.6, but I reviewed and edited everything before posting (too lazy to take out all the AI EM dash —), so hopefully nobody considers this AI slop. And more importantl…

  144. Title: Gemma 4 QAT MTP assistant heads now public on HuggingFace + PARALLEL=2 crash fix + 12B 2-slot bench (Strix Halo / Vulkan) Three things in one update: the converted QAT-matched draft heads are now uploaded for anyone to use, we found…

  145. First we never saw an upgraded Air model after 4.5. Then GLM 4.7 Turbo was great, but quickly surpassed for coding.

  146. Google just released the QAT (Quantization-Aware Training) variant of their Gemma 4 models, including 12B, so it was only natural for me to benchmark it on my 12GB GPU since it fits entirely in VRAM. I was pleasantly surprised of the resul…

  147. Now someone needs to quantize them to 4bit, also I have intentionally kept the divergence and refusal different from original Gemma 4 heretic collection, so you can even try these as alternative to original model.

  148. Table from https://unsloth.ai/docs/models/gemma-4/qat#qat-analysis I heard that MoE models are usually more susceptible to quantization error, but what happened with the 12B? I thought lower-parameter models usually quantized worse and yet…

  149. I’m a web developer doing mostly coding, but also project management, requirements analysis, testing, etc. I recently started experimenting with local LLMs, mostly because agentic stuff finally made them feel useful.

  150. Gemma 4 QAT Q4_0 Bench on Strix Halo These are Google's official Gemma 4 QAT Q4_0 GGUF models, served locally through llama.cpp Vulkan/RADV on a Strix Halo APU. QAT means quantization-aware training.

  151. While everyone's obsessing over giant cloud-based AI models, a quiet revolution is happening in local AI. We're seeing the emergence of extremely token-efficient, super-small system prompts, and modular agents designed specifically for loc…

  152. I picked models I consider local (usable on 3×3090), so there are no 300B models, and you should probably skip 200B models too (but MiniMax and Step are pretty fast in Q3) Gemma-4 12B is still missing

  153. This subreddit is where I learned the most about using Local LLMs. I've been on this journey for 4 months now, and I'm already using Local LLMs in very complex pipelines.

  154. What's with the switch guys? now imagine if google gonna drop 128B model or a MoE version (I bet those Qwen lovers will forget Qwen even existed).

  155. They started uploading to Gemma 4 MTP QAT but forgot to upload 12B quants to the Gemma 4 QAT 😭.

  156. I’ve been doing lots of testing back and forth with this 7900xtx. All of my workloads were relying on qwen3.6 models, which are amazing fwiw, but I wanted some diversity in thought.

  157. First time hearing it. I also heard about the gemma 4 qat quants and if any one of them is good for 4gb vram and 16gb ram.

  158. Posting to share my results with others, I think the big bottom line is MTP acceptance rates offering a huge speedup, during coding tasks it's over 90% acceptance! Haven't hit my soft goal results or llm as judge benchmarks yet to compare…

  159. could not extract summary

  160. This is a PSA for people like me who tried it and hit the wall with tool calls failing left and right, so much so that harnesses like OpenCode just didn't work: There is a fix for that. You need to pass a better chat template file, which i…

  161. Apples® to Apples®: MLX vs llama.cpp for Gemma 4 12B on an M1 16GB A matched-quant MLX-vs-raw-llama.cpp benchmark for Gemma 4 12B on one M1 16GB - decode is a tie, both pinned at the bandwidth wall. The cost that differs is startup and CPU…

  162. Release Blog Post: Gemma 4 with quantization-aware training HuggingFace for mobile: Gemma 4 QAT Mobile - a google Collection HuggingFace for Q4_0: Gemma 4 QAT Q4_0 - a google Collection

  163. Gemma 4 QAT models: Optimizing model compression for mobile and laptop efficiency Since releasing Gemma 4 two months ago, we've been continuously working to expand its capabilities. First, we introduced Multi-Token Prediction (MTP) to acce…

  164. Their collection: https://huggingface.co/collections/unsloth/gemma-4-qat And their guide, always a very interesting read: https://unsloth.ai/docs/models/gemma-4/qat

  165. Hi guys. I have been working on Hitoku Draft, an open-source, voice-first AI assistant that runs entirely locally.

  166. Following the announcement in our launch blog, we are releasing Gemma 4 12B, a dense multimodal model with a unified, encoder-free architecture. Gemma 4 12B introduces several milestones for local AI: Traditional multimodal models rely on…

  167. The generative AI boom has driven the cost of memory into the stratosphere, and Google is a key part of that trend. So it’s only fitting that Google should offer some less RAM-hungry local AI models.

  168. gemma-4-12B-it-GGUF Recommended way to run this model: llama-server -hf ggml-org/gemma-4-12B-it-GGUF Then, access http://localhost:8080

  169. 2026-05-05 Gemma 4 26B on consumer-grade 5070Ti GPU A week running Google's Gemma 4 26B as my daily local agent on a single RTX 5070 Ti. No API calls, no cloud, no rate limits.

  170. I made my first macOS utility app that ships with a bundled Gemma 4 model, specifically the Gemma E4B one. It made my app DMG have 5.3 GB in size, but I think it is a small size for the power that this free local model can provide.

  171. Discover how developers at the Google Tunix Hackathon trained small Gemma models to reason under a limited compute budget. Learn the winning, open-source post-training recipes—combining SFT, GRPO, and SimPO—to build your own structured rea…

  172. Hey HN, We believe we have the easiest onboarding from signup to being able to spin up coding agents in slack like Stripe, Ramp & Coinbase. Demo of the onboarding: https://www.tella.tv/video/connecting-cord-to-slack-1-19ep Every signup get…

  173. Fun local LLM comparisons with Gemma, Granite, and Qwen Ekorbia v0.2 features a comparison-chat mode that runs 2-3 local models against the same prompt in parallel. Here are a few fun prompts running across Gemma 4 (e2b), IBM Granite 4.1 (…

  174. Hoping the community can guide me on this one. I'm on the fence about the following purchase: Refurbished 16-inch MacBook Pro Apple M4 Max Chip with 16‑Core CPU and 40‑Core GPU, 64gb ram for $3,479.00 vs The new 16-inch MacBook Pro Apple M…

  175. Provided in both Safetensors and GGUFs. Safetensors, llmfan46/Gemma-4-Harmonia-31B-it-uncensored-heretic: https://huggingface.co/llmfan46/Gemma-4-Harmonia-31B-uncensored-heretic GGUFs, llmfan46/Gemma-4-Harmonia-31B-it-uncensored-heretic-GG…

  176. Okay fun time I got access to two Nvlinked A100s for some research project I benchmarked my work against the Gemma 4 31b-it available through Google, but my dataset is rather massive, so I need to run it on the "local" resources. Basically…

  177. Howdy everyone! Quick disclosure: I work on this - it's a project my studio created called the Null Epoch.

  178. Has anyone got it to work? I tried it with dense models (eg qwen 27b, gemma 31b, mistral 128b) since that's where I need it most, but it always core dumps.

  179. Been experimenting with an idea — what if your AI assistant actually remembered everything you did on your computer? Not stateless chats, but real persistent context.

  180. Built a local ReAct-style calculator agent with 6 tools: add subtract multiply divide modulo etc. The setup is: orchestrator agent dynamic tool selection ReAct loop tools exposed as functions Problem: Even when the user asks multi-step ari…

  181. It started with I just want to make a chat app like roleplay with characters but Gemma 4 26B A4B Q4_KM doesn't have info some old character so I crawl back to those online services as those model is much bigger parameter and quite update i…

  182. I seen this one mentioned but it was a source from about 14 months ago. In the age of the Qwen 3.6 and Gemma 4- is there still a use for QwQ 32B?

  183. I've fine-tuned Qwen 3.5 0.8B on the dataset provided by Pangram with their EditLens paper. It's available via a Chrome extension; you can just click selected text and it's going to give you the probability distribution of how likely it is…

  184. As per the title Such as Gemma 4 31B Q4 K S vs Gemma 4 26B A4B Q8 Or Qwen 3.6 27B Q4 K M vs Qwen 3.6 35B A3B Q6 K Etc At what point is it worth switching? My use case is mostly creative writing.

  185. Currently experimenting with building a React-style looping agent system using small LLMs like Qwen 3.5 9B and Gemma 4 (E2B), and I wanted to ask if anyone here has worked on something similar. Current setup: Using LangGraph Around 5 tools…

  186. Gemma 4: A new, budget-focused model in Posit AI Gemma 4 is now available in Posit Assistant via the Posit AI provider. It's priced at a tenth of the price of Claude Sonnet 4.6 and less than a third of the price of our current cheapest off…

  187. I’ve tried openclaw locally for about a month. Hardware: M5 Pro w/48 gb ram.

  188. i want the best installation that fit my use and my low-compute H.W , i want to run small to above small llm like "qwen" 2b ,4b and 27b , and "gemma" 31B. rely completely on only old CPU 4th.gen i7 with that few 32gb 'slow' ddr3.

  189. Wanted to share a result I didn't expect to work. Running google/gemma-4-e2b locally through LM Studio, exposed via OpenAI-compatible endpoint, called from a Spring Boot app using Spring AI's ChatClient abstraction.

  190. running gemma e2b via llama-server for continuous background tasks on a 1650 4gb. works great initially but after maybe 30-40 calls the outputs start getting noticeably worse — shorter responses, missing fields in json output, sometimes ju…

  191. The only thread was 2 months ago, when the model had just dropped. Since then, more versions from different authors have appeared, and users have had time to test them.

  192. Everyone remembers that sneaky download of Gemini Nano earlier this month? and if you talk to it, it will happily tell you it’s a Gemma.

  193. I tried mudler's apex quant for gemma4 26b a4b and it was amazing! I got 38tps at 90.000 context with no loop and suprisingly no quality degradation.

  194. https://huggingface.co/stevelikesrhino/gemma-4-31B-it-nvfp4-GGUF/blob/main/gemma4-improved.jinja Yall are more than welcome to try it out and provide feedback. In my own testing in Pi-coding-agent I no longer have the "forgot to close thin…

  195. Ever since the latest Gemma releases, there is literally zero reason to use Qwen. Better architecture, cleaner code output, and it doesn't get stuck in weird multi-turn reasoning loops.

  196. When I previously posted the uncensored version of the 31B version of the MeroMero finetune, quite a few people asked for the 26B-A4B version, I wasn't so keen on it because I considered the 31B to be the better version, but I understand t…

  197. BeeLlama v0.2.0 is here! Not quite a pegasus, but close enough.

  198. I've spent the last year building Charm, a native macOS menu bar app that corrects spelling, fixes grammar, and predicts your next word. Three features: - Spells: NSSpellChecker plus a local LLM for context-aware corrections (catches "defi…

  199. Think Local started with a simple idea: AI should work for you, not collect from you. So I built an app that lets you run modern AI models completely on-device - privately and fully offline.

  200. Hi everyone, I’m looking for advice on local AI setups. My goal is to have a local AI generate text documentation from my one-hour therapy sessions.

  201. Edit: does this happen every time a newbie tries to post here. Getting roasted despite having valid results?

  202. new model from LatitudeGames - Gemma 31B finetune https://huggingface.co/LatitudeGames/Equinox-31B-GGUF Equinox draws its name from the balance between extremes. Trained on a balanced blend of Wayfarer 2's unforgiving dark adventures and H…

  203. Hi all this is to inform you all that many frontline models like GPT, sonnet opus and or Gemma even are at stage of collapsing as they have frequently started drifting and running away from provided work either stretching that work too lon…

  204. The other day I posted about playing one night werewolf on my custom made UI via tool calls. Since then I’ve played a few games and improved the prompts.

  205. Hello, I built myself a news feed which scores and summarizes research papers along with relevant AI news from Huggjngface, Reddit, hacker news etc. I used Claude code to build the whole thing.

  206. Found this ACM paper on prompt injection and jailbreak attacks against open-source LLMs. The authors tested 10 open-source models across 94 prompt injection and 73 jailbreak scenarios, including Phi, Mistral, DeepSeek-R1, Llama 3.2, Qwen,…

  207. Asked new Google Model to build SuperMario. Compared with Local Gemma4.

  208. I am running Gemma 4 31B for a project using LlamaCPP. There is no integrated main model + MTP drafter GGUF.

  209. Some people say they’d never go under Q8, and others say they find Q3 acceptable! What’s your take?

  210. Google AI Edge Gallery ✨ Explore, Experience, and Evaluate the Future of On-Device Generative AI with Google AI Edge. AI Edge Gallery is the premier destination for running the world's most powerful open-source Large Language Models (LLMs)…

  211. I use cotpyist and typeWhisper. Both of them use local AI models to enhance my writing.

  212. currently in order to use MTP one needs to enable it in the starting argument of llama server. --spec-type draft-mtp --spec-draft-n-max 2 But then other models that do not use MTP currently like Gemma or basically all other models fail to…

  213. Provided in both Safetensors and GGUFs. Safetensors: llmfan46/Gemma-4-Gembrain-31B-it-uncensored-heretic: https://huggingface.co/llmfan46/Gemma-4-Gembrain-31B-it-uncensored-heretic GGUFs: llmfan46/Gemma-4-Gembrain-31B-it-uncensored-heretic…

  214. Hey all, While nvfp4 still seems to be a work in progress, the latest version of vllm 0.21 finally has mtp working for gemma. With all the talk of qwen being badass I thought I would revisit gemma.

  215. I have a build with 2 x MI50 32GBs and 64 gigs of DDR4 (bought before rampocolypse for ~630 USD total, I’m not rich) and I’m not gonna upgrade it for a long while. Are there any good MOE models that are around 60B in parameters so I can ma…

  216. could not extract summary

  217. Not deeply technically fluent but have ran few models locally before, around the time before gemma 4 dropped. I tried some low quant of qwen 2.5 coder and after some tinkering I got it to run but it was just so slow, obviously.

  218. I really benefit a lot from having claude code act as a personal assistant - it reminds me of things I need to do, helps me focus on what matters, and keeps me accountable on making sure I don't let important things slip But I am well awar…

  219. I can't decide that Qwen 3.6 35b q4 (130k context) or Gemma 4 26b q4 (95k context) is better for C# coding with 24GB VRAM. Please share your experiences!

  220. Hello, I'm currently using Ollama / lm studio for things like code inference and proof reading emails, etc. Definitely not experienced in this space but looking to grow.

  221. I kept seeing inference-speed claims for these models and wanting an apples-to-apples comparison on the hardware I actually have. So I built a harness and a public page that dumps every run as YAML.

  222. Provided in both Safetensors and GGUFs. llmfan46/gemma-4-Ortenzya-The-Creative-Wordsmith-31B-it-uncensored-heretic: https://huggingface.co/llmfan46/gemma-4-Ortenzya-The-Creative-Wordsmith-31B-it-uncensored-heretic llmfan46/gemma-4-Ortenzya…

  223. Recent Developments in LLM Architectures: KV Sharing, mHC, and Compressed Attention From Gemma 4 to DeepSeek V4, How New Open-Weight LLMs Are Reducing Long-Context Costs After a short family break, I am excited to be back and catching up o…

  224. Man, AI agents getting pretty crazy these days. :) (local, I just decided to try to get an orchestrator in there, when Qwen and Gemma aren't up to it.)

  225. It is suppose to be 2-4x faster but i am only getting 6TK/s on Gemma4-31B . What am i doing wrong?

  226. I am curious if anyone here has tried doing this, I did a bit of digging and it seems like it would be easier to do then I first thought and would like to ask ask for correction if my assumptions are wrong. Here is how I would go about it:…

  227. Okay so i've been stalking this sub for some time and i run the occasional small 2-8b model on my laptop (not the best) for fun but say my role at a company is to set up a local LLM since we obviously don't want confidential data going to…

  228. I don't think alibaba officially stated about "no qwen3.6 smaller models", and according to the patterns, she should ave been released it in the first week of may, but I think they delayed a little bit to catch the spotlight from Google I/…

  229. Sparky runs entirely on the Jetson. Gemma 4 E4B at Q4_K_M via llama.cpp with q8_0 KV cache and flash attention.

  230. Hi, has anybody succeeded in running llama.cpp with Gemma 31b dense and Gemma e4b as draft model, and simultaneously inhibit the voice recognition feature? Is it even (theoretically) possible?

  231. Hi r/LocalLLaMA - I've been paying close attention to the edge AI ecosystem because it's an area where i see huge potential and where I truly believe AI will become more useful for day to day tasks. Around the gemma 4 release I was already…

  232. I spend most of my day writing prompts to Claude. Read a study recently that said people speak ~3x faster than they type, which lands differently when "writing" is basically your whole workflow.

  233. I'm the founder behind Hedy, an AI meeting app. I'm a huge supporter of Local AI, and we've been working on making it "consumer friendly".

  234. Many local models have a problem (that raised due to excessive RHLF training): They mostly think that everything that is beyond their knowledge cutoff date would be "fictional" or "satirical". To be fair: Even the Gemini API without web ac…

  235. I got Qwen 3.6 35B-A3B and Gemma 4 26B-A4B running on a $200 secondhand machine (i7-6700 / GTX 1080 / 32 GB RAM) using llama.cpp (the TurboQuant/RotorQuant KV cache quantisation allows 128k context within the 8 GB VRAM). Results (Q4_K_M mo…

  236. I can set my context length as high as 64k and the vram usage is not even remotely close to the maximum utilisation. My TPS is also 40+.

  237. I have been curious to see how small LLMs like Gemma-4-E2B-it run on a flagship smartphone (S25+ with Snapdragon 8 Elite) in terms of prompt processing and token generation. I have created a script that uses llama-cli and I achieve 48 tps…

  238. Hi, i have a 5090 and i was tyoing around with hermes-agent. To utilize 128K i thought about switching from LM Studio to llama-cpp (the turboquant fork) expecting better tok/s and also saving some VRAM from context quantization.

  239. Between a solid model from Qwen or Gemma 4, when translating a text, does "thinking mode" significantly boost the quality of the translation, or is the difference negligible?

  240. Z-Lab did some good work with speeding up output, while Luce managed to use smaller models of the same family to accelerate prefill... Since Heretic and other "smart ablation" tools can decensor a model, would they work with these multi-mo…

  241. We'll be getting those features(check bottom link) on mainline soon or later anyway. But for now this fork could be useful to see the full potential of our poor GPUs(and also big, large GPUs).

  242. Watched All About AI's 100% local Fireship-style video automation experiment over the weekend (link in comments). A few things worth flagging if you're trying the same stack.

  243. My partner uses Duolingo for learning and practicing languages, but has been getting increasingly sick of it. I decided to experiment with whether local models would be good for creating and grading language exercises.

  244. My setup: Windows 10/11 i7 12700K | RTX 3090 TI | 96GB RAM Local server: LM Studio Models: Qwen 3.5/3.6 27B|35B Q5 UD K XL + Gemma 4 31B| 26B Q4 UD K XL Up until this point, I've only used sota models for coding. When Qwen 3.5 dropped, it…

  245. I would like to dedicate a budget of about 500 euros to upgrade my workstation and run inference on the qwen 3.6 27b and gemma 4 31b models. I currently have an RTX 5060Ti 16GB.

  246. Benchmarked Gemma 4 MTP and z-lab's DFlash on a single H100 80GB using vLLM and NVIDIA's SPEED-Bench qualitative dataset. Setup: Hardware: 1x H100 80GB Runtime: vLLM Dataset: SPEED-Bench qualitative Prompts: 880 total, 80 prompts across ea…

  247. Hello everyone, I developed the snn architecture from scratch based on the human brain. I had several successful launches of training spike models from scratch and I also had an idea: what would happen if I took the gemma 4 model and conve…

  248. Yes, for material that is an hour long, there is no getting around tools like Whisper - or something even better. However, for transcribing short snippets, Gemma works very quickly and reliably- even in foreign languages.

  249. Question in title. Would be awesome to have this on macs, especially q8 or whatever the minimal-loss quant is, since macs can have lots of ram.

  250. could not extract summary

  251. Hi, I recently tried to get llama.cpp with SYCL running on an Arrow Lake system but gave up halfway through since Vulkan is just way easier to set up. But, the pp/tg I'm getting on Vulkan w/ Arc 130T is disgustingly bad - 100 tokens/s for…

  252. Per title. Been trying to load Gemma E2b as draft model for 26b as target using LM Studio's UI but it can't seem to recognise what's already been downloaded.

  253. Turboderp has a been on an absolute tear recently, in the endless battle to cram new llamas into smaller, faster boxes. We started off last month with the release of gemma 4 support, and continued with improved caching efficiency.

  254. Per my experience, Opus 4.7 is so slow, Sonnet 4.6 is ok. I am also using local models wondering if Claude is already leveraging drafters/assistant AIs and despite that so slow or not?

  255. I wrote up this little python app to cycle through a bunch of prompts like this: Single HTML file using three.js from CDN. A central rotating MeshNormalMaterial torus knot.

  256. Hi, I’m having a very hard time right now. I used to use ChatGPT 4o and 5.1 Thinking for helping me write my story, and I was very happy with them.

  257. Due to old GPU I run on CPU and came to appreciate value of MoE. I know of MoE for Qwen 3.6 and Gemma-4, which are <40B.

  258. Gemma Chat Vibe code without the internet. A local coding agent powered by Google's Gemma 4 — runs entirely on your Mac via Apple's MLX framework.

  259. I had been looking to try <500M parameter language models but you wouldn't find an API to try them anywhere, so I built this cloudflare hosted static website that hosts weights and built an inference runtime for these models that uses WebG…

  260. So I'm brand new to this scene but I'm using Claude to help me fine tune a model for a startup idea I have in the Healthcare space. I have been working with the 27-35B parameter mdoels (Qwen3.6, Gemma 4) and the couple of 120B+ models (Qwe…

  261. Grafting a Speech Head onto Gemma 4 E4B For a Discord buddy, the tempting model shape is small, fast, and multimodal. It should hear the call, see the game, read the chat, and respond quickly enough that the moment is still alive.

  262. I am trying out some tools that are from the openwebui community that I have directed towards my LM Studio server instance. It seems really hit or miss on most of the tools being called by the LLM or not.

  263. I have setup a workflow to process website translations with Gemma 4, I just host it on LM Studio, and a custom Python wrapper iterates through and runs overnight. My question is..

  264. Trying to figure out which is better for LLM. Mainly Gemma 4.

  265. We ran GPT-5.4 vs Gemma 3 27B on 2 prompts. One open-source model won.

  266. Past few days, its all been about MTPs. Somehow people missed out the fact that Z lab released the Dflash for Gemma4 26B a couple of days ago.

  267. I ran a benchmark to see how much DFlash speculative decoding actually helps in vLLM. Setup: GPU: RTX 5090, 32GB VRAM vLLM: 0.19.2rc1 Main model: cyankiwi/gemma-4-26B-A4B-it-AWQ-4bit Draft model: z-lab/gemma-4-26B-A4B-it-DFlash Workload: r…

  268. Private offline AI chat for macOS. Free for 14 days, then €29.99 once.

  269. Is there a 7B parameter model in your life struggling to understand sarcasm? A tiny 1.5B that can't afford one more epoch?

  270. Anthropic has released new research to show what an LLM is thinking when generating a next token using NLA or "Natural Language Autoencoders", the NLAs are a pair to LLMs that can translate internal thoughts of LLM for any specific token.…

  271. The script to graft MTP tensors requires a full GGUF model file. I felt that was a bit hefty, so I asked local Gemma to write something to just extract what's required.

  272. Hey folks, I posted here a few months back about how I was basically working for Claude -- pasting the same emails, re-explaining the same backstory, being its memory across every chat. Today I'm launching Contextify.

  273. Hey everyone! I’ve just uploaded a GGUF version of nvidia/Gemma-4-26B-A4B-NVFP4.

  274. In my line of work, PDF documents tend to be combinations of text, math formulas, tables and images. llama.cpp added support for PDFs a few months ago, but I believe it treats PDFs either as text (discarding everything else), or as images.

  275. With every new model release there's the "better than Opus 6.13" guys vs the "this is so bad, why did they even release it" camp and I'm always wondering which one is using it wrong. So I did a little test with 2 related prompts, 3 models…

  276. Hi guys 👋 When I started my adventure with Qwen 3.6 27B I felt wow.... Now when I connect it with Gemma 4 I'm feeling more wow...

  277. I have a 4060 (8GB Vram) and 16GB of ram wondering which models could fit in my setup for coding, the new Qwen 3.6 and Gemma 4 MoE models look good but might not fit, wondering about your experiences

  278. Just saw this and wanted to ask the obligatory GGUF when?

  279. Google launched its Gemma 4 open models this spring, promising a new level of power and performance for local AI. Google’s take on edge AI could be getting even faster already with the release of Multi-Token Prediction (MTP) drafters for G…

  280. Hey everyone, ​I'm running into a weird issue and hoping someone here might have a fix or some troubleshooting ideas. I'm currently trying to run the new Gemma 4 31b-it model using vLLM (v0.20.0-cu130) deployed via Helm chart (https://gith…

  281. Very quick initial test of Gemma 4 new MTP model via Ollama (llama.cpp doesnt support yet) https://blog.google/innovation-and-ai/technology/developers-tools/multi-token-prediction-gemma-4/ Running in Open Webui to view token/s output and I…

  282. Just tested Gemma 4 31B with the new official MTP Drafter on my H100 today and compared the approach with DFlash to help you decide which one to use. Without drafter: 13.7 tok/s.

  283. The idea of offload-mcp is simple: instead of running hardware-hungry local models for routine work, let Claude offload that work to FREE model APIs and SAVE tokens. I’m using Gemma via the Google GenAI API because I like it in my processi…

  284. Let's test the Gemma 4 MTP implementation using the HuggingFace Transformers library and the new drafter model by Google. We'll load both models and test on a couple of prompts with and without the MTP support https://www.youtube.com/live/…

  285. Hey everyone, I’ve been thinking about a project idea and I’d love to get your feedback. The idea is to take a 1TB SSD and turn it into a fully portable AI system.

  286. Not affiliated with Kaitchup, but a fan of their testing. I was looking forward to this article...

  287. Libraries llama-cpp-python How to use unsloth/gemma-4-E4B-it-GGUF with llama-cpp-python: !pip install llama-cpp-python from llama_cpp import Llama llm = Llama.from_pretrained( repo_id="unsloth/gemma-4-E4B-it-GGUF", filename="gemma-4-E4B-it…

  288. Don’t miss what’s happening People on X are the first to know. Log in Sign up Post Conversation Software Mansion @swmansion On-device @googlegemma in React Native with react-native-executorch Coming to the library very very soon!

  289. When dealing with untrusted outside input, I think you should handle it based on the situation. If you're processing structured data files, it's better to use tools to isolate and handle them.

  290. Whenever I write here that I use gemma 31B I get answers that qwen 27B is better. I switched in the pi from gemma 31B Q5 to qwen 27B Q8 and generally I manage to code, document and run tests but somewhere after exceeding 100k context qwen…

  291. offload-mcp MCP server for offloading routine coding-assistant work to a cheaper model. The default model chain uses Gemma because the models are useful, open, and fun to experiment with.

  292. I have a GPU server (4 × RTX3090s) that I've been using for research and PoC in the past 2 years. Mostly running vLLM for Qwen, GPT-OSS, and Gemma.

  293. Increase your CPU Thread Pool Size to your processor's max. In LM Studio, the max is 10.

  294. Hey everyone, I built a tool that creates movie recap videos automatically using local models. The problem: making recap videos takes forever.

  295. Talkie-1930-13b-it and Gemma 4 31b in the same chat. Talkie is a 13B vintage language model from 1930.

  296. Chat Template was fixed a few days ago choose your fav dealer: https://huggingface.co/bartowski/google_gemma-4-31B-it-GGUF https://huggingface.co/bartowski/google_gemma-4-26B-A4B-it-GGUF https://huggingface.co/bartowski/google_gemma-4-E4B-…

  297. I saw someone on this forum demonstrate using gemma 4 - live streaming audio and video from his webcam to it asking it what it was seeing. It was pretty great but I cant find that post anymore and I can't find a good repo on github where I…

  298. Hello, posting again on this since my last post was removed. I am working on an AI agent solution to help me with my multiple daily tasks for different business activities; a few rental properties, a manufacturer trying to enter the Mexico…

  299. I'm confused about what to get. I don't want to get something super expensive, but would like to have something that's "good enough" for coding etc.

  300. Been running Gemma 4 E2B locally on my OnePlus CE 5 (8GB RAM) for a few months. Chat quality is fine for the size.

  301. I am quite curious as I tried Gemma 4 31B, Qwen 3.6 27B, GLM 4.7 30B and some others in my native language (czech). Gemma performs "best" and considering the fact its "just" 18GB model - it actually blows my mind how well it can respond in…

  302. llmconfig Local Large Model Config — manage local inference with llama.cpp, stable-diffusion.cpp, and whisper.cpp from a single YAML file and a single CLI. llmconfig up gemma # or just: llmc up gemma ✓ gemma is ready at http://127.0.0.1:80…

  303. Hey there people. So let's talk about GEMMA 4 per layer embeddings.

  304. I read a lot of "whengguf" type posts. I think we should sometimes stop and be grateful.

  305. I built hfviewer.com, a small tool for visually exploring Hugging Face model architectures. You can paste a Hugging Face URL and get an interactive visualization of the architecture, which can make it easier to understand how different mod…

  306. I've been experimenting with using Ollama to run Claude Code locally with models like Gemma 4, thinking I could avoid API costs. However, I quickly realised these models aren't really optimised for Claude Code's agentic workflows — they te…

  307. Hey guys, A couple of weeks ago, I asked this sub for the hardest Vision use cases you were dealing with to test the newly dropped Qwen 3.6 against Gemma 4. I finally finished running the gauntlet side-by-side locally on vLLM (FP8 quants)…

  308. Does anyone know of any use cases of Gemma Embedding 2? Or is it solely for search?

  309. It is pluggin made for ONLYOFFICE, much simpler than copy-paste from webui. PS.

  310. I would like to experiment but before investing loads of money, I do have a MacBook Pro with 32GB RAM, M2 Pro. Which model would maximize versatility given this hardware?

  311. So in response to the Great Token Reconning of 2026, I decided to try out Qwen 3.6 as a daily driver, and although it's only been about a day, I have to say I'm thoroughly impressed. I had to download the VSCode insiders edition and set up…

  312. https://huggingface.co/z-lab/gemma-4-31B-it-DFlash I guess we'll have to wait until this PR is merged before we can test it. https://github.com/ggml-org/llama.cpp/pull/22105

  313. We run Qwen3.6-27B-FP8 at AI Router Switzerland and hit two issues, so I wanted to share in case anyone else runs into them. FP8 KV cache produces silent garbage output with radix cache prefix hits (PR #24198 — ✅ approved) We were running…

  314. I experienced this with Q4 and Q3 versions of Qwen3.6-35B-A3B and Gemma-4-26B-A4B. It starts saying things which sound similar in thinking mode: I must do ....

  315. 1990s. I was the kid with the dial-up sound burned into my brain.

  316. True story, I got interested in AI after seeing it at work and wanted to run models locally. I started with an M3 Ultra 96GB, quickly learned it was not enough for what I wanted, and kept upgrading hardware (including refurbished Mac Studi…

  317. I'm testing running local LLMs on a gaming mini PC (AMD 7840HS, 32 GB RAM) paired with an eGPU (Radeon 9060XT with 16 GB VRAM). Since I'm not very familiar with using llama.cpp, I kept getting unsatisfactory results, but with the recent Ge…

  318. https://github.com/ggml-org/llama.cpp/blob/master/docs/backend/snapdragon/README.md I have an Oneplus 12 with Snapdragon 8 Gen 3. I followed the above README to cross-compile llama.cpp on Ubuntu and then copy to the Termux directory on the…

  319. Hello folks What is best code editor for local LLM deployment (LM Studio, llama.cpp)? I wish to test my LM studio + Qwen 3.6 27B and Gemma 4 31B with a legit local code editor.

  320. Gemma just crushed Qwen in a local LLM gamedev contest! Device: MacBook Pro M5 Max, 64GB RAM Qwen 3.6 27B: 32 tokens/sec · 18m 04s · 33,946 tokens.

  321. I can’t really run 70B models on my current setup, but I’m curious haha

  322. I've built a system where models like Llama 3, Qwen, and Gemma play Pokémon Showdown battles autonomously. Instead of simple prompt-response, they analyze the full battle state every turn (type matchups, HP, weather, field conditions, reve…

  323. I've got to the point where I need some help. I'm trying to run Qwen 3.6, and it will eventually fall into a loop where it's just outputting "/" symbols when it's "thinking".

  324. Bench 3 from my 18GB M3 Pro. Bench 2 was the 4B-class post where the comments were mostly right: I gave thinking models a fixed 1024-token cap, Qwen got kneecapped, Gemma E4B needed clearer active-param labeling, and the headline was partl…

  325. Have Qwen 3.6 27B and Qwen 3.6 35B basically made most of the older ~30B models irrelevant? They seem to beat stuff like Qwen coder 30B, GPT OSS 20B, Gemma models, especially for coding and agent workflows.

  326. Longtime lurker here, thought i should post my speeeeds... I have a RTX 4070S 12 GB Vram (+10% OC), AMD 9800x3D with 4x16 Gb DDR5 6000Mhz CL30.

  327. I see a big difference in agentic coding between gemma-4-31B-it-Q5_K_M and gemma-4-26B-A4B-it-UD-Q8_K_XL. The 26B model is much faster because of A4B and generally works well, but there is a big difference in thinking.

  328. Basically, I’m really into the idea of a fully offline setup. (Another way to say it: I’m a data hoarder.) For LLMs, I’m using uncensored models from both Western (Gemma, GPT-OSS) and Eastern ones (GLM 4.7 Flash, Qwen 35B).

  329. Some of the larger models (like Llama) weren't available on OpenRouter, so I had to work with what was there. Best small model: Gemma 4 26B For its size, I think it had the best output.

  330. I have an RTX 4060 8GB(+16GB RAM) laptop, and when asking Gemini or ChatGPT, they say the Gemma 4 Q4 K M is the best fit for my hardware with Context Length around 16k-32k. However, in practice, after loading even a higher quantization lik…

  331. Qwen3.5-122B-A10B at Q6_K is really good. Do you think we will see a larger MoE Gemma-4 or Qwen3.6 at some point?

  332. I’ve been building a local-first AI runtime that wraps local LLMs with a σ-gate — a measurement layer that decides ACCEPT, RETHINK, or ABSTAIN before an answer reaches you. The idea: local models should be able to say “I don’t know” instea…

  333. With the release of Gemma 4 models and a slew of open weight/source models subsequently, some of the workflows like drafting emails/ trivial coding tasks have become possible. I’m exploring the possibility of integrating some of the powerf…

  334. QVAC-Fabric Gemma 4 Architecture Patch Adds full Gemma 4 (gemma4) architecture support to QVAC-Fabric, Tether's llama.cpp fork. Base: QVAC-Fabric temp-upstream branch Target: All Gemma 4 variants (E2B, E4B, etc.

  335. Based on the recent TRiP source code by Carlo Valenti. Ported to Zig and headless Vulkan Compute shaders.

  336. I've been running local LLMs since Qwen 3.5 dropped and I was really impressed by what we could run on consumer hardware. Fast forward another two months and we have gotten a handful more gems such as Gemma 4 and Qwen 3.6, so I wanted to p…

  337. Whew! Big work today getting optimized llama.cpp MIPS R8000 kernel running on the SGI Power Challenge deskside from 1995 with Gemma 3 270M.

  338. Well or pretty close to it, they are excellent work horses. I run them in real work scenarios doing some of the work I used to do myself as an skilled expert in my field, billing 200$ an hour.

  339. Uhh I guess Gemma 4 is so much shittier that it hallucinated this event that happened in china in 1989? According to qwen, nothing of significance happened at Tiananmen square in 1989 - and based on all of the benchmarks of qwen, I believe…

  340. TLDR: tool parameters using the common JSON Schema pattern `anyOf: [$ref, null]` are rendered into the prompt as empty `type` fields. This strips the useful schema information before the model sees it.

  341. https://github.com/ggml-org/llama.cpp/pull/22196 And somehow we already got some GGUFs for it! https://huggingface.co/CISCai/gemma-4-31B-it-NVFP4-turbo-GGUF https://huggingface.co/stevelikesrhino/gemma-4-31B-it-nvfp4-GGUF (the below one is…

  342. another day with pi + gemma 26B

  343. https://huggingface.co/firstbober/gemma-3-270M-it-smol-thinker Here is an example of the output: ``` ==================== THINKING ==================== Here is the thinking process: This is a large community with a wide range of interests…

  344. Gemma-4 is great. On a MacBook M5, using lm-studio, the MLX versions (specifically looking at https://huggingface.co/lmstudio-community/gemma-4-26B-A4B-it-MLX-8bit) rock.

  345. I now use (mostly) Gemma 4 and Qwen 3.5 models *. And seems that all of them, after context grows a bit, after providing long output for me and getting a short prompt in response, are starting to process many new tokens as input and I have…

  346. TLDR: I've been running gemma4 e2b extensively on iOS with llama.cpp and found some interesting quirks and info you guys may like! These are specifics for the iPhone and what I've found worked across 20+ devices.

  347. Did some quick tests after building llama.cpp with ROCm 6.4.2 and latest Vulkan for my 6900 XT gemma4 E2B Q4_K ubatch ROCm pp512 Vulkan pp512 ROCm tg128 Vulkan tg128 32 1536.60 1423.49 151.92 174.59 64 1590.65 1930.60 151.41 173.76 128 265…

  348. The gemma 4 E4B and E2B models have built-in multimodal capabilities. However, as far as I am aware, llama.cpp does not have proper support for vision and audio inputs (specially audio) for these models as of now.

  349. I think gave it a fair shot over the past few weeks, forcing myself to use local models for non-work tech asks. I use Claude Code at my job so that's what I'm comparing to.

  350. MIT license and fully open source. MiMo-V2.5-Pro was just 3 points from Opus 4.7 max and the normal V2.5 is only a step behind SOTA.

  351. https://huggingface.co/mlx-community/Qwen3.6-35B-A3B-OptiQ-4bit https://huggingface.co/mlx-community/Qwen3.6-27B-OptiQ-4bit https://huggingface.co/mlx-community/gemma-4-26B-A4B-it-OptiQ-4bit https://huggingface.co/mlx-community/gemma-4-31B…

  352. I did give same prompt same document to 1660ti running Gemma 4 e2b q4 coz of the small vram and another to and igpu running Gemma 4 e4b q8 prefill rate before token generation was like 4-5 times faster with the 890m igpu then token generat…

  353. Tutorial from the Google guy, I use very similar setup (llama.cpp instead of lmstudio)

  354. Using Google's Gemma 4 E4B local AI model to Reverse Engineer a simple Crackme I was playing around with the new Gemma E4B open weights local model which Google released, and to my surprise I was seeing a great deal of success in using it…

  355. At this moment, the models such as Qwen 3.6 35b/27b crush the competition, yet I can't help, but notice this pattern. While the local RP scene is abundant with the Western model tunes: LLaMA, Mistral (all sizes), Nemo and more recently Gem…

  356. A practical configuration for a 32 GB M5 Mac that still needs to remain usable Running large language models locally has become surprisingly practical on Apple Silicon. With a modern Mac, Ollama, and a carefully quantized GGUF model, it is…

  357. I have a Macbook Air M3 with 24gb RAM. The other day, I wanted to try running an LLM locally for the first time ever.

  358. I tried with Qwen but it sucked, Gemma3/4 was better but not good enough. From Gemma: https://pastebin.com/raw/Qr5iMgYj Still looks like a bloody car accident though.

  359. I love using openrouter but I would also love a smaller model that can fit within 16gb of VRAM and 64b of ram, that can pack a punch for its size specifically in the creative writing section. Any good recommendations?

  360. Spent a week doing LoRA fine-tuning on Gemma 4 E2B (~5.1B total params, ~2B active in text decoder) for a narrow Python code-generation task. Bad outputs went from ~5% to 0% (greedy) and 1.5% (sampled) across 134 tests.

  361. So for my project I was using up until now either Gemini 3 / 2.5 Flash or Flash-lite. All my use cases are not agentic, simply LLM workflows for atomic tasks like extracting references from the law, classifying, adjusting titles to nominat…

  362. I’ve been tinkering with a small side project (just for fun) where I’m trying to extend llama-swap with a bridge from /chat/completions to the newer /responses API so I can run the latest Gemma and Qwen models together with Codex-style too…

  363. Pocket LLM v1.5.0🚀 New in this release: - 🎙️ Voice input - 🖼️ Image input with OCR, Gemma vision, and FastVLM support - 📷 Camera capture with retake, crop, and photo review - 🗂️ Previous chats side panel - 💾 Downloaded model deletion to sa…

  364. As a life-long Windows user (don't hate me, I was exposed to it at a young age) I was wondering how much (if any) performance I'm leaving on the table. So I did the sensible thing and run some benchmarks.

  365. 3090 (24G) + 32G DDR4 Currently running --mmproj mmproj-BF16.gguf --chat-template-kwargs '{"enable_thinking":true}' \ --flash-attn on \ --cache-type-k q4_0 \ --cache-type-v q4_0 \ -np 1 \ -c 160000 \ --jinja at 26B-A4B-it-UD-Q5_K_XL and ge…

  366. OpenAI, Gemma, Kimi K2, or run locally. Free via OpenRouter.

  367. Spent a week doing LoRA fine-tuning on Gemma 4 E2B (gemma-4-e2b-it, ~5.1B total params, ~2B active in the text decoder) for a narrow Python code-generation task. Setup: Model: Gemma 4 E2B, bf16, language_model only (vision + audio towers f…

  368. I made a simple app using openrouter, hoping to use the new small qwen models (the a3b moe and the 27b dense one), but they aren’t listed. Also, I swear some qwen3.6 models that were listed before are missing now.

  369. I wanted to figure out which of the newer small and mid-size models are actually worth running on a single H100, so I put 8 of them through a proper vLLM benchmark and recorded what came out. The setup was simple.

  370. When Qwen3.6-35B-A3B was released a week or so ago, I sort of expected an iterative improvement on the previous Qwen3.5 models. After all, those models were pretty decent as compared with the previous local models I had tried, and Qwen3.5…

  371. other companies are slowly going away from open weight, not releasing base models, delaying open weight distribution, not releasing top models (this one I think is fair, but still), and I also noticed they stopped publishing research (old…

  372. We have a chat system which we use haiku for because it is mostly about tool calling and summarisation of them. But we have many tools with pretty complex input schemas, and stuff like gemma didn't cut it, so we went with haiku.

  373. Gemma 4 and Qwen 3.6 with q8_0 and q4_0 KV cache: KL divergence results 4 models tested with q8_0 and q4_0 KV cache against full-precision baseline What this measures KV cache quantization stores the key-value cache in lower precision to s…

  374. Hey guys, I built a custom vLLM pipeline to run Gemma 4 (31B FP8) and Qwen 3.5 side-by-side locally to see how they actually perform in the wild with preprocessing of audio and images. But of course new model Qwen 3.6 27B came out just whe…

  375. I am currently working on a system to translate books/web novels. I got a working prototype, but now I am looking into optimizing it.

  376. Hey everyone, I run llamacpp precompiled with CUDA 12.4 on Windows 11 with a RTX 4090. With small models like gemma-4-E4B everything runs fine, but as soon as I run a bigger model like Qwen3.6-27B (IQ4_NL) or a medium sized model with larg…

  377. Hi guys, I’ve been running side-by-side experiments on Gemma 4 (31B FP8) and Qwen 3.5 Vision for the last few days using vLLM in Docker to see how they actually handle real-world images and video. A few things I found out: 1.

  378. Always been stuck with models that fit on my 16gb .... Going to have about a week for free with 4x rtx6000pro .

  379. Hi everyone, as a power user I hit Claude Code's usage cap too often I wanted to set up my own local model, however I only have RTX 5070 with 12 GB of VRAM so the only realistic option was Gemma 4 with effective 4B params. When I tried to…

  380. Gemma 4 VLA Demo on Jetson Orin Nano Super You speak → Parakeet STT → Gemma 4 → [Webcam if needed] → Kokoro TTS → Speaker Press SPACE to record, SPACE again to stop. This is a simple VLA: the model decides on its own whether to act based o…

  381. Both amounts are in euro. The AMD is actually 599 but it's sold by a shop, so I can get a VAT return as a company, while for the nvidia I'd have to go to the second hand market and I can't get VAT back, so at the end it's like a 495 vs 850…

  382. Hi LocalLLaMA, I created a post a few weeks ago, but this time this project has become more reliable and easier to use. This is a manga translator that can also be used to translate any image.

  383. I created this chart with recent open models from last 6 months. Few might be older than that possibly.

  384. Gemma 4 makes five quiet departures from the standard transformer recipe. QK-norm instead of 1/√d, partial RoPE on global layers, per-layer input gating, KV sharing across layers, and an MoE that sits alongside the MLP rather than replacin…

  385. I have run two tests on each LLM with OpenCode to check their basic readiness and convenience: - Create IndexNow CLI in Golang (Easy Task) and - Create Migration Map for a website following SiteStructure Strategy. (Complex Task) Tested Qwe…

  386. +30% avg accuracy lift on classification & extraction tasks vs. base Gemma ~7 days until your first auto-improvement run lands in production 0 lines of fine-tuning code you have to write, ever $0/retrain starting price.

  387. My old Samsung S10 was sitting in a drawer so I turned it into an always-on LLM endpoint. PocketPal is great for on-phone chat, but I wanted the phone itself to be an OpenAI-compatible endpoint for the rest of my network.

  388. I gave 9 local models the same flight combat sim prompt. The results broke a few of my assumptions about quant providers and parameter count.

  389. How do I run Gemma 4 e4b, extracted via adb from Google AI Edge Gallery on Android, whose image is in litertlm format and weighs 3.6 gigs, in a browser? I mean using web technologies?

  390. The difference is quite big: likes downloads last month finetunes Qwen3.5-27B 952 3,233,034 263 Qwen3.5-35B-A3B 1,397 3,977,637 87 Qwen3.6-35B-A3B 1,115 458,436 60 gemma-4-31B 323 343,895 13 gemma-4-26B-A4B 227 118,464 13

  391. Why does Gemma 4 e4b from Google AI Edge Gallery on Android weigh only 3.6 gigs, while the one from Unsloth (gemma-4-E4B-it-UD-Q2_K_XL.gguf) weighs 3.7, and for some reason the model image in litertlm format extracted via adb from Google A…

  392. I was building a dedicated-vision-model feature for an open-source browser agent and wanted to figure out which local model to actually recommend. Wrote a small probe that sends the same image + same system prompt + same params (temperatur…

  419. I have Gemma4-E2B working within home assistant as STT, and E2B seems fast and accurate for STT (maybe a bit better than Parakeet), however, it responds with the entire thought process: https://preview.redd.it/v8zhb5elltvg1.png?width=599&f…

  420. So the question I've seen posed many times in /r/singularity is if the Gemini models are actually that bad at coding compared to their benchmarks, or whether the harness used makes an absolutely gigantic difference in model performance. Gi…

  421. Hardware: Intel Core Ultra 7 258V, 32GB Unified Memory. Model: Qwen 3.6 35B A3B (Quant: Q3_K_S) via LM Studio.

  422. I have a personal eval harness: A repo with around 30k lines of code that has 37 intentional issues for LLMs to debug and address through an agentic setup (I use OpenCode) A subset of the harness also has the LLM extract key information fr…

  423. Just gave the new Qwen3.6-35B-A3B a spin. It’s a MoE model (35B total, ~3B active), but honestly the more interesting part is how much they’re pushing agent-style coding.

  424. I've tried a few different local models in the past (gemma 4 being the latest), but none of them felt as good as this. (Or maybe I just didn't give them a proper chance, you guys let me know).

  425. hello. i have the oportunity of buying Nvidia V100 with 32GB for about 915$ / 775 euro.

  426. Vercel Security Checkpoint | cle1::1776468758-lOAcIwtVVUa8cG9OLlcTtnlZwlvTxsBe

  427. Processing img 8ofni1q6dpvg1... Hello everyone, I’m building an offline RAG system for my company, we are trying to run an app that retrieves infromation from two manuals in an android tablet with the idea of an AI to provide precise answe…

  428. I have observed that many ai teams try to always use the best model to ensure quality. When a new model drops out, they are forced to pay for it, because their competitors will.

  429. I'm using the https://github.com/PrismML-Eng/llama.cpp fork for Bonsai, regular llama.cpp for Gemma. Without embedding parameters: Gemma 4 has 2.3B at 4.8 bpw (Q4_K_M) = 1104 MB Bonsai-8B has 6.95B at 1.125 bpw (Q1_0) = 782 MB (only 29% sm…

  430. The showcased examples were made using Gemma 4 31b. Any LLM with tool calling support should work.

  431. This spring is really hot since the localLLM giant, both Qwen and Gemma released major models. I'm really excited with those release and happy with their capability.

  432. I have 3090ti and i will add 3080ti to my system soon. With 3090ti only, i found it little bit slow to run gemma 4 26b 4q.

  433. could not extract summary

  434. Hi, I got new MB Pro 24GB/1TB. I've test Gemma 4 26B with ollama, 16k context.

  435. I was disappointed with Gemma 4 due to various bugs and in the end lackluster performance for the internet research/information synthesis type tasks I use local AI for. Even after every last fix and update of both mode quants and llama.cpp…

  436. If the LLM supports video—which most VLLMs nowadays do—then try the following prompt with the accompanying video: With the given video, which is about 16 seconds long, your task is to write JavaScript for an animation that faithfully repli…

  437. Ryzen AI MAX+ 395, Bosgame M5, 128GB LPDDR5x. Proxmox VE 9.1 LXC containers with GPU passthrough.

  438. Gemma 4 just dropped on Modular, Day Zero! Read More → Inference Products Shared Endpoints Access frontier models via an API Dedicated Endpoints Mission critical reliability Custom models Your model, peak performance Deployment Options Our…

  439. https://preview.redd.it/w6ssjgidjlvg1.png?width=2786&format=png&auto=webp&s=f52736d40580fe8a8ff74adbbb5be81f12fbcbfc So I was playing with Gemma 4 and was trying to figure out whether the model could determine its own training data cutoff…

  440. Writing a fine-tuning and deployment pipeline isn't as easy as it looks (Gemma 4 Version) Fine-tune and deploy Gemma 4 on Oxen.ai Google's Gemma 4 dropped in April 2026 with multimodal support (text, image, video, audio), a novel hybrid KV…

  441. I configured it to the best of my abilities, even at Q8. It fails to give the correct number of tools it supports on Claude Code and it fails the car wash test.

  442. I just launched ios app that uses Gemma 4 (E2B 4-bit via mlx-community) to rewrite oral transcripts into heirloom-quality paragraphs, 100% offline. What made this interesting technically: MLX Swift + MLXLLM in production (not a demo) — fir…

  443. Published: April 16, 2026 This is an encyclopedia of some core content from Biology and Health Sciences, Physical Sciences, and Technology. It contains 2,259 small entries of about a paragraph each.

  444. Hello, everybody! I'm building and hybrid database with Qdrant and Neo4j for a few personal projects.

  445. Google and Alibaba recently shipped Gemma 4 and Qwen3.5, so I wanted to see whether the new generations are actually better on my setup. My context is private local chat running on my own hardware, a Mac mini M4 Pro.

  446. I’m trying to reduce my reliance on Claude. I have a 5090/128GB RAM.

  447. I have been nothing but impressed by the quality of Gemma 4 since release. In general conversation it's adaptable to different personas.

  448. Hey everyone, I just shipped an iOS app that runs local AI models. Current has 12 models: Gemma 4, Llama 3.3, Qwen3, DeepSeek R1 Distill, Phi-4, etc.

  449. question, I'm using it for text translations and on each large prompt (20K+) it stops with a remark 'now I'm going to put that to the file' or some other operation I have asked in the prompt for but it did nothing, just stopped. I'm runnin…

  450. Im searching for build version of LiteRT LM framework can use and utilize the NPU of the RKNN 3588. It would be great since I can run gemma 4 e2b model using this framework on the machine, because I wont have to migrate my codebase from li…

  451. I’ve been running various AI harnesses like OpenClaw, ForgeCode, ClaudeCode, etc. Most of these are running via OpenRouter or Minimax (credits/subscription model).

  452. Dataset: MMLU subset (DEV+TEST) Llamacpp setting: 3 params only ctx 8192 , seed 42 , fa on Let me know whatelse do you want to see. Thanks.

  453. I cant get gemma 4 e2b or gemma 4 e4b to run on my laptop. I am runnning it via docker as per vllm website and i get the error : Free memory on device cuda:0 (9.71/11.5 GiB) on startup is less than desired GPU memory utilization (0.9, 10.3…

  454. I want to download one and usually do inference on CPU having old GPU so I'm concerned with speed. One link on the web (I have posted with it and post been removed): Multiple users are reporting that Gemma 4's MoE model (26B-A4B) runs sign…

  455. I had a nice old cracked pixel 5a laying around that I wanted to get some use out of, so I turned it into a local AI Voice assistant. A server on a laptop running llama.cpp gemma-3-4b-q4.gguf served by flask connects to a script running on…

  456. How is it possible the seller also has no record just wanted to run gemma 4 31B q4 with 150k ctx

  457. I'm now having fun with Gemma-4-E4B and Qwen3.5-9B, trying different variants like Gemopus and Qwopus, and Qwen3.5-9B-Uncensored-HauhauCS-Aggressive-Q8_0 don't quite know other models, so what's your favorite? why and how are them?

  458. I try with Gemma 4 E4B via llama-sever to play chess at https://www.chess.com/play/computer (any platform or site you convenient), result quite unexpected for me. Result: 9 moves before it make cheating move (like try to move a pawn take a…

  459. Hey all, Looking for some real-world advice on GPU choices for running the new dense models — mainly Qwen 3.5 27B and Gemma 4 31B. What I’m targeting Context: 64K+ (ideally higher later) Speed: 30+ tok/s @ tg128 minimum Power: not critical…

  460. Here is my experience, I've been using LM Studio with RTX 5080 and 64GB RAM using Windows 11. I'm very happy with LM Studio except the speed.

  461. Gemma 4 on iOS: Anyone else stuck on CPU because of the "Buffer(31)" Metal crash? Hey everyone, I’m hitting a massive performance wall building an on-device AI app for the iPhone 17 Pro.

  462. My pre-gemma 4 setup was as follows: Llama-swap, open-webui, and Claude code router on 2 RTX 3090s + 1 P40 (My third 3090 died, RIP) and 128gb of system memory Qwen 3.5 4B for semantic routing to the following models, with n_cpu_moe where…

  463. I've been testing a few models lately and I'm running into a weird issue with the bigger Qwen3.5s. Tested: Gemma 4 26B Qwen3.5 9B Qwen3.5 27B Qwen3.5 35B The 27B and 35B are driving me nuts.

  464. Hello, I have noticed an annoying issue with Gemma 4 26b a4b. It seems like it cannot do multiple think->tool call->think->tool call turns.

  465. I currently am looking for models to fit into my single DGX Spark for use. I have an RTX Pro 6000 and also a 5090 as well that I'm considering using in combination if the DGX Spark is too slow, but the intent here is to play around with Op…

  466. CPUs Aren't Dead. Gemma 2B Just Scored Higher Than GPT-3.5 Turbo on the Test That Made It Famous — Your Laptop Can Run It, or Cloudflare for $5/Mo.

  467. Based on real-world usage by the community, roughly which version of which model is Gemma 4 comparable to? It would be great if you could also mention the hardware requirements for running it (like VRAM or GPU needs)

  468. Hi everyone, I'm trying to run a local LLM via Ollama on a Hetzner cax21 VPS (ARM64, 4 vCPUs, 8GB RAM, 80GB SSD). I have Ollama running successfully via Coolify.

  469. could not extract summary

  470. Can default settings be optimized, or is it the best it is going to get? M1 Max Is it best in llama.cpp, LM Studio, or ?

  471. Hi. I've been looking at ollama cloud's Pro offering ($20), which says "Run 3 cloud models at a time".

  472. I own an Oppo Find X3 Pro (Snapdragon 888, 12/256 GB, Android 14.0) unused because of 3 green vertical lines on the screen and poor battery. I tried Google AI Edge Gallery with Gemma-4-E2B-it and it performs well so I thinked: "why don't t…

  473. has anyone really tried running models bigger than physical memory capacity? I'd guess most users stick with running models that fit in DRAM + VRAM https://unsloth.ai/docs/models/qwen3.5 even google gemma 4 are released with about 30+ bill…

  474. Thanks

  475. So, I'm relatively new to the scene and I kind of want to do a sanity check. I've been using gemma-4-26B.

  476. Use the following system prompt to allow Gemma (and most open source models) to talk about anything you wish. Add or remove from the list of allowed content as needed.

  477. I’ve been experimenting with running LLMs fully on-device, and managed to get Gemma 4 running locally on an iPhone 13 Pro. This is built on top of a lightweight Swift wrapper I open-sourced: https://github.com/mylovelycodes/LiteRTLM-Swift…

  478. I'd like to self-host some LLM models but a couple different ones for different usecases, and they don't all fit in VRAM at the same time. So i'm kind of looking for a tool in which i can define "profiles" or "stacks" of LLM's that get loa…

  479. we got it good boys! how many of you are doing this??

  480. Google Gemma 4 Runs Natively on iPhone With Full Offline AI Inference - GizmoWeek GizmoWeek Read the News News Reviews Apple How to Phones Products Subscribe Subscribe to newsletter [x] I've read and accept the Privacy Policy. Follow us Fa…

  481. Hi everyone, I've been working on Pocket LLM, an Android app for running local LLMs fully offline for private, real-time chat. The latest v1.3.0 update adds: • LiteRT support for Gemma 4 E2B, Gemma 4 E4B, and Qwen3-0.6B • Persistent local…

  482. I'm currently vibe-coding (I'm new to vibe-coding) with Gemma 4 4EB Q4 and Qwen 3.5 9B Q5 (KV is quantized to 4 bits with new Google TurboQuant implemented in llama.cpp - I use koboldcpp and release said it's automatically activated): the…

  483. I've been waiting for a capable free local LLM for a while. I think we're close — the quality is getting there fast, and Gemma 4 is the first open-weight model where I genuinely considered using it in production for simple-to-medium tasks.

  484. About Press Copyright Contact us Creators Advertise Developers Terms Privacy Policy & Safety How YouTube works Test new features NFL Sunday Ticket © 2026 Google LLC

  485. Both Groq and Cerebras haven't really updated their provided model for a while, long enough to notice the difference between old and new models on the market. So why don't they add any new models?

  486. For example, in LMStudio the official Gemma 4 is a GGUF that has Vision, Reasoning, and Tools flags. But the MLX version does not.

  487. Hey there, I noticed something odd when trying out the latest and greatest local reasoning models recently. First, I just noticed it for Qwen3.5, but Gemma 4 seems to do it too: The reasoning traces do that weird thing of starting with "He…

  488. 🌟 Gemopus-4-26B-A4B-it [!NOTE] Gemopus is an attempt at fine-tuning Gemma 4 with a core philosophy of "stability first". While preserving the original reasoning order of Gemma 4 as much as possible, we conducted targeted refinements for an…

  489. Benchmarks, rankings, and live play for AI models and agents.

  490. If you're using Gemma 4 with external MCP servers in LM Studio and getting this error: Error rendering prompt with jinja template: "Unknown test: sequence" This is a bug in Google's official Gemma 4 Jinja prompt template. LM Studio's Jinja…

  491. I was building an Android app and integrated Gemma 4 E2B directly using LiteRT-LM. On-device translation, zero server cost, the dream setup.

  492. Hello, I've seen reviews that gemma 4 31b base is very good at roleplaying. But I can't find the gguf version of the basic gemma 4 anywhere.

  493. I'm trying to figure out what sort of hardware setup i will need to accomodate a userbase of 100 users (not necessarily concurrent). Does anyone have any idea what sort of setup i'd be looking at?

  494. Does anyone know of a good edge local llm that is good in math's. I tried Gemma 4 E2B, microsoft phi mini reasoning but both can't answer some basic apti question's.

  495. Hi, my current system hardware RTX 3090 24GB VRAM & Sysrem RAM 64GB using windows 11 been playing around with hermes agent and local llm (Qwopus3.5-27B-v3-GGUF & gemma-4-26B-A4B-it-GGUF) when i try asking the hermes agent to do a task with…

  496. This is V2 of my previous post. What's new: --ai-tune — the model starts tuning its own flags in a loop and caches the fastest config it finds.

  497. Gemma quant comparison on M5 Max MacBook Pro 128GB (subjective of course, but on variety of categories): gemma 4 leaderboard the surprising bit: Gemma 4 31B 4bit scored higher than 8bit. 91.3% vs 88.4%.

  498. Two questions: which model? In my mind, Qwen3.5 27b or Gemma 4 31b are top options.

  499. Could you imagine if the new Apple Foundation Models was based on Gemma 4 E4B text like the LiteRT version is? That would be one amazing built in model.

  500. Hey folks, I've been tinkering with Gemma 4 and absolutely the fact this model can run locally on Android phone! I am experienced fullstackdev, open to solve any real-world problem that has an impact.

  501. If you are on Gemma (like me), you basically have to compile llama.cpp daily now

  502. I'm sure everyone has seen the posts from people talking about Qwen 3.5 over-thinking, or maybe you've experienced it yourself. Considering we're like 2 months out from the release and I still see people talk about this issue, I decided it…

  503. I did some searching and didn't find a extremely similar situation. I'm jumping head first into hosting locally, and my experience has been good so far.

  504. Hello everyone, a newbie here. Amazed by OpenClaw and worried by its high API consumption, I decided to buy two Asus Ascent GX10s (like the Nvidia Spark), so I have a pretty powerful inference cluster with 220GB of real available memory.

  505. Con el lanzamiento de modelos optimizados para ejecutarse localmente (como lo que estamos viendo con la evolución de Gemma 4), parece que el péndulo de la IA se está alejando de la nube.

  506. Hey guys! How would you reckon a 30-50b model would run on a 48 GBs m5 pro?

  507. I'm trying to replace openai codex which i used for development all the time, with gemma4 on 4090, small tasks it solves quite impressively, but i need to have some agent. So I tried to connect 31b to cline and to aider and it didn't reall…

  508. Gemma 4 E2B vs the Gemma Family: The 2B Underdog That Punches Above Its Weight Google's newest 2B model tested across 10 enterprise task suites against Gemma 2 2B, Gemma 3 4B, Gemma 4 E4B, and Gemma 3 12B. Run locally on Apple Silicon.

  509. Recently, I try playing with gemma 4 (gemma-4-E4B-it-Q5_K_S.guff) and find out it fail at easy grammar check (it try to fix the already corrected word "contemporary"). I noticed the same mistake from openai/gpt-oss-20b and qwen3-next-80b-a…

  510. I’ve been experimenting with running a local coding assistant on Gemma 4 26B, focused on understanding full codebases instead of single-file prompts. Main idea: - build a project map (files, symbols, structure) - run a planning step to dec…

  511. I was surprised at how much smaller the LiteRT versions of Gemma 4 E2B used in Edge Gallery were (2.0-3.3 GB) compared to the main release (10.2 GB), so I had Claude code take a look. Claude tells me that the vocab size for the LiteRT vers…

  512. Models compared: Qwen3.5-27B-UD-Q5_K_XL gemma-4-31B-it-UD-Q5_K_XL Main flags for boths --flash-attn on \ --n-gpu-layers 99 \ --no-mmap \ -c 150000 \ --temp 1 --top-p 0.9 --min-p 0.1 --top-k 20 \ --ctx-checkpoints 1 \ --jinja \ -np 1 \ --re…

  513. I ran Gemma 4 as a local model in Codex CLI | by Daniel Vaughan | Google Cloud - Community | Apr, 2026 | Medium Sitemap Open in app Sign up Sign in Get app Write Search Sign up Sign in Google Cloud - Community · A collection of technical a…

  514. I actually make some tests with lm studio and Opencode with the new Gemma 4 26b model. The results are really impressive especially on small refactoring and integration tasks.

  515. Question: can i do the thinking with a smaller model, like Gemma 4 4B, then use that as the prompt for Gemma 4 31B, to speed things up? Has anyone done this and measure if it's worth it?

  516. My specs: RTX 5060ti(16gb), 16gb DDR5 ram. (os : Fedora 43) I want an uncensored model, it would be preferable if it can do image gen but if the quality of text is high enough it should not be problem if it does not support it.

  517. I’ve been keeping a personal journal for the past few years. The entire thing is made up of over 100k+ tokens.

  518. Hello, I've been trying for several days to train Gemma-4 for extracting data from a string and convert it into a structured JSON. I've tried a fair amount of different configurations, I've tried Unsloth studio and Llamafactory, but in eac…

  519. A year ago I would just read about 397B league of models. Today I can run it on my laptop.

  520. GLM 5.1 is dominant in almost every aspect in Design arena, surpassing Opus 4.6 in many tasks. Although user experiences vary dependent on subscription plans for both of those one of them is open source.

  521. Gemma 4: Byte for byte, the most capable open models Today, we are introducing Gemma 4 — our most intelligent open models to date. Purpose-built for advanced reasoning and agentic workflows, Gemma 4 delivers an unprecedented level of intel…

  522. Welcome Gemma 4: Frontier multimodal intelligence on device These models are the real deal: truly open with Apache 2 licenses, high quality with pareto frontier arena scores, multimodal including audio, and sizes you can use everywhere inc…

← all threads