#gemma

Use the following system prompt to allow Gemma (and most open source models) to talk about anything you wish. Add or remove from the list of allowed content as needed.

↯ Security↯ Gemma↯ Jailbreak↯ Gemma 4jailbreakgemmasecurity

My pre-gemma 4 setup was as follows: Llama-swap, open-webui, and Claude code router on 2 RTX 3090s + 1 P40 (My third 3090 died, RIP) and 128gb of system memory Qwen 3.5 4B for semantic routing to the following models, with n_cpu_moe where…

↯ Llama↯ Gemma↯ Qwen 3.5gemmallamaqwen+1

Gemma 4 and Qwen 3.6 with q8_0 and q4_0 KV cache: KL divergence results 4 models tested with q8_0 and q4_0 KV cache against full-precision baseline What this measures KV cache quantization stores the key-value cache in lower precision to s…

↯ Gemma↯ Qwen 3.6gemmaqwen

Google Gemma 4 Runs Natively on iPhone With Full Offline AI Inference - GizmoWeek GizmoWeek Read the News News Reviews Apple How to Phones Products Subscribe Subscribe to newsletter [x] I've read and accept the Privacy Policy. Follow us Fa…

↯ Gemma↯ Gemma 4gemma

I’ve been keeping a personal journal for the past few years. The entire thing is made up of over 100k+ tokens.

↯ Gemma↯ Gemma 4gemma

Hi LocalLLaMA, I created a post a few weeks ago, but this time this project has become more reliable and easier to use. This is a manga translator that can also be used to translate any image.

↯ Llama↯ Gemma↯ Qwen 3.5gemmallama

Hi HN, I built a specialized inference engine for running 4-bit Gemma 4 26B-A4B-IT on any M-series Mac using about 2 GB of RAM. It is called TurboFieldfare and is written in Swift and Metal.

↯ Gemma↯ Gemma 4gemma

other companies are slowly going away from open weight, not releasing base models, delaying open weight distribution, not releasing top models (this one I think is fair, but still), and I also noticed they stopped publishing research (old…

↯ Glm↯ Minimax↯ Gemma↯ Qwen 3.5minimaxglmgemma+2

Sparky runs entirely on the Jetson. Gemma 4 E4B at Q4_K_M via llama.cpp with q8_0 KV cache and flash attention.

↯ Llama↯ Gemma↯ Gemma 4gemmallama

↯ Gemmagemma

Man, AI agents getting pretty crazy these days. :) (local, I just decided to try to get an orchestrator in there, when Qwen and Gemma aren't up to it.)

↯ Gemmagemmaqwen

I've tried a few different local models in the past (gemma 4 being the latest), but none of them felt as good as this. (Or maybe I just didn't give them a proper chance, you guys let me know).

↯ Gemma↯ Qwen 3.6gemmaopusclaude-code

I created this chart with recent open models from last 6 months. Few might be older than that possibly.

↯ Glm↯ Mistral↯ Gemma↯ DeepSeek 3.2mistralglmgemma+1

Gemma just crushed Qwen in a local LLM gamedev contest! Device: MacBook Pro M5 Max, 64GB RAM Qwen 3.6 27B: 32 tokens/sec · 18m 04s · 33,946 tokens.

↯ Gemma↯ Qwen 3.6gemmaqwen

Turboderp has a been on an absolute tear recently, in the endless battle to cram new llamas into smaller, faster boxes. We started off last month with the release of gemma 4 support, and continued with improved caching efficiency.

↯ Gemma↯ Gemma 4gemmaagentic

This is V2 of my previous post. What's new: --ai-tune — the model starts tuning its own flags in a loop and caches the fastest config it finds.

↯ Llama↯ Gemma↯ Qwen 3.5gemmallama

could not extract summary

↯ Gemmagemma

I built hfviewer.com, a small tool for visually exploring Hugging Face model architectures. You can paste a Hugging Face URL and get an interactive visualization of the architecture, which can make it easier to understand how different mod…

↯ Gemma↯ Qwen 3.6gemmaqwen

CPUs Aren't Dead. Gemma 2B Just Scored Higher Than GPT-3.5 Turbo on the Test That Made It Famous — Your Laptop Can Run It, or Cloudflare for $5/Mo.

↯ Gemmagemma

GLM 5.1 is dominant in almost every aspect in Design arena, surpassing Opus 4.6 in many tasks. Although user experiences vary dependent on subscription plans for both of those one of them is open source.

↯ Glm↯ Gemma↯ Qwen 3.6glmgemmaqwen+1

I have a personal eval harness: A repo with around 30k lines of code that has 37 intentional issues for LLMs to debug and address through an agentic setup (I use OpenCode) A subset of the harness also has the LLM extract key information fr…

↯ Gemma↯ Qwen 3.6gemmaqwenagentic

I'm using the https://github.com/PrismML-Eng/llama.cpp fork for Bonsai, regular llama.cpp for Gemma. Without embedding parameters: Gemma 4 has 2.3B at 4.8 bpw (Q4_K_M) = 1104 MB Bonsai-8B has 6.95B at 1.125 bpw (Q1_0) = 782 MB (only 29% sm…

↯ Llama↯ Gemma↯ Gemma 4gemmallama

↯ Glm↯ Gemma↯ Qwen 3.6glmgemmamcp

↯ Gemma↯ Gemma 4gemma

↯ Gemma↯ Qwen 3.6gemma

↯ Gemma↯ Gemma 4gemma

We have a chat system which we use haiku for because it is mostly about tool calling and summarisation of them. But we have many tools with pretty complex input schemas, and stuff like gemma didn't cut it, so we went with haiku.

↯ Haiku↯ Gemma↯ DeepSeek 4haikugemmadeepseek+1

BeeLlama v0.2.0 is here! Not quite a pegasus, but close enough.

↯ Gemma↯ Qwen 3.6gemmaqwen

Chat Template was fixed a few days ago choose your fav dealer: https://huggingface.co/bartowski/google_gemma-4-31B-it-GGUF https://huggingface.co/bartowski/google_gemma-4-26B-A4B-it-GGUF https://huggingface.co/bartowski/google_gemma-4-E4B-…

↯ Gemma↯ Gemma 4gemma

How to Setup a Local Coding Agent on macOS Running Gemma 4 26B-A4B and Qwen3.6 35B-A3B locally with llama.cpp, MTP speculative decoding, multimodal support, and PI as a coding agent. I'd had my internet fail a few times recently leaving me…

↯ Llama↯ Gemma↯ Qwen 3.6↯ Qwen 3.6gemmallama

Is there a 7B parameter model in your life struggling to understand sarcasm? A tiny 1.5B that can't afford one more epoch?

↯ Gemmagemma

Gemma quant comparison on M5 Max MacBook Pro 128GB (subjective of course, but on variety of categories): gemma 4 leaderboard the surprising bit: Gemma 4 31B 4bit scored higher than 8bit. 91.3% vs 88.4%.

↯ Gemma↯ Gemma 4gemma

↯ Gemma↯ Qwen 3.5gemmaqwenagentic+1

Google AI Edge Gallery ✨ Explore, Experience, and Evaluate the Future of On-Device Generative AI with Google AI Edge. AI Edge Gallery is the premier destination for running the world's most powerful open-source Large Language Models (LLMs)…

↯ Gemma↯ Gemma 4gemmamcp

I'm sure everyone has seen the posts from people talking about Qwen 3.5 over-thinking, or maybe you've experienced it yourself. Considering we're like 2 months out from the release and I still see people talk about this issue, I decided it…

↯ Gemma↯ Qwen 3.5gemmaqwengemini

This spring is really hot since the localLLM giant, both Qwen and Gemma released major models. I'm really excited with those release and happy with their capability.

↯ Gemmagemmamoeqwen

Models compared: Qwen3.5-27B-UD-Q5_K_XL gemma-4-31B-it-UD-Q5_K_XL Main flags for boths --flash-attn on \ --n-gpu-layers 99 \ --no-mmap \ -c 150000 \ --temp 1 --top-p 0.9 --min-p 0.1 --top-k 20 \ --ctx-checkpoints 1 \ --jinja \ -np 1 \ --re…

↯ Gemma↯ Qwen 3.5gemmaagentic

I think gave it a fair shot over the past few weeks, forcing myself to use local models for non-work tech asks. I use Claude Code at my job so that's what I'm comparing to.

↯ Gemma↯ Gemma 4gemmaqwenagentic+1

When I previously posted the uncensored version of the 31B version of the MeroMero finetune, quite a few people asked for the 26B-A4B version, I wasn't so keen on it because I considered the 31B to be the better version, but I understand t…

↯ Gemma↯ Gemma 4gemma

could not extract summary

↯ Gemma↯ Gemma 4gemma

another day with pi + gemma 26B

↯ Gemmagemma

Pocket LLM v1.5.0🚀 New in this release: - 🎙️ Voice input - 🖼️ Image input with OCR, Gemma vision, and FastVLM support - 📷 Camera capture with retake, crop, and photo review - 🗂️ Previous chats side panel - 💾 Downloaded model deletion to sa…

↯ Gemmagemma

Not affiliated with Kaitchup, but a fan of their testing. I was looking forward to this article...

↯ Gemma↯ Qwen 3.6gemmaqwen

June 8, 2026 · By Ryan Findley Running Gemma 4 26B at 5 tokens/sec on a 13-year-old Xeon with no GPU There’s a server in my basement that has no business running a modern language model. It’s a repurposed HP StoreVirtual storage box, rough…

↯ Gemma↯ Gemma 4gemma

Anthropic has released new research to show what an LLM is thinking when generating a next token using NLA or "Natural Language Autoencoders", the NLAs are a pair to LLMs that can translate internal thoughts of LLM for any specific token.…

↯ Gemmagemmaanthropic

Talkie-1930-13b-it and Gemma 4 31b in the same chat. Talkie is a 13B vintage language model from 1930.

↯ Gemma↯ Gemma 4gemma

Howdy everyone! Quick disclosure: I work on this - it's a project my studio created called the Null Epoch.

↯ Gemmagemma

Hey HN, we’re the developers of OpenLake, an open source storage engine for offloading LLM KV caches from GPU memory into a shared tier of RAM and NVMe. We built OpenLake because KV caches are outgrowing GPU memory.

↯ Gemma↯ Gemma 4gemma

So in response to the Great Token Reconning of 2026, I decided to try out Qwen 3.6 as a daily driver, and although it's only been about a day, I have to say I'm thoroughly impressed. I had to download the VSCode insiders edition and set up…

↯ Copilot↯ Gemma↯ Qwen 3.6gemmacopilotqwen

↯ Gemma↯ Qwen 3.6gemmamoeqwen

If you are on Gemma (like me), you basically have to compile llama.cpp daily now

↯ Llama↯ Gemma↯ Gemma 4gemmallama

As a life-long Windows user (don't hate me, I was exposed to it at a young age) I was wondering how much (if any) performance I'm leaving on the table. So I did the sensible thing and run some benchmarks.

↯ Llama↯ Gemma↯ Qwen 3.6gemmallama

TLDR: tool parameters using the common JSON Schema pattern `anyOf: [$ref, null]` are rendered into the prompt as empty `type` fields. This strips the useful schema information before the model sees it.

↯ Llama↯ Gemma↯ Qwen 3.5gemmagpt-5llama+1

Benchmarked Gemma 4 MTP and z-lab's DFlash on a single H100 80GB using vLLM and NVIDIA's SPEED-Bench qualitative dataset. Setup: Hardware: 1x H100 80GB Runtime: vLLM Dataset: SPEED-Bench qualitative Prompts: 880 total, 80 prompts across ea…

↯ Vllm↯ Gemma↯ Gemma 4vllmgemmamoe

I ran a benchmark to see how much DFlash speculative decoding actually helps in vLLM. Setup: GPU: RTX 5090, 32GB VRAM vLLM: 0.19.2rc1 Main model: cyankiwi/gemma-4-26B-A4B-it-AWQ-4bit Draft model: z-lab/gemma-4-26B-A4B-it-DFlash Workload: r…

↯ Vllm↯ Gemma↯ Gemma 4vllmgemma

Qwen3.5-122B-A10B at Q6_K is really good. Do you think we will see a larger MoE Gemma-4 or Qwen3.6 at some point?

↯ Gemma↯ Qwen 3.6gemmamoe

I have run two tests on each LLM with OpenCode to check their basic readiness and convenience: - Create IndexNow CLI in Golang (Easy Task) and - Create Migration Map for a website following SiteStructure Strategy. (Complex Task) Tested Qwe…

↯ Glm↯ Gemma↯ Qwen 3.6glmgemmaqwen

Well or pretty close to it, they are excellent work horses. I run them in real work scenarios doing some of the work I used to do myself as an skilled expert in my field, billing 200$ an hour.

↯ Mistral↯ Gemma↯ Qwen 3.6mistralgemmaqwen

I tried mudler's apex quant for gemma4 26b a4b and it was amazing! I got 38tps at 90.000 context with no loop and suprisingly no quality degradation.

↯ Llama↯ Gemma↯ Gemma 4gemmallama

I wrote up this little python app to cycle through a bunch of prompts like this: Single HTML file using three.js from CDN. A central rotating MeshNormalMaterial torus knot.

↯ Gemma↯ Gemma 4gemma

I have been nothing but impressed by the quality of Gemma 4 since release. In general conversation it's adaptable to different personas.

↯ Gemma↯ Sonnet 4.6gemmasonnetgemini+1

The script to graft MTP tensors requires a full GGUF model file. I felt that was a bit hefty, so I asked local Gemma to write something to just extract what's required.

↯ Gemmagemma

Provided in both Safetensors and GGUFs. Safetensors, llmfan46/Gemma-4-Harmonia-31B-it-uncensored-heretic: https://huggingface.co/llmfan46/Gemma-4-Harmonia-31B-uncensored-heretic GGUFs, llmfan46/Gemma-4-Harmonia-31B-it-uncensored-heretic-GG…

↯ Gemma↯ Gemma 4↯ Gemma 4↯ Gemma 4↯ Gemma 4↯ Gemma 4↯ Gemma 4gemma

Past few days, its all been about MTPs. Somehow people missed out the fact that Z lab released the Dflash for Gemma4 26B a couple of days ago.

↯ Vllm↯ Gemma↯ Qwen 3.6vllmgemmaqwen

Have Qwen 3.6 27B and Qwen 3.6 35B basically made most of the older ~30B models irrelevant? They seem to beat stuff like Qwen coder 30B, GPT OSS 20B, Gemma models, especially for coding and agent workflows.

↯ Gemma↯ Qwen 3.6gemmaqwen

Asked new Google Model to build SuperMario. Compared with Local Gemma4.

↯ Gemma↯ Gemini 3.5gemmagemini

Some people say they’d never go under Q8, and others say they find Q3 acceptable! What’s your take?

↯ Gemmagemmaqwen

Provided in both Safetensors and GGUFs. Safetensors: llmfan46/Gemma-4-Gembrain-31B-it-uncensored-heretic: https://huggingface.co/llmfan46/Gemma-4-Gembrain-31B-it-uncensored-heretic GGUFs: llmfan46/Gemma-4-Gembrain-31B-it-uncensored-heretic…

↯ Gemma↯ Gemma 4gemma

I've been waiting for a capable free local LLM for a while. I think we're close — the quality is getting there fast, and Gemma 4 is the first open-weight model where I genuinely considered using it in production for simple-to-medium tasks.

↯ Gemma↯ Gemini 3.1gemmamoegemini+1

A year ago I would just read about 397B league of models. Today I can run it on my laptop.

↯ Gemma↯ Qwen 3.5gemmaqwen

I read a lot of "whengguf" type posts. I think we should sometimes stop and be grateful.

↯ Llama↯ Mistral↯ Gemmamistralgemmallama+1

When Qwen3.6-35B-A3B was released a week or so ago, I sort of expected an iterative improvement on the previous Qwen3.5 models. After all, those models were pretty decent as compared with the previous local models I had tried, and Qwen3.5…

↯ Gemma↯ Qwen 3.6gemmaqwen

I had a nice old cracked pixel 5a laying around that I wanted to get some use out of, so I turned it into a local AI Voice assistant. A server on a laptop running llama.cpp gemma-3-4b-q4.gguf served by flask connects to a script running on…

↯ Llama↯ Gemma↯ Gemma 3gemmallama

I kept seeing inference-speed claims for these models and wanting an apples-to-apples comparison on the hardware I actually have. So I built a harness and a public page that dumps every run as YAML.

↯ Vllm↯ Gemma↯ Qwen 3.5vllmgemmamoe+1

Been running Gemma 4 E2B locally on my OnePlus CE 5 (8GB RAM) for a few months. Chat quality is fine for the size.

↯ Gemma↯ Gemma 4gemma

Hi everyone, I've been working on Pocket LLM, an Android app for running local LLMs fully offline for private, real-time chat. The latest v1.3.0 update adds: • LiteRT support for Gemma 4 E2B, Gemma 4 E4B, and Qwen3-0.6B • Persistent local…

↯ Gemmagemma

I'm the founder behind Hedy, an AI meeting app. I'm a huge supporter of Local AI, and we've been working on making it "consumer friendly".

↯ Llama↯ Gemma↯ Qwen 3.6gemmallamaqwen

When dealing with untrusted outside input, I think you should handle it based on the situation. If you're processing structured data files, it's better to use tools to isolate and handle them.

↯ Security↯ Gemma↯ Gemma 4prompt-injectiongemmagrok+1

Hey all, Looking for some real-world advice on GPU choices for running the new dense models — mainly Qwen 3.5 27B and Gemma 4 31B. What I’m targeting Context: 64K+ (ideally higher later) Speed: 30+ tok/s @ tg128 minimum Power: not critical…

↯ Gemma↯ Qwen 3.5gemmamoeqwen

We're excited to announce AI text and image detection in Bouncer! We've been seeing a lot of Bouncer users try to filter out AI content from their feeds by adding "AI slop", "AI video", or "AI written" as filter terms.

↯ Gemmagemma

Before Fable 5 was shut down, it pushed Gemma 4 to 255 tok/s on WebGPU. Some didn't believe it was real.

↯ Gemma↯ Gemma 4↯ Gemma 4↯ Gemma 4↯ Gemma 4gemma

https://github.com/ggml-org/llama.cpp/blob/master/docs/backend/snapdragon/README.md I have an Oneplus 12 with Snapdragon 8 Gen 3. I followed the above README to cross-compile llama.cpp on Ubuntu and then copy to the Termux directory on the…

↯ Llama↯ Gemma↯ Gemma 3gemmallama

https://huggingface.co/firstbober/gemma-3-270M-it-smol-thinker Here is an example of the output: ``` ==================== THINKING ==================== Here is the thinking process: This is a large community with a wide range of interests…

↯ Gemma↯ Gemma 3gemma

I wanted to figure out which of the newer small and mid-size models are actually worth running on a single H100, so I put 8 of them through a proper vLLM benchmark and recorded what came out. The setup was simple.

↯ Vllm↯ Gemma↯ Qwen 3.6vllmgemmaqwen

↯ Gemma↯ Qwen 3.6gemmaqwen

we got it good boys! how many of you are doing this??

↯ Gemma↯ Gemma 4gemmaagentic

https://huggingface.co/stevelikesrhino/gemma-4-31B-it-nvfp4-GGUF/blob/main/gemma4-improved.jinja Yall are more than welcome to try it out and provide feedback. In my own testing in Pi-coding-agent I no longer have the "forgot to close thin…

↯ Llama↯ Gemma↯ Gemma 4gemmallama

Hello, I'm currently using Ollama / lm studio for things like code inference and proof reading emails, etc. Definitely not experienced in this space but looking to grow.

↯ Ollama↯ Llama↯ Vllm↯ Gemma↯ Gemma 4vllmollamagemma+3

https://huggingface.co/z-lab/gemma-4-31B-it-DFlash I guess we'll have to wait until this PR is merged before we can test it. https://github.com/ggml-org/llama.cpp/pull/22105

↯ Llama↯ Gemma↯ Gemma 4gemmallama

Longtime lurker here, thought i should post my speeeeds... I have a RTX 4070S 12 GB Vram (+10% OC), AMD 9800x3D with 4x16 Gb DDR5 6000Mhz CL30.

↯ Gemma↯ Qwen 3.6clinegemma

↯ Ollama↯ Gemma↯ Gemma 4ollamagemma

question, I'm using it for text translations and on each large prompt (20K+) it stops with a remark 'now I'm going to put that to the file' or some other operation I have asked in the prompt for but it did nothing, just stopped. I'm runnin…

↯ Gemma↯ Gemma 4gemma

I have been benchmarking local LLMs on a Mac M4 Pro 24 GB RAM using LM Studio. I've tested mostly with 4-bit quantization, both MLX and GGUF, from 4b to 35b models, with speeds of 3 to 40 tokens/second.

↯ Gemma↯ Qwen 3.6gemmaragqwen

Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference

↯ Gemma↯ Gemma 4↯ Gemma 4↯ Gemma 4↯ Gemma 4gemma

As per the title Such as Gemma 4 31B Q4 K S vs Gemma 4 26B A4B Q8 Or Qwen 3.6 27B Q4 K M vs Qwen 3.6 35B A3B Q6 K Etc At what point is it worth switching? My use case is mostly creative writing.

↯ Gemma↯ Qwen 3.6gemmaqwen

The only thread was 2 months ago, when the model had just dropped. Since then, more versions from different authors have appeared, and users have had time to test them.

↯ Gemma↯ Gemma 4gemma

new model from LatitudeGames - Gemma 31B finetune https://huggingface.co/LatitudeGames/Equinox-31B-GGUF Equinox draws its name from the balance between extremes. Trained on a balanced blend of Wayfarer 2's unforgiving dark adventures and H…

↯ Gemmagemma

Hi r/LocalLLaMA - I've been paying close attention to the edge AI ecosystem because it's an area where i see huge potential and where I truly believe AI will become more useful for day to day tasks. Around the gemma 4 release I was already…

↯ Llama↯ Gemma↯ Gemma 4gemmallama

We'll be getting those features(check bottom link) on mainline soon or later anyway. But for now this fork could be useful to see the full potential of our poor GPUs(and also big, large GPUs).

↯ Gemma↯ Qwen 3.6gemmaagentic

Watched All About AI's 100% local Fireship-style video automation experiment over the weekend (link in comments). A few things worth flagging if you're trying the same stack.

↯ Gemma↯ Qwen 3.6gemmaqwen

With every new model release there's the "better than Opus 6.13" guys vs the "this is so bad, why did they even release it" camp and I'm always wondering which one is using it wrong. So I did a little test with 2 related prompts, 3 models…

↯ Gemmagemmaqwenopus

MIT license and fully open source. MiMo-V2.5-Pro was just 3 points from Opus 4.7 max and the normal V2.5 is only a step behind SOTA.

↯ Hallucination↯ Gemma↯ DeepSeek 4hallucinationgemmadeepseek+1

So for my project I was using up until now either Gemini 3 / 2.5 Flash or Flash-lite. All my use cases are not agentic, simply LLM workflows for atomic tasks like extracting references from the law, classifying, adjusting titles to nominat…

↯ Gemma↯ Gemini 2.5gemmageminiagentic

Here is my experience, I've been using LM Studio with RTX 5080 and 64GB RAM using Windows 11. I'm very happy with LM Studio except the speed.

↯ Llama↯ Gemma↯ Gemma 4gemmallamaqwen

Benchmarks, rankings, and live play for AI models and agents.

↯ Gemma↯ Gemma 4gemma

Question: can i do the thinking with a smaller model, like Gemma 4 4B, then use that as the prompt for Gemma 4 31B, to speed things up? Has anyone done this and measure if it's worth it?

↯ Gemma↯ Gemma 4gemma

Hi guys. I have been working on Hitoku Draft, an open-source, voice-first AI assistant that runs entirely locally.

↯ Gemma↯ Qwen 3.5gemmaqwen

I got Qwen 3.6 35B-A3B and Gemma 4 26B-A4B running on a $200 secondhand machine (i7-6700 / GTX 1080 / 32 GB RAM) using llama.cpp (the TurboQuant/RotorQuant KV cache quantisation allows 128k context within the 8 GB VRAM). Results (Q4_K_M mo…

↯ Llama↯ Gemma↯ Qwen 3.6gemmamoellama+1

Per title. Been trying to load Gemma E2b as draft model for 26b as target using LM Studio's UI but it can't seem to recognise what's already been downloaded.

↯ Gemmagemma

I am quite curious as I tried Gemma 4 31B, Qwen 3.6 27B, GLM 4.7 30B and some others in my native language (czech). Gemma performs "best" and considering the fact its "just" 18GB model - it actually blows my mind how well it can respond in…

↯ Glm↯ Gemma↯ Qwen 3.6glmgemmaqwen

True story, I got interested in AI after seeing it at work and wanted to run models locally. I started with an M3 Ultra 96GB, quickly learned it was not enough for what I wanted, and kept upgrading hardware (including refurbished Mac Studi…

↯ Minimax↯ Gemma↯ DeepSeek 4minimaxgemmadeepseek+1

Did some quick tests after building llama.cpp with ROCm 6.4.2 and latest Vulkan for my 6900 XT gemma4 E2B Q4_K ubatch ROCm pp512 Vulkan pp512 ROCm tg128 Vulkan tg128 32 1536.60 1423.49 151.92 174.59 64 1590.65 1930.60 151.41 173.76 128 265…

↯ Llama↯ Gemma↯ Qwen 3.5gemmallamaqwen

I did give same prompt same document to 1660ti running Gemma 4 e2b q4 coz of the small vram and another to and igpu running Gemma 4 e4b q8 prefill rate before token generation was like 4-5 times faster with the 890m igpu then token generat…

↯ Gemma↯ Gemma 4gemma

↯ Gemma↯ Qwen 3.5gemmaqwenagentic

↯ Gemma↯ Gemma 4gemma

I'm now having fun with Gemma-4-E4B and Qwen3.5-9B, trying different variants like Gemopus and Qwopus, and Qwen3.5-9B-Uncensored-HauhauCS-Aggressive-Q8_0 don't quite know other models, so what's your favorite? why and how are them?

↯ Gemma↯ Qwen 3.5gemma

Gemma 4 E2B vs the Gemma Family: The 2B Underdog That Punches Above Its Weight Google's newest 2B model tested across 10 enterprise task suites against Gemma 2 2B, Gemma 3 4B, Gemma 4 E4B, and Gemma 3 12B. Run locally on Apple Silicon.

↯ Gemma↯ Gemma 4gemma

Recently, I try playing with gemma 4 (gemma-4-E4B-it-Q5_K_S.guff) and find out it fail at easy grammar check (it try to fix the already corrected word "contemporary"). I noticed the same mistake from openai/gpt-oss-20b and qwen3-next-80b-a…

↯ Gemma↯ Gemma 4gemmaopenai

openjev Disclaimer: This project was inspired by vinnylarouge/jevlike. It is an independent reimplementation and is not affiliated with or endorsed by the original author.

↯ Gemmagemma

gemma4.c Gemma 4 E2B CPU inference in 700 lines of pure C. An educational project made to understand how LLM inference works.

↯ Gemma↯ Gemma 4↯ Gemma 4↯ Gemma 4↯ Gemma 4↯ Gemma 4gemma

The generative AI boom has driven the cost of memory into the stratosphere, and Google is a key part of that trend. So it’s only fitting that Google should offer some less RAM-hungry local AI models.

↯ Gemma↯ Gemma 4gemma

I made my first macOS utility app that ships with a bundled Gemma 4 model, specifically the Gemma E4B one. It made my app DMG have 5.3 GB in size, but I think it is a small size for the power that this free local model can provide.

↯ Gemma↯ Gemma 4↯ Gemma 4gemma

Fun local LLM comparisons with Gemma, Granite, and Qwen Ekorbia v0.2 features a comparison-chat mode that runs 2-3 local models against the same prompt in parallel. Here are a few fun prompts running across Gemma 4 (e2b), IBM Granite 4.1 (…

↯ Gemma↯ Gemma 4↯ Gemma 4↯ Gemma 4↯ Gemma 4↯ Gemma 4↯ Gemma 4gemmaqwen

I use cotpyist and typeWhisper. Both of them use local AI models to enhance my writing.

↯ Gemmagemma

Provided in both Safetensors and GGUFs. llmfan46/gemma-4-Ortenzya-The-Creative-Wordsmith-31B-it-uncensored-heretic: https://huggingface.co/llmfan46/gemma-4-Ortenzya-The-Creative-Wordsmith-31B-it-uncensored-heretic llmfan46/gemma-4-Ortenzya…

↯ Gemma↯ Gemma 4gemma

Okay so i've been stalking this sub for some time and i run the occasional small 2-8b model on my laptop (not the best) for fun but say my role at a company is to set up a local LLM since we obviously don't want confidential data going to…

↯ Gemma↯ Qwen 3.6gemmaragqwen

I don't think alibaba officially stated about "no qwen3.6 smaller models", and according to the patterns, she should ave been released it in the first week of may, but I think they delayed a little bit to catch the spotlight from Google I/…

↯ Gemma↯ Qwen 3.6gemma

Many local models have a problem (that raised due to excessive RHLF training): They mostly think that everything that is beyond their knowledge cutoff date would be "fictional" or "satirical". To be fair: Even the Gemini API without web ac…

↯ Gemma↯ Gemma 4gemmagemini

Z-Lab did some good work with speeding up output, while Luce managed to use smaller models of the same family to accelerate prefill... Since Heretic and other "smart ablation" tools can decensor a model, would they work with these multi-mo…

↯ Gemma↯ Qwen 3.6gemma

I had been looking to try <500M parameter language models but you wouldn't find an API to try them anywhere, so I built this cloudflare hosted static website that hosts weights and built an inference runtime for these models that uses WebG…

↯ Gemmagemmacodex

Very quick initial test of Gemma 4 new MTP model via Ollama (llama.cpp doesnt support yet) https://blog.google/innovation-and-ai/technology/developers-tools/multi-token-prediction-gemma-4/ Running in Open Webui to view token/s output and I…

↯ Ollama↯ Llama↯ Gemma↯ Qwen 3.6ollamagemmallama

The idea of offload-mcp is simple: instead of running hardware-hungry local models for routine work, let Claude offload that work to FREE model APIs and SAVE tokens. I’m using Gemma via the Google GenAI API because I like it in my processi…

↯ Gemma↯ Gemma 4gemmacodexmcp+1

I'm testing running local LLMs on a gaming mini PC (AMD 7840HS, 32 GB RAM) paired with an eGPU (Radeon 9060XT with 16 GB VRAM). Since I'm not very familiar with using llama.cpp, I kept getting unsatisfactory results, but with the recent Ge…

↯ Llama↯ Gemma↯ Gemma 4gemmallama

I have observed that many ai teams try to always use the best model to ensure quality. When a new model drops out, they are forced to pay for it, because their competitors will.

↯ Gemma↯ Gpt 4gpt-4gemma

Writing a fine-tuning and deployment pipeline isn't as easy as it looks (Gemma 4 Version) Fine-tune and deploy Gemma 4 on Oxen.ai Google's Gemma 4 dropped in April 2026 with multimodal support (text, image, video, audio), a novel hybrid KV…

↯ Fine Tuning↯ Gemma↯ Gemma 4gemmafine-tuning

Dataset: MMLU subset (DEV+TEST) Llamacpp setting: 3 params only ctx 8192 , seed 42 , fa on Let me know whatelse do you want to see. Thanks.

↯ Gemma↯ Claude 4.6mmlugemmaopus

I’ve been experimenting with running LLMs fully on-device, and managed to get Gemma 4 running locally on an iPhone 13 Pro. This is built on top of a lightweight Swift wrapper I open-sourced: https://github.com/mylovelycodes/LiteRTLM-Swift…

↯ Gemma↯ Gemma 4gemma

If you're using Gemma 4 with external MCP servers in LM Studio and getting this error: Error rendering prompt with jinja template: "Unknown test: sequence" This is a bug in Google's official Gemma 4 Jinja prompt template. LM Studio's Jinja…

↯ Gemma↯ Gemma 4gemmamcp

Open models behind one OpenAI-compatible endpoint. No account yet?

↯ Llama↯ Gemmagemmallamaqwen+1

I've built fully on-device (macOS) meeting transcription/summaries/action-items app. It uses parakeet and gemma 4 or OpenClaw or Hermes Agent to drive main functionalities.

↯ Gemma↯ Gemma 4gemmaopenclaw

Gemma 4 on Cerebras—The Fastest Inference is Now Multimodal Gemma 4 31B is now running at over 1,800 tokens per second on Cerebras Inference. This multimodal model unlocks an entirely new class of applications, from computer use to image-d…

↯ Gemma↯ Gemma 4gemma

How do people measure power usage of GPUs at large (32x) self-hosted setups or small multi-rack setups? I've seen some PDUs which collect and transmit data, but I'm unsure of the processes and if/how people do this on small builds.

↯ Llama↯ Mistral↯ Gemmamistralgemmallama

9 min read 23 hours ago Building a phone agent on a multimodal LLM: dropping faster-whisper and letting Gemma 4 hear the caller directly — a response-time and reply-accuracy benchmark across English, French, and Mandarin Press enter or cli…

↯ Gemma↯ Gemma 4↯ Gemma 4gemma

Okay fun time I got access to two Nvlinked A100s for some research project I benchmarked my work against the Gemma 4 31b-it available through Google, but my dataset is rather massive, so I need to run it on the "local" resources. Basically…

↯ Vllm↯ Gemma↯ Gemma 4↯ Gemma 4↯ Gemma 4↯ Gemma 4vllmgemmaopenai

I seen this one mentioned but it was a source from about 14 months ago. In the age of the Qwen 3.6 and Gemma 4- is there still a use for QwQ 32B?

↯ Gemma↯ Qwen 3.6gemmaqwen

I've fine-tuned Qwen 3.5 0.8B on the dataset provided by Pangram with their EditLens paper. It's available via a Chrome extension; you can just click selected text and it's going to give you the probability distribution of how likely it is…

↯ Llama↯ Fine Tuning↯ Gemmagemmafine-tuningllama+1

Everyone remembers that sneaky download of Gemini Nano earlier this month? and if you talk to it, it will happily tell you it’s a Gemma.

↯ Llama↯ Vllm↯ Gemma↯ Gemma 4vllmgemmallama+1

I am running Gemma 4 31B for a project using LlamaCPP. There is no integrated main model + MTP drafter GGUF.

↯ Gemma↯ Gemma 4gemma

It is suppose to be 2-4x faster but i am only getting 6TK/s on Gemma4-31B . What am i doing wrong?

↯ Llama↯ Gemma↯ Gemma 4gemmallama

Between a solid model from Qwen or Gemma 4, when translating a text, does "thinking mode" significantly boost the quality of the translation, or is the difference negligible?

↯ Gemma↯ Gemma 4gemmaqwen

I would like to dedicate a budget of about 500 euros to upgrade my workstation and run inference on the qwen 3.6 27b and gemma 4 31b models. I currently have an RTX 5060Ti 16GB.

↯ Copilot↯ Llama↯ Vllm↯ Gemma↯ Qwen 3.6vllmgemmacopilot+2

Question in title. Would be awesome to have this on macs, especially q8 or whatever the minimal-loss quant is, since macs can have lots of ram.

↯ Gemma↯ Qwen 3.6gemma

Hi, I recently tried to get llama.cpp with SYCL running on an Arrow Lake system but gave up halfway through since Vulkan is just way easier to set up. But, the pp/tg I'm getting on Vulkan w/ Arc 130T is disgustingly bad - 100 tokens/s for…

↯ Llama↯ Gemma↯ Gemma 4gemmallama

Trying to figure out which is better for LLM. Mainly Gemma 4.

↯ Gemma↯ Gemma 4gemma

I have a GPU server (4 × RTX3090s) that I've been using for research and PoC in the past 2 years. Mostly running vLLM for Qwen, GPT-OSS, and Gemma.

↯ Vllm↯ Gemmavllmgemmaqwen

Does anyone know of any use cases of Gemma Embedding 2? Or is it solely for search?

↯ Gemmagemma

Bench 3 from my 18GB M3 Pro. Bench 2 was the 4B-class post where the comments were mostly right: I gave thinking models a fixed 1024-token cap, Qwen got kneecapped, Gemma E4B needed clearer active-param labeling, and the headline was partl…

↯ Ollama↯ Gemma↯ Qwen 3.5ollamagemmamoe+1

Based on the recent TRiP source code by Carlo Valenti. Ported to Zig and headless Vulkan Compute shaders.

↯ Gemmagemma

Tutorial from the Google guy, I use very similar setup (llama.cpp instead of lmstudio)

↯ Llama↯ Gemma↯ Gemma 4gemmallama

At this moment, the models such as Qwen 3.6 35b/27b crush the competition, yet I can't help, but notice this pattern. While the local RP scene is abundant with the Western model tunes: LLaMA, Mistral (all sizes), Nemo and more recently Gem…

↯ Llama↯ Mistral↯ Gemma↯ Qwen 3.6mistralgemmallama+1

Hey guys, I built a custom vLLM pipeline to run Gemma 4 (31B FP8) and Qwen 3.5 side-by-side locally to see how they actually perform in the wild with preprocessing of audio and images. But of course new model Qwen 3.6 27B came out just whe…

↯ Vllm↯ Gemma↯ Qwen 3.6vllmgemmaqwen

Hi guys, I’ve been running side-by-side experiments on Gemma 4 (31B FP8) and Qwen 3.5 Vision for the last few days using vLLM in Docker to see how they actually handle real-world images and video. A few things I found out: 1.

↯ Vllm↯ Gemma↯ Qwen 3.5vllmgemmaqwen

↯ Gemma↯ Gemma 4gemma

↯ Gemmagemmaqwen

↯ Gemma↯ Qwen 3.6gemmaqwengemini+1

Processing img 8ofni1q6dpvg1... Hello everyone, I’m building an offline RAG system for my company, we are trying to run an app that retrieves infromation from two manuals in an android tablet with the idea of an AI to provide precise answe…

↯ Gemmagemmarag

Hi, I got new MB Pro 24GB/1TB. I've test Gemma 4 26B with ollama, 16k context.

↯ Copilot↯ Ollama↯ Gemma↯ Gemma 4ollamagemmacopilot

Ryzen AI MAX+ 395, Bosgame M5, 128GB LPDDR5x. Proxmox VE 9.1 LXC containers with GPU passthrough.

↯ Llama↯ Minimax↯ Gemma↯ Qwen 3.5minimaxgemmallama

Hello, I have noticed an annoying issue with Gemma 4 26b a4b. It seems like it cannot do multiple think->tool call->think->tool call turns.

↯ Gemma↯ Gemma 4gemma

I'd like to self-host some LLM models but a couple different ones for different usecases, and they don't all fit in VRAM at the same time. So i'm kind of looking for a tool in which i can define "profiles" or "stacks" of LLM's that get loa…

↯ Gemma↯ Qwen 3.5gemmaopenai

Hey there, I noticed something odd when trying out the latest and greatest local reasoning models recently. First, I just noticed it for Qwen3.5, but Gemma 4 seems to do it too: The reasoning traces do that weird thing of starting with "He…

↯ Gemma↯ Qwen 3.5gemma

Hi, my current system hardware RTX 3090 24GB VRAM & Sysrem RAM 64GB using windows 11 been playing around with hermes agent and local llm (Qwopus3.5-27B-v3-GGUF & gemma-4-26B-A4B-it-GGUF) when i try asking the hermes agent to do a task with…

↯ Llama↯ Gemma↯ Gemma 4gemmallamaqwen

Open-weight models, fully deterministic. The same answer every time, byte for byte, at the floor price.

↯ Gemmagemma

About Press Copyright Contact us Creators Advertise Developers Terms Privacy Policy & Safety How YouTube works Test new features NFL Sunday Ticket © 2026 Google LLC

↯ Gemmagemma

Hi, I'm not a fan of chat as the main UI for interacting with AI, so I'm trying something where you interact with LLMs by handling regular files and folders. Agentray is an invisible AI agent.

↯ Gemma↯ Gemma 4gemma

Gemma is Google DeepMind's family of lightweight, state-of-the-art open models. Contents Start Here Models Core Models Variants Inference Local Hosted Fine-Tune Tutorials Demos and Applications Gemma 4 Good Challenge Gemma in Space Researc…

↯ Deepmind↯ Gemma↯ Gemma 4deepmindgemma

blog Gemini 3.7 Flash offered for free Gemini 3.7 Flash is now included with every pad. It joins Gemma 4 31B and Gemini 3.6 Flash: three models in OpenCode, no provider account, no key to paste, usage included with the pad.

↯ Gemma↯ Gemini 3.7gemmagemini

blog Free coding models on every pad Every reachpad now includes two models you can use through OpenCode without bringing an API key: Gemma 4 31B and Gemini 3.6 Flash. Model usage is included with the pad.

↯ Gemma↯ Gemini 3.6↯ Gemini 3.6↯ Gemini 3.6↯ Gemini 3.6↯ Gemini 3.6gemmagemini

So i had been building a side project screenmind ,sort of local ai desktop assistant, and hit a confusing multimodel failure... texts worked,but vision and image failed no crash just returning <unsused49>.

↯ Llama↯ Gemma↯ Gemma 4gemmallama

I started thinking over why doesn't fireworks support voice models. There are really good opensource models available now, like parakeet, kokoro, Qwen ASR etc but no way to use it without managing a bunch of GPUs yourself.

↯ Gemma↯ Gemma 4gemmaqwen

I wanted to share a use case for Noema Overfit, a model-paging system available today in the Noema app. For disclosure, I founded Noema and I am part of the team that has helped develop the system.

↯ Gemma↯ Gemma 4gemma

Hey HN, Henry & Roman here from Cactus. A small, on-device model is fast and private, but sometimes wrong, but frontier models are getting expensive pretty fast.

↯ Gemma↯ Gemini 3.1mmlugemmagemini

I asked Codex 5.6 Sol Medium to one-shot a tree-view version of The Odyssey because I wanted to familiarize myself with it before I see the film. Think of it as an interactive Cliffs Notes where you can stay high level or drill down to the…

↯ Gemmagemmacodex

I set out to find an answer to a completely different question: Does a model, when attempting to solve a cyber CTF (find the vulnerability in this app, and then Capture The Flag) while knowing how many steps it has left, perform differentl…

↯ Security↯ Gemmagemmasecurity

Gemma 4 is now significantly faster in Ollama 0.31 on Apple Silicon via multi-token prediction (MTP), powered by MLX. Performance is now up to 90% faster when used with coding agents, as measured using the Aider polyglot benchmark.

↯ Ollama↯ Gemma↯ Gemma 4aiderollamagemma

I got tired of sending every text I translate to Google/DeepL. Even with all the opt-out options and privacy policies, it never felt right especially for some work documents, personal writing, or anything sensitive.

↯ Ollama↯ Llama↯ Gemma↯ Gemma 4ollamagemmallama

On your phone A small open model, like Gemma 4, runs on the device. Completely offline.

↯ Gemma↯ Gemma 4gemma

Your data stays on your phone. No bank sync track, no AI advice.

↯ Gemmagemma

Gemma-4 31B at 256K Context on a $1,400 AMD GPU — TurboQuant KV Cache on RDNA4 Running **Gemma-4-31B-it with a TurboQuant KV cache and HIP graphs together on AMD RDNA4 (gfx1201) — a combination that crashes out of the box and, to our knowl…

↯ Gemma↯ Gemma 4gemma

Gemma 4 E4B 6bit is now the local model of my choice and loaded 24/7 on my Mac (using @lmstudio), replacing Qwen3, 3.5 4B after ~9 months of usage What an insane model, congrats @GoogleDeepMind 🤠 The new setup replaces his nine-month daily…

↯ Gemma↯ Gemma 4gemmaqwen

Tried Gemma, Qwen and a few others. Need vision and larger context windows for an application I am working on.

↯ Gemma↯ Gemini 3.1gemmagpt-5qwen+1

Apples® to Apples®: MLX vs llama.cpp for Gemma 4 12B on an M1 16GB A matched-quant MLX-vs-raw-llama.cpp benchmark for Gemma 4 12B on one M1 16GB - decode is a tie, both pinned at the bandwidth wall. The cost that differs is startup and CPU…

↯ Llama↯ Gemma↯ Gemma 4gemmallama

Gemma 4 QAT models: Optimizing model compression for mobile and laptop efficiency Since releasing Gemma 4 two months ago, we've been continuously working to expand its capabilities. First, we introduced Multi-Token Prediction (MTP) to acce…

↯ Gemma↯ Gemma 4gemma

Following the announcement in our launch blog, we are releasing Gemma 4 12B, a dense multimodal model with a unified, encoder-free architecture. Gemma 4 12B introduces several milestones for local AI: Traditional multimodal models rely on…

↯ Gemma↯ Gemma 4gemma

gemma-4-12B-it-GGUF Recommended way to run this model: llama-server -hf ggml-org/gemma-4-12B-it-GGUF Then, access http://localhost:8080

↯ Llama↯ Gemma↯ Gemma 4gemmallama

2026-05-05 Gemma 4 26B on consumer-grade 5070Ti GPU A week running Google's Gemma 4 26B as my daily local agent on a single RTX 5070 Ti. No API calls, no cloud, no rate limits.

↯ Gemma↯ Gemma 4↯ Gemma 4gemma

Hey HN, We believe we have the easiest onboarding from signup to being able to spin up coding agents in slack like Stripe, Ramp & Coinbase. Demo of the onboarding: https://www.tella.tv/video/connecting-cord-to-slack-1-19ep Every signup get…

↯ Glm↯ Minimax↯ Gemma↯ DeepSeek 4↯ DeepSeek 4minimaxglmgemma+5

Hoping the community can guide me on this one. I'm on the fence about the following purchase: Refurbished 16-inch MacBook Pro Apple M4 Max Chip with 16‑Core CPU and 40‑Core GPU, 64gb ram for $3,479.00 vs The new 16-inch MacBook Pro Apple M…

↯ Gemma↯ Qwen 3.6↯ Qwen 3.6↯ Qwen 3.6↯ Qwen 3.6↯ Qwen 3.6gemma

It started with I just want to make a chat app like roleplay with characters but Gemma 4 26B A4B Q4_KM doesn't have info some old character so I crawl back to those online services as those model is much bigger parameter and quite update i…

↯ Gemma↯ Gemma 4↯ Gemma 4gemma

I've spent the last year building Charm, a native macOS menu bar app that corrects spelling, fixes grammar, and predicts your next word. Three features: - Spells: NSSpellChecker plus a local LLM for context-aware corrections (catches "defi…

↯ Gemma↯ Qwen 2.5gemmaqwen

Hey all, While nvfp4 still seems to be a work in progress, the latest version of vllm 0.21 finally has mtp working for gemma. With all the talk of qwen being badass I thought I would revisit gemma.

↯ Vllm↯ Gemma↯ Gemma 4vllmgemmaqwen

I have a build with 2 x MI50 32GBs and 64 gigs of DDR4 (bought before rampocolypse for ~630 USD total, I’m not rich) and I’m not gonna upgrade it for a long while. Are there any good MOE models that are around 60B in parameters so I can ma…

↯ Gemma↯ Gemma 4gemmamoe

I really benefit a lot from having claude code act as a personal assistant - it reminds me of things I need to do, helps me focus on what matters, and keeps me accountable on making sure I don't let important things slip But I am well awar…

↯ Gemmagemmamcpclaude-code

Hello everyone, I developed the snn architecture from scratch based on the human brain. I had several successful launches of training spike models from scratch and I also had an idea: what would happen if I took the gemma 4 model and conve…

↯ Gemma↯ Gemma 4gemma

So I'm brand new to this scene but I'm using Claude to help me fine tune a model for a startup idea I have in the Healthcare space. I have been working with the 27-35B parameter mdoels (Qwen3.6, Gemma 4) and the couple of 120B+ models (Qwe…

↯ Minimax↯ Gemma↯ MiniMax 2.7minimaxgemmaqwen

Grafting a Speech Head onto Gemma 4 E4B For a Discord buddy, the tempting model shape is small, fast, and multimodal. It should hear the call, see the game, read the chat, and respond quickly enough that the moment is still alive.

↯ Gemma↯ Gemma 4gemma

I am trying out some tools that are from the openwebui community that I have directed towards my LM Studio server instance. It seems really hit or miss on most of the tools being called by the LLM or not.

↯ Gemmagemma

I have setup a workflow to process website translations with Gemma 4, I just host it on LM Studio, and a custom Python wrapper iterates through and runs overnight. My question is..

↯ Gemma↯ Gemma 4gemma

Hey everyone! I’ve just uploaded a GGUF version of nvidia/Gemma-4-26B-A4B-NVFP4.

↯ Llama↯ Gemma↯ Gemma 4gemmallama

Don’t miss what’s happening People on X are the first to know. Log in Sign up Post Conversation Software Mansion @swmansion On-device @googlegemma in React Native with react-native-executorch Coming to the library very very soon!

↯ Gemmagemma

Hello, posting again on this since my last post was removed. I am working on an AI agent solution to help me with my multiple daily tasks for different business activities; a few rental properties, a manufacturer trying to enter the Mexico…

↯ Gemma↯ Gemma 4gemmaopenclawopenai

llmconfig Local Large Model Config — manage local inference with llama.cpp, stable-diffusion.cpp, and whisper.cpp from a single YAML file and a single CLI. llmconfig up gemma # or just: llmc up gemma ✓ gemma is ready at http://127.0.0.1:80…

↯ Llama↯ Gemmagemmallama

I've been experimenting with using Ollama to run Claude Code locally with models like Gemma 4, thinking I could avoid API costs. However, I quickly realised these models aren't really optimised for Claude Code's agentic workflows — they te…

↯ Ollama↯ Gemma↯ Gemma 4ollamagemmaanthropic+2

It is pluggin made for ONLYOFFICE, much simpler than copy-paste from webui. PS.

↯ Gemma↯ Qwen 3.6gemmaqwenopenai

I would like to experiment but before investing loads of money, I do have a MacBook Pro with 32GB RAM, M2 Pro. Which model would maximize versatility given this hardware?

↯ Gemmagemmaopenclawdeepseek+2

I've got to the point where I need some help. I'm trying to run Qwen 3.6, and it will eventually fall into a loop where it's just outputting "/" symbols when it's "thinking".

↯ Gemma↯ Qwen 3.6gemmaqwen

Whew! Big work today getting optimized llama.cpp MIPS R8000 kernel running on the SGI Power Challenge deskside from 1995 with Gemma 3 270M.

↯ Llama↯ Gemmagemmallama

https://github.com/ggml-org/llama.cpp/pull/22196 And somehow we already got some GGUFs for it! https://huggingface.co/CISCai/gemma-4-31B-it-NVFP4-turbo-GGUF https://huggingface.co/stevelikesrhino/gemma-4-31B-it-nvfp4-GGUF (the below one is…

↯ Llama↯ Gemma↯ Qwen 3.5gemmallama

Gemma-4 is great. On a MacBook M5, using lm-studio, the MLX versions (specifically looking at https://huggingface.co/lmstudio-community/gemma-4-26B-A4B-it-MLX-8bit) rock.

↯ Gemma↯ Gemma 4gemma

I now use (mostly) Gemma 4 and Qwen 3.5 models *. And seems that all of them, after context grows a bit, after providing long output for me and getting a short prompt in response, are starting to process many new tokens as input and I have…

↯ Llama↯ Gemma↯ Qwen 3.5gemmallamaqwen

TLDR: I've been running gemma4 e2b extensively on iOS with llama.cpp and found some interesting quirks and info you guys may like! These are specifics for the iPhone and what I've found worked across 20+ devices.

↯ Llama↯ Gemma↯ Gemma 4gemmallama

A practical configuration for a 32 GB M5 Mac that still needs to remain usable Running large language models locally has become surprisingly practical on Apple Silicon. With a modern Mac, Ollama, and a carefully quantized GGUF model, it is…

↯ Ollama↯ Gemma↯ Gemma 4ollamagemma

I made a simple app using openrouter, hoping to use the new small qwen models (the a3b moe and the 27b dense one), but they aren’t listed. Also, I swear some qwen3.6 models that were listed before are missing now.

↯ Gemma↯ Qwen 3.6gemmamoeqwen

Gemma 4 makes five quiet departures from the standard transformer recipe. QK-norm instead of 1/√d, partial RoPE on global layers, per-layer input gating, KV sharing across layers, and an MoE that sits alongside the MLP rather than replacin…

↯ Gemma↯ Gemma 4gemmamoe

↯ Llama↯ Gemma↯ Qwen 3.6gemmallamaqwen

I have Gemma4-E2B working within home assistant as STT, and E2B seems fast and accurate for STT (maybe a bit better than Parakeet), however, it responds with the entire thought process: https://preview.redd.it/v8zhb5elltvg1.png?width=599&f…

↯ Llama↯ Gemma↯ Gemma 4gemmallama

Hardware: Intel Core Ultra 7 258V, 32GB Unified Memory. Model: Qwen 3.6 35B A3B (Quant: Q3_K_S) via LM Studio.

↯ Gemma↯ Qwen 3.6gemmamoeqwen

Vercel Security Checkpoint | cle1::1776468758-lOAcIwtVVUa8cG9OLlcTtnlZwlvTxsBe

↯ Gemma↯ Gemma 4gemmaclaude-code

could not extract summary

↯ Gemma↯ Gemma 4gemma

Hello, everybody! I'm building and hybrid database with Qdrant and Neo4j for a few personal projects.

↯ Ollama↯ Gemma↯ Gemma 3ollamagemma

I currently am looking for models to fit into my single DGX Spark for use. I have an RTX Pro 6000 and also a 5090 as well that I'm considering using in combination if the DGX Spark is too slow, but the intent here is to play around with Op…

↯ Minimax↯ Gemma↯ Qwen 3.5minimaxgemmaopenclaw

Hi. I've been looking at ollama cloud's Pro offering ($20), which says "Run 3 cloud models at a time".

↯ Ollama↯ Minimax↯ Gemma↯ Gemma 4minimaxollamagemma+1

For example, in LMStudio the official Gemma 4 is a GGUF that has Vision, Reasoning, and Tools flags. But the MLX version does not.

↯ Gemma↯ Gemma 4gemma

Does anyone know of a good edge local llm that is good in math's. I tried Gemma 4 E2B, microsoft phi mini reasoning but both can't answer some basic apti question's.

↯ Gemma↯ Gemma 4gemma

I'm trying to replace openai codex which i used for development all the time, with gemma4 on 4090, small tasks it solves quite impressively, but i need to have some agent. So I tried to connect 31b to cline and to aider and it didn't reall…

↯ Ollama↯ Gemma↯ Gemma 4aiderclineollama+3

TL;DR Three facts and one command. Bench — Mac mini M4 · 24 GB unified memory · macOS 26.5 · Ollama 0.34.0 · gemma4:26b Q4_K_M (MoE, 25.2B total / ~4B active) · measured 2026-09-13 ollama ps )sudo sysctl iogpu.wired_limit_mb=20480 # raise…

↯ Ollama↯ Gemmaollamagemmamoe

Gemma 4 E4B LLM Wiki Turn your notes into a living Karpathy LLM wiki. Chat with your vault, connect ideas, find gaps, and generate quizzes & flashcards — with Gemma 4 E4B running locally inside Obsidian.

↯ Gemmagemma

PicoLM is an LLM inference engine written in C99. It currently supports llama-2, GPT-2, Qwen 3.6/3.8(+MoE) and Gemma-3n models.

↯ Llama↯ Gemmagemmamoellama+2

I spent the last several months building a photo journal that runs a full multimodal LLM — Gemma 4 E4B — entirely on an iPhone. No cloud inference, no accounts, no analytics, full privacy; the App Store privacy label is "Data Not Collected…

↯ Gemma↯ Gemma 4gemma

Apertura A from-scratch Objective-C++ / MLX rebuild of Google's Gemma-4 for Apple Silicon — built to be inspected, observed, and experimented with. Most language models are run behind glass: you send a prompt and get an answer, with no way…

↯ Gemma↯ Gemma 4gemma

Gemma 4b vs Gemini Flash: You Don't Need Frontier Models For Tool Calling Workflows What our Acebench analysis showed about the models Most of what makes an AI assistant useful isn’t prose — it’s calling tools: booking the meeting, adding…

↯ Gemmagemmagemini

Imagine building a fully autonomous robot that can see, hear, and react to its environment in real time, completely offline on a single compact device like Raspberry Pi. Edge AI unlocks this exact autonomy.

↯ Gemmagemma

About Press Copyright Contact us Creators Advertise Developers Terms Privacy Policy & Safety How YouTube works Test new features NFL Sunday Ticket © 2026 Google LLC

↯ Gemma↯ Gemma 4gemma

the-veil-12b The first official model for the nl-veil harness. A Gemma 4 12B derivative, fine-tuned on nl-veil's own tool surface.

↯ Gemma↯ Gemma 4gemma

Put together an extensive open source test suite for voice assistants. You can view it including current leaderboard at: https://git.cicero.sh/aquila/ha-voice-test-suite/ Tests are reproduceable, with clear instructions on how to run them…

↯ Gemma↯ Sonnet 5gemmagroksonnet

I'm the founder of Scalattice, this is my second company, third total product. I'm a 2x founder building some challenging software, some easy software, and some curiosity based tools that I've just always wanted to be a part of!

↯ Llama↯ Gemmagemmadeepseekllama+1

Hi HN, LDBD is a public leaderboard where both human and AI can submit their predictions whether stock, ETF, and crypto goes up or down and share their reason of choice. This service starts from one question: does anyone or any AI can real…

↯ Gemmagemmachatgpt

LLM Red Team Lab A hands-on kit for educational, authorized red teaming of any locally-run LLM. It works with any OpenAI-compatible model — Llama, Mistral, Qwen, Gemma, DeepSeek R1, and more — and covers the two ways an LLM system gets exp…

↯ Security↯ Llama↯ Mistral↯ Gemma↯ Jailbreakred-teammistraljailbreak+6

Experimental Godot 4.7 project running Gemma 4 inference entirely in GDScript and Vulkan compute shaders.

↯ Gemma↯ Gemma 4gemma

Hi HN, Roman and Henry here from Cactus (https://github.com/cactus-compute/cactus). We just shipped the biggest upgrade to our on-device inference platform: - Built-in model confidence-based routing to hand off inference runs to the cloud…

↯ Gemma↯ Gemma 4gemma

Today we’re releasing ZML/LLMD. It’s a self-contained inference server that runs LLaMa, Gemma, Qwen and Mistral LLMs transparently on 5 architectures: NVIDIA CUDA, AMD ROCm, Google TPU, Intel oneAPI and Apple Metal.

↯ Llama↯ Mistral↯ Gemmamistralgemmallama+1

mulot [-4285F4?logo=googlechrome&logoColor=white)]() Agentic AI web pentester that drives a browser. An open-weights LLM (GLM-5.2, Gemma or Qwen) drives a real headless Chromium through a Burp-style toolkit and works a target the way a hum…

↯ Glm↯ Gemma↯ GLM 5.2glmgemmaqwen+2

Hi C.J. here, I made a free app that does on device translation using Gemma Translate 4B AI model repackaged for iOS.

↯ Gemmagemma

MetalChat - LLM inference for Apple Silicon MetalChat is a Metal-accelerated C++ framework and command line interpreter for inference of Meta Llama and Google Gemma models. [!IMPORTANT] The library API and CLI are under active development,…

↯ Llama↯ Gemmagemmallama

1. Coding 2.

↯ Gemma↯ Qwen 3gemma

A from-scratch build book Small Kernels Learn the Zig programming language and how large language models actually work — by writing every piece yourself, from a single dot product to GPT-2, a modern Gemma, and the doorway to Whisper. I ori…

↯ Gemmagemma

About Press Copyright Contact us Creators Advertise Developers Terms Privacy Policy & Safety How YouTube works Test new features NFL Sunday Ticket © 2026 Google LLC

↯ Gemmagemma

Discover how developers at the Google Tunix Hackathon trained small Gemma models to reason under a limited compute budget. Learn the winning, open-source post-training recipes—combining SFT, GRPO, and SimPO—to build your own structured rea…

↯ Gemmagemma

Has anyone got it to work? I tried it with dense models (eg qwen 27b, gemma 31b, mistral 128b) since that's where I need it most, but it always core dumps.

↯ Llama↯ Mistral↯ Gemmamistralgemmallama+1

Been experimenting with an idea — what if your AI assistant actually remembered everything you did on your computer? Not stateless chats, but real persistent context.

↯ Llama↯ Gemma↯ Gemma 4↯ Gemma 4gemmallamacursor+1

Built a local ReAct-style calculator agent with 6 tools: add subtract multiply divide modulo etc. The setup is: orchestrator agent dynamic tool selection ReAct loop tools exposed as functions Problem: Even when the user asks multi-step ari…

↯ Gemma↯ Qwen 3.5tool-callinggemma

Currently experimenting with building a React-style looping agent system using small LLMs like Qwen 3.5 9B and Gemma 4 (E2B), and I wanted to ask if anyone here has worked on something similar. Current setup: Using LangGraph Around 5 tools…

↯ Gemma↯ Qwen 3.5gemmaqwen

Gemma 4: A new, budget-focused model in Posit AI Gemma 4 is now available in Posit Assistant via the Posit AI provider. It's priced at a tenth of the price of Claude Sonnet 4.6 and less than a third of the price of our current cheapest off…

↯ Gemma↯ Sonnet 4.6gemmasonnet

I’ve tried openclaw locally for about a month. Hardware: M5 Pro w/48 gb ram.

↯ Ollama↯ Gemma↯ Qwen 2.5ollamagemmaopenclaw+1

i want the best installation that fit my use and my low-compute H.W , i want to run small to above small llm like "qwen" 2b ,4b and 27b , and "gemma" 31B. rely completely on only old CPU 4th.gen i7 with that few 32gb 'slow' ddr3.

↯ Llama↯ Gemmagemmallamaqwen

running gemma e2b via llama-server for continuous background tasks on a 1650 4gb. works great initially but after maybe 30-40 calls the outputs start getting noticeably worse — shorter responses, missing fields in json output, sometimes ju…

↯ Llama↯ Gemma↯ Gemma 4gemmallama

The other day I posted about playing one night werewolf on my custom made UI via tool calls. Since then I’ve played a few games and improved the prompts.

↯ Gemma↯ Qwen 3.6gemmaqwen

Hello, I built myself a news feed which scores and summarizes research papers along with relevant AI news from Huggjngface, Reddit, hacker news etc. I used Claude code to build the whole thing.

↯ Gemmagemmaclaude-code

currently in order to use MTP one needs to enable it in the starting argument of llama server. --spec-type draft-mtp --spec-draft-n-max 2 But then other models that do not use MTP currently like Gemma or basically all other models fail to…

↯ Llama↯ Gemmagemmallama

Not deeply technically fluent but have ran few models locally before, around the time before gemma 4 dropped. I tried some low quant of qwen 2.5 coder and after some tinkering I got it to run but it was just so slow, obviously.

↯ Gemma↯ Qwen 3.6gemmaqwen

I can't decide that Qwen 3.6 35b q4 (130k context) or Gemma 4 26b q4 (95k context) is better for C# coding with 24GB VRAM. Please share your experiences!

↯ Gemma↯ Qwen 3.6gemmaqwen

Recent Developments in LLM Architectures: KV Sharing, mHC, and Compressed Attention From Gemma 4 to DeepSeek V4, How New Open-Weight LLMs Are Reducing Long-Context Costs After a short family break, I am excited to be back and catching up o…

↯ Gemma↯ DeepSeek 4gemmadeepseek

I am curious if anyone here has tried doing this, I did a bit of digging and it seems like it would be easier to do then I first thought and would like to ask ask for correction if my assumptions are wrong. Here is how I would go about it:…

↯ Gemmagemma

Hi, has anybody succeeded in running llama.cpp with Gemma 31b dense and Gemma e4b as draft model, and simultaneously inhibit the voice recognition feature? Is it even (theoretically) possible?

↯ Llama↯ Gemma↯ Gemma 4gemmallama

I spend most of my day writing prompts to Claude. Read a study recently that said people speak ~3x faster than they type, which lands differently when "writing" is basically your whole workflow.

↯ Gemma↯ Gemma 4gemmaclaude-code

I have been curious to see how small LLMs like Gemma-4-E2B-it run on a flagship smartphone (S25+ with Snapdragon 8 Elite) in terms of prompt processing and token generation. I have created a script that uses llama-cli and I achieve 48 tps…

↯ Llama↯ Gemma↯ Gemma 4gemmallama

Hi, i have a 5090 and i was tyoing around with hermes-agent. To utilize 128K i thought about switching from LM Studio to llama-cpp (the turboquant fork) expecting better tok/s and also saving some VRAM from context quantization.

↯ Llama↯ Gemma↯ Gemma 4gemmallama

My partner uses Duolingo for learning and practicing languages, but has been getting increasingly sick of it. I decided to experiment with whether local models would be good for creating and grading language exercises.

↯ Gemma↯ Gemma 4gemma

My setup: Windows 10/11 i7 12700K | RTX 3090 TI | 96GB RAM Local server: LM Studio Models: Qwen 3.5/3.6 27B|35B Q5 UD K XL + Gemma 4 31B| 26B Q4 UD K XL Up until this point, I've only used sota models for coding. When Qwen 3.5 dropped, it…

↯ Gemma↯ Qwen 3.5gemmaopenclawqwen+2

Yes, for material that is an hour long, there is no getting around tools like Whisper - or something even better. However, for transcribing short snippets, Gemma works very quickly and reliably- even in foreign languages.

↯ Gemma↯ Gemma 4gemma

Per my experience, Opus 4.7 is so slow, Sonnet 4.6 is ok. I am also using local models wondering if Claude is already leveraging drafters/assistant AIs and despite that so slow or not?

↯ Gemma↯ Sonnet 4.6gemmasonnetopus

Gemma Chat Vibe code without the internet. A local coding agent powered by Google's Gemma 4 — runs entirely on your Mac via Apple's MLX framework.

↯ Gemma↯ Gemma 4gemma

We ran GPT-5.4 vs Gemma 3 27B on 2 prompts. One open-source model won.

↯ Gemma↯ GPT 5.5gemmagpt-5

Private offline AI chat for macOS. Free for 14 days, then €29.99 once.

↯ Gemmagemma

Hey folks, I posted here a few months back about how I was basically working for Claude -- pasting the same emails, re-explaining the same backstory, being its memory across every chat. Today I'm launching Contextify.

↯ Gemma↯ Gemma 4gemma

In my line of work, PDF documents tend to be combinations of text, math formulas, tables and images. llama.cpp added support for PDFs a few months ago, but I believe it treats PDFs either as text (discarding everything else), or as images.

↯ Llama↯ Gemma↯ Gemma 4gemmallama

Hi guys 👋 When I started my adventure with Qwen 3.6 27B I felt wow.... Now when I connect it with Gemma 4 I'm feeling more wow...

↯ Gemma↯ Qwen 3.6gemmaqwen

I have a 4060 (8GB Vram) and 16GB of ram wondering which models could fit in my setup for coding, the new Qwen 3.6 and Gemma 4 MoE models look good but might not fit, wondering about your experiences

↯ Gemma↯ Qwen 3.6gemmamoeqwen

Hey everyone, I’ve been thinking about a project idea and I’d love to get your feedback. The idea is to take a 1TB SSD and turn it into a fully portable AI system.

↯ Ollama↯ Llama↯ Mistral↯ Gemmamistralollamagemma+1

Libraries llama-cpp-python How to use unsloth/gemma-4-E4B-it-GGUF with llama-cpp-python: !pip install llama-cpp-python from llama_cpp import Llama llm = Llama.from_pretrained( repo_id="unsloth/gemma-4-E4B-it-GGUF", filename="gemma-4-E4B-it…

↯ Llama↯ Gemmagemmallama

Whenever I write here that I use gemma 31B I get answers that qwen 27B is better. I switched in the pi from gemma 31B Q5 to qwen 27B Q8 and generally I manage to code, document and run tests but somewhere after exceeding 100k context qwen…

↯ Llama↯ Gemma↯ Qwen 3.6gemmallamaqwen

offload-mcp MCP server for offloading routine coding-assistant work to a cheaper model. The default model chain uses Gemma because the models are useful, open, and fun to experiment with.

↯ Gemmagemmamcp

Hey everyone, I built a tool that creates movie recap videos automatically using local models. The problem: making recap videos takes forever.

↯ Ollama↯ Mistral↯ Gemma↯ Gemma 4mistralollamagemma+3

I saw someone on this forum demonstrate using gemma 4 - live streaming audio and video from his webcam to it asking it what it was seeing. It was pretty great but I cant find that post anymore and I can't find a good repo on github where I…

↯ Gemma↯ Gemma 4gemma

Hey there people. So let's talk about GEMMA 4 per layer embeddings.

↯ Gemma↯ Gemma 4gemma

We run Qwen3.6-27B-FP8 at AI Router Switzerland and hit two issues, so I wanted to share in case anyone else runs into them. FP8 KV cache produces silent garbage output with radix cache prefix hits (PR #24198 — ✅ approved) We were running…

↯ Gemma↯ DeepSeek 4gemmadeepseek

I experienced this with Q4 and Q3 versions of Qwen3.6-35B-A3B and Gemma-4-26B-A4B. It starts saying things which sound similar in thinking mode: I must do ....

↯ Gemma↯ Qwen 3.6gemma

I can’t really run 70B models on my current setup, but I’m curious haha

↯ Gemmagemmaqwen

Some of the larger models (like Llama) weren't available on OpenRouter, so I had to work with what was there. Best small model: Gemma 4 26B For its size, I think it had the best output.

↯ Glm↯ Llama↯ Minimax↯ Gemma↯ DeepSeek 4minimaxglmgemma+2

I have an RTX 4060 8GB(+16GB RAM) laptop, and when asking Gemini or ChatGPT, they say the Gemma 4 Q4 K M is the best fit for my hardware with Context Length around 16k-32k. However, in practice, after loading even a higher quantization lik…

↯ Gemma↯ Gemma 4gemmageminichatgpt

I’ve been building a local-first AI runtime that wraps local LLMs with a σ-gate — a measurement layer that decides ACCEPT, RETHINK, or ABSTAIN before an answer reaches you. The idea: local models should be able to say “I don’t know” instea…

↯ Gemma↯ Gemma 3gemmaqwenmcp

With the release of Gemma 4 models and a slew of open weight/source models subsequently, some of the workflows like drafting emails/ trivial coding tasks have become possible. I’m exploring the possibility of integrating some of the powerf…

↯ Gemma↯ Gemma 4gemma

QVAC-Fabric Gemma 4 Architecture Patch Adds full Gemma 4 (gemma4) architecture support to QVAC-Fabric, Tether's llama.cpp fork. Base: QVAC-Fabric temp-upstream branch Target: All Gemma 4 variants (E2B, E4B, etc.

↯ Llama↯ Gemma↯ Gemma 4gemmallama

I've been running local LLMs since Qwen 3.5 dropped and I was really impressed by what we could run on consumer hardware. Fast forward another two months and we have gotten a handful more gems such as Gemma 4 and Qwen 3.6, so I wanted to p…

↯ Llama↯ Gemma↯ DeepSeek 4gemmadeepseekllama+1

The gemma 4 E4B and E2B models have built-in multimodal capabilities. However, as far as I am aware, llama.cpp does not have proper support for vision and audio inputs (specially audio) for these models as of now.

↯ Llama↯ Gemma↯ Gemma 4gemmallama

https://huggingface.co/mlx-community/Qwen3.6-35B-A3B-OptiQ-4bit https://huggingface.co/mlx-community/Qwen3.6-27B-OptiQ-4bit https://huggingface.co/mlx-community/gemma-4-26B-A4B-it-OptiQ-4bit https://huggingface.co/mlx-community/gemma-4-31B…

↯ Gemma↯ Qwen 3.6gemma

Using Google's Gemma 4 E4B local AI model to Reverse Engineer a simple Crackme I was playing around with the new Gemma E4B open weights local model which Google released, and to my surprise I was seeing a great deal of success in using it…

↯ Gemma↯ Gemma 4gemma

I have a Macbook Air M3 with 24gb RAM. The other day, I wanted to try running an LLM locally for the first time ever.

↯ Gemma↯ Opus 4.6gemmachatgptopus

Spent a week doing LoRA fine-tuning on Gemma 4 E2B (~5.1B total params, ~2B active in text decoder) for a narrow Python code-generation task. Bad outputs went from ~5% to 0% (greedy) and 1.5% (sampled) across 134 tests.

↯ Fine Tuning↯ Gemma↯ Gemma 4gemmafine-tuning

3090 (24G) + 32G DDR4 Currently running --mmproj mmproj-BF16.gguf --chat-template-kwargs '{"enable_thinking":true}' \ --flash-attn on \ --cache-type-k q4_0 \ --cache-type-v q4_0 \ -np 1 \ -c 160000 \ --jinja at 26B-A4B-it-UD-Q5_K_XL and ge…

↯ Gemma↯ Gemma 4gemma

OpenAI, Gemma, Kimi K2, or run locally. Free via OpenRouter.

↯ Cowork↯ Gemmagemmacoworkopenai

I am currently working on a system to translate books/web novels. I got a working prototype, but now I am looking into optimizing it.

↯ Vllm↯ Gemmavllmgemma

Hey everyone, I run llamacpp precompiled with CUDA 12.4 on Windows 11 with a RTX 4090. With small models like gemma-4-E4B everything runs fine, but as soon as I run a bigger model like Qwen3.6-27B (IQ4_NL) or a medium sized model with larg…

↯ Gemma↯ Qwen 3.6gemma

Always been stuck with models that fit on my 16gb .... Going to have about a week for free with 4x rtx6000pro .

↯ Vllm↯ Gemma↯ Gemma 4vllmgemmaclaude-code

Hi everyone, as a power user I hit Claude Code's usage cap too often I wanted to set up my own local model, however I only have RTX 5070 with 12 GB of VRAM so the only realistic option was Gemma 4 with effective 4B params. When I tried to…

↯ Gemma↯ Gemma 4aidergemmaagentic+1

+30% avg accuracy lift on classification & extraction tasks vs. base Gemma ~7 days until your first auto-improvement run lands in production 0 lines of fine-tuning code you have to write, ever $0/retrain starting price.

↯ Fine Tuning↯ Gemmagemmafine-tuning

I gave 9 local models the same flight combat sim prompt. The results broke a few of my assumptions about quant providers and parameter count.

↯ Gemma↯ Qwen 3.6gemmamoeclaude-code

The difference is quite big: likes downloads last month finetunes Qwen3.5-27B 952 3,233,034 263 Qwen3.5-35B-A3B 1,397 3,977,637 87 Qwen3.6-35B-A3B 1,115 458,436 60 gemma-4-31B 323 343,895 13 gemma-4-26B-A4B 227 118,464 13

↯ Gemma↯ Qwen 3.6gemmaqwen

Why does Gemma 4 e4b from Google AI Edge Gallery on Android weigh only 3.6 gigs, while the one from Unsloth (gemma-4-E4B-it-UD-Q2_K_XL.gguf) weighs 3.7, and for some reason the model image in litertlm format extracted via adb from Google A…

↯ Gemma↯ Gemma 4gemma

I was building a dedicated-vision-model feature for an open-source browser agent and wanted to figure out which local model to actually recommend. Wrote a small probe that sends the same image + same system prompt + same params (temperatur…

↯ Llama↯ Gemma↯ Qwen 3.6gemmamoellama+1

↯ Gemma↯ Gemma 4gemma

↯ Gemma↯ Gemma 4gemma

↯ Vllm↯ Gemma↯ Gemma 4vllmgemmamoe

↯ Gemmagemma

↯ Gemma↯ Gemma 4gemma

↯ Gemmagemma

So the question I've seen posed many times in /r/singularity is if the Gemini models are actually that bad at coding compared to their benchmarks, or whether the harness used makes an absolutely gigantic difference in model performance. Gi…

↯ Gemma↯ Gemma 4gemmageminiclaude-code

I have 3090ti and i will add 3080ti to my system soon. With 3090ti only, i found it little bit slow to run gemma 4 26b 4q.

↯ Gemma↯ Qwen 3.5gemmaopenclaw

If the LLM supports video—which most VLLMs nowadays do—then try the following prompt with the accompanying video: With the given video, which is about 16 seconds long, your task is to write JavaScript for an animation that faithfully repli…

↯ Gemma↯ Gemini 3.1gemmaqwengemini

Gemma 4 just dropped on Modular, Day Zero! Read More → Inference Products Shared Endpoints Access frontier models via an API Dedicated Endpoints Mission critical reliability Custom models Your model, peak performance Deployment Options Our…

↯ Gemma↯ Gemma 4gemma

Published: April 16, 2026 This is an encyclopedia of some core content from Biology and Health Sciences, Physical Sciences, and Technology. It contains 2,259 small entries of about a paragraph each.

↯ Gemma↯ Gemma 4gemma

I’m trying to reduce my reliance on Claude. I have a 5090/128GB RAM.

↯ Gemma↯ Qwen 3.5gemmaqwensonnet

Hey everyone, I just shipped an iOS app that runs local AI models. Current has 12 models: Gemma 4, Llama 3.3, Qwen3, DeepSeek R1 Distill, Phi-4, etc.

↯ Llama↯ Gemma↯ Llama 3.3gemmadeepseekllama+1

Im searching for build version of LiteRT LM framework can use and utilize the NPU of the RKNN 3588. It would be great since I can run gemma 4 e2b model using this framework on the machine, because I wont have to migrate my codebase from li…

↯ Llama↯ Gemma↯ Gemma 4gemmallama

I've been testing a few models lately and I'm running into a weird issue with the bigger Qwen3.5s. Tested: Gemma 4 26B Qwen3.5 9B Qwen3.5 27B Qwen3.5 35B The 27B and 35B are driving me nuts.

↯ Gemma↯ Qwen 3.5gemma

Hi everyone, I'm trying to run a local LLM via Ollama on a Hetzner cax21 VPS (ARM64, 4 vCPUs, 8GB RAM, 80GB SSD). I have Ollama running successfully via Coolify.

↯ Ollama↯ Gemmaollamagemmaqwen

could not extract summary

↯ Gemmagemma

I own an Oppo Find X3 Pro (Snapdragon 888, 12/256 GB, Android 14.0) unused because of 3 green vertical lines on the screen and poor battery. I tried Google AI Edge Gallery with Gemma-4-E2B-it and it performs well so I thinked: "why don't t…

↯ Llama↯ Gemma↯ Gemma 4gemmallama

I'm currently vibe-coding (I'm new to vibe-coding) with Gemma 4 4EB Q4 and Qwen 3.5 9B Q5 (KV is quantized to 4 bits with new Google TurboQuant implemented in llama.cpp - I use koboldcpp and release said it's automatically activated): the…

↯ Llama↯ Gemma↯ Qwen 3.5gemmallamaqwen

🌟 Gemopus-4-26B-A4B-it [!NOTE] Gemopus is an attempt at fine-tuning Gemma 4 with a core philosophy of "stability first". While preserving the original reasoning order of Gemma 4 as much as possible, we conducted targeted refinements for an…

↯ Fine Tuning↯ Gemma↯ Gemma 4gemmafine-tuning

I was building an Android app and integrated Gemma 4 E2B directly using LiteRT-LM. On-device translation, zero server cost, the dream setup.

↯ Gemma↯ Gemma 4gemma

Hello, I've seen reviews that gemma 4 31b base is very good at roleplaying. But I can't find the gguf version of the basic gemma 4 anywhere.

↯ Gemma↯ Gemma 4gemma

Hey folks, I've been tinkering with Gemma 4 and absolutely the fact this model can run locally on Android phone! I am experienced fullstackdev, open to solve any real-world problem that has an impact.

↯ Gemma↯ Gemma 4gemma

I ran Gemma 4 as a local model in Codex CLI | by Daniel Vaughan | Google Cloud - Community | Apr, 2026 | Medium Sitemap Open in app Sign up Sign in Get app Write Search Sign up Sign in Google Cloud - Community · A collection of technical a…

↯ Gemma↯ Gemma 4gemmacodex

Hello, I've been trying for several days to train Gemma-4 for extracting data from a string and convert it into a structured JSON. I've tried a fair amount of different configurations, I've tried Unsloth studio and Llamafactory, but in eac…

↯ Gemma↯ Gemma 4gemma

Who the judge is can affect an LLM-as-judge result, but measuring that effect without confusing it with candidate quality is difficult. We study four open-weight families (Llama 3.1, Qwen 2.5, Gemma 2, and Yi 1.5) in a fully crossed pairwi…

↯ Llama↯ Qwen 2.5↯ Gemmagemmallamaqwen

Sparse autoencoder (SAE) features are increasingly used to explain and steer language-model behavior, but it remains unclear whether a feature found in one language context plays the same causal role when processing prompts in another lang…

↯ Gemmagemma

About two weeks ago I started an experiment: instead of prompting a local model to play a character, give it a folder and let it become one. The idea is simple.

↯ Ollama↯ Gemma↯ Gemma 4ollamagemma

hi, im desperate. at this point i dont know whats happening.

↯ Gemmagemmachatgptclaude-code

This paper proposes a layer bit allocation method for Gemma-3-1B, formulating the problem as performance maximization (latency decrease) given a degradation budget constraint (allowable level of generation quality loss). This approach is d…

↯ Gemmagemma

I’ve been working on a small project called gemma4.c. The idea is pretty simple: you can download a modern language model, compile one 700-line C file, and have it generate text on an ordinary CPU.

↯ Gemma↯ Gemma 4↯ Gemma 4↯ Gemma 4↯ Gemma 4↯ Gemma 4gemma

specs hardware: M4 Max 128GB Studio inference engine: oMLX & lllama.cpp insights it still very early, so had to disable oMLX K/V caching, qwen4_exp architectureis not yet supported + the obvious n-grams with which the whole 4 bit quant tak…

↯ Gemma↯ Qwen 3.8gemmaqwen

I heard qwen3.8 27b is only good for coding really. Is that true I could do qwen 3.5 27b but qwen3.6 27b doesn’t fit on my Vram at IQ_XS I feel Gemma 4 31b might be good but it’s kinda not fitting in vram unless I go iQ3xxs and the qat wit…

↯ Gemma↯ Qwen 3.8gemmaqwen

Hi all, I'm looking for the best model for a hobby project and trying to make sense of the various data I came across. I know benchmarks do not often translate to the real world, especially to your particular use case (whatever it may be).

↯ Gemma↯ Qwen 3.8gemmaqwen

So I was testing this technique of runtime steering on tiny versions of Qwen 3.5 and Gemma 4 (2B and 4B). Basically, without changing the weights (like with Heretic/ablation, for example), we steer the model in the opposite direction of a…

↯ Gemma↯ Qwen 3.5↯ Qwen 3.5gemmaqwen

I ran 11 uncensored variants of Gemma 4 12B that I grabbed from huggingface, sorting by downloads. 10 full abliterations plus 2 LoRA adapters which were requested to be added in the comparison, against the official base.

↯ Gemma↯ Gemma 4gemma

Hey guys, I'm setting up a local workflow on a single 24GB RTX 3090 to handle project planning—specifically digesting massive (~128k context) requirements documents/PRDs and spitting out a ton of structured .md files to act like Jira ticke…

↯ Vllm↯ Gemma↯ Qwen 38vllmgemmamoe+1

Exciting news: Qwen3.8-27B by @Alibaba_Qwen just landed in Code Arena: WebDev at #9 overall with 1595 pts. It is the only model in its size class in the top 10, and also reshapes the Pareto Frontier!

↯ Gemma↯ Qwen 3.8gemmaqwen

As per title. I'm not affiliated with the team behind this model in any way, shape or form.

↯ Gemmagemmaqwen

Hi everybody! Every now and then these days, we’re seeing really huge open-weight models popping up.

↯ Llama↯ Gemma↯ Gemma 4gemmallama

Gemma 12B is obviously a very well trained model, I always thought the fine tuning they did on it wasn't really cut out for agentic coding. From my own experiences it struggles to use the tools it's given from Github Copilot and is also ve…

↯ Copilot↯ Ollama↯ Llama↯ Gemma↯ Gemma 4ollamagemmacopilot+2

Hello, I have a few questions that I can't seem to find a clear answer to. Does it make sense to make your own GGUF?

↯ Llama↯ Gemma↯ Gemma 4gemmallama

I was finally able to replicate tensor level allocation outside the Gemma family. https://huggingface.co/ByteOtter/Qwen3.5-4B-CADA-IQ2_XS After the Gemma 4 12b, e4b and gemma 3 4b results, I attempted to expand into qwen and ran into a few…

↯ Gemma↯ Qwen 3.5gemmaqwen

I'm here to show some benchmarks while using llama cpp with an AMD V620 on Windows 11 via Vulkan & ROCM. The benchmarks were written out by AI, but are verified by myself to be correct.

↯ Llama↯ Gemma↯ Qwen 3.8gemmallamaqwen

I (like thousands of other engineers) got tired of trying to understand Claude's writing. ASD-STE100 instructions didn't change much, so I decided to approach the problem in the 2026 style: use an LLM.

↯ Ollama↯ Gemma↯ Gemma 4ollamagemmaclaude-code

The integration of large language models (LLMs) into the nuclear power industry requires outputs grounded in domain-specific knowledge. This study evaluates a 31-billion-parameter open-weight multimodal model (Gemma 4 31B-IT) on its capaci…

↯ Fine Tuning↯ Gemma↯ Gemma 4operatorgemmafine-tuning

This paper tests whether holonomy concentrates on active sparse-autoencoder (SAE) feature planes in Gemma 2 2B, a concrete operationalization of the broader semantic-concentration prediction. Holonomy is measured at the final-token layer-1…

↯ Gemmagemma

This study examines the cross-prompt generalization and first-language (L1) scoring effects of a LoRA-adapted open-weight large language model (Gemma-3-27B-it) applied to automated essay scoring. Using the identical model and inference con…

↯ Gemmagemma

I'm making Vox (a voice-to-text app for macOS and Windows that I built with Claude) free for everyone, and the repo is now open. You hold a hotkey, talk, release, and the transcribed + polished text lands in your clipboard (or gets pasted…

↯ Gemma↯ Gemma 4gemmacursor

We introduce Gemma 4, a new generation of open-weight, natively multimodal language models in the Gemma model family. Designed to advance compute efficiency and reasoning, the Gemma 4 model suite features dense and Mixture-of-Experts archi…

↯ Gemma↯ Gemma 4gemma

Deploying Large Language Models (LLMs) in high-stakes clinical settings remains limited by structural hallucinations, weak deterministic reasoning over tabular patient data, and omissions in vector retrieval. This paper presents the archit…

↯ Gemmagemma

If you have an Apple Silicon Mac you can run Claude Code completely locally (and free) by pointing it at a local server. Here's how: Setup (2 minutes) brew install mlx-serve mlx-serve run gemma-4-e4b-it # downloads + starts the server Then…

↯ Gemma↯ Qwen 3.6gemmaqwenanthropic+1

I was auditing my codes using the now back fable 5 and it kept failing to run runtime tests and this the error i got. so is anthropic now using gemma-4-12b-agentic-fable5-composer2.5-v2-3.5x-tau2 to run tests?

↯ Gemma↯ Gemma 4gemmaanthropicagentic

HF Realtime Voice Voice chat over WebSocket against a HF speech-to-speech The result is a speech-to-speech experience that feels dramatically more natural. Instead of waiting for an AI to respond, conversations flow with the responsiveness…

↯ Gemma↯ Gemma 4gemma

One thing I miss when using local models is the artifact experience from Claude. With Claude, if you ask for a dashboard, chart, diagram, or landing page, you actually get the thing rendered in the chat.

↯ Gemma↯ Gemma 4↯ Gemma 4↯ Gemma 4↯ Gemma 4↯ Gemma 4↯ Gemma 4gemma

Hi everyone, I just released Gemma-4-12B-Uncensored-Opus4.7-CoT. To remove the safety filters without destroying the model's reasoning, I combined a precise ablation method with a CoT (Chain-of-Thought) data fine-tune to fully recover the…

↯ Gemma↯ Opus 4.7gemma

Requirements: iPhone with A17 Pro or newer (8 GB RAM floor for the model), iOS 26+. TestFlight beta is open to anyone with a compatible device.

↯ Gemma↯ Gemma 4gemmaclaude-code

A text that asks for nothing still changes the model's answer — and the shift is invisible at both the input and the output TL;DR: Gave Gemma a neutral-topic text to read before asking it about NATO. It refused.

↯ Security↯ Gemma↯ Jailbreakjailbreakgemmasecurity

Update to my original post: https://www.reddit.com/r/ClaudeAI/comments/1u8nagi/connected_a_robinhood_account_to_claude_code_and/ I'm building a fully autonomous daily stock-trading desk in a Robinhood "Agentic" account. Opus is the CEO/PM,…

↯ Gemmagemmacodexopus+2

I was not sure myself, seeing a lot of statements here and around like "you need XXX VRAM / Unified Memory to run this model". So today I finally tested it.

↯ Gemma↯ Gemma 4gemmamoe

Hello everyone, i had a 3080ti 12gb and added a 3080 20gb, so it has a bit less speed but more memory than my main card. I could finally get some speed with the usual suspects (i am testing gemma 4 31b/26b-a4b and qwen 3.6 27b/35b-a3b), BU…

↯ Llama↯ Gemma↯ Qwen 3.6gemmallamaqwen

Currently recompiling my llama.cpp with support for diffusion Gemma, but I know on my hardware it won't likely be all that viable. I feel like if the goal was to take better advantage of consume GPUs for fast, intelligent generation, build…

↯ Llama↯ Gemma↯ Gemma 4gemmallama

I've been experimenting a bit today with letting models reason for creative tasks, rationale being that it might help with keeping track of details and prompt adherence. And predictably, the wall I'm running into is that they all want to d…

↯ Gemma↯ Qwen 3.6gemma

So after a few hours of reserach, im torn between getting either a radeon vii or 2 p100 (both options for roughly 240€). The Radeon would give me 32gb of vram and fast inferference, while the 2 p100 would give me a total of 48gb, but rough…

↯ Gemmagemmamoeqwen

Looking at older Nvidia cards under £200 for Gemma/Qwen MOE coding. Is there any reason to avoid older Titan 12GB cards other than being power hungry?

↯ Gemmagemmamoeqwen

I recently had a big family photo digitalization done for photos up to 130 years old. There are tons of people that I don't know or I don't recognize as young people in a soft lens.

↯ Gemmagemmaagentic

If Gemma 4 is better, does anyone have a link for the latest fixed template? Using LMstudio.

↯ Gemma↯ Qwen 3.6gemmaqwen

Model Overview Description: DiffusionGemma 26B A4B IT is an open-weights multimodal generative model developed by Google DeepMind that processes text, image, and video inputs to produce text output via discrete diffusion. Built on the Gemm…

↯ Deepmind↯ Gemma↯ Function Calling↯ Gemma 4function-callingdeepmindgemma+1

TLDR: I just added an MCP to the Observer framework making it 10x easier to use, so you can create micro-agents that monitor your screen autonomously, literally one sentence and you're done! So just typing "Monitor my Steam download and se…

↯ Llama↯ Gemma↯ Gemma 4gemmallamamcp

I've spent the last six months trying to build a fully local, agentic pipeline for a text_processing and extraction tool I use daily. ​Because I’m running everything on a single consumer GPU setup, my choices are limited to smaller, quanti…

↯ Gemma↯ Qwen 3.5↯ Qwen 3.5gemmaqwenagentic

https://huggingface.co/mradermacher/gemma-4-31B-it-qat-q4_0-unquantized-i1-GGUF/tree/main https://huggingface.co/mradermacher/gemma-4-31B-it-qat-q4_0-unquantized-GGUF/tree/main I waited a bit before asking this. I have 3060 12GB and 32GB d…

↯ Gemma↯ Gemma 4gemma

Maybe a niche question, but most of the time, Q8_0 and Q4_0 is a reference to the weights themselves. The activations themselves are available in BF16 format.

↯ Gemmagemma

Pretty exciting, wonder what it will take from llama.cpp to get it working locally

↯ Llama↯ Gemmagemmamoellama

26B 4B active parameters with crazy TPS Claims of 700+ TPS on a 5090 Introducing DiffusionGemma

↯ Gemmagemma

I'm trying to use Gemma 4 12B — the new encoder-free unified model (audio/vision/text in one) — for a one-pass audio → response voice assistant: feed the recorded WAV + system prompt and get the reply back as text directly, collapsing the…

↯ Vllm↯ Gemma↯ Gemma 4vllmgemma

I have been curious about how will having a infrastructure that provides agents the capability to explore code bases as relations, rather than text will change the performance of the AI agents So, for the last few weeks, I have been buildi…

↯ Gemma↯ Gemma 4gemmageminimcp+1

Hey everyone. I'm brand new to running LLMs in general, even more new to running them locally, and the sheer number of tools available is absolutely overwhelming.

↯ Ollama↯ Gemma↯ Qwen 3.6ollamagemmaqwen

We have summaries annotated by real humans that we benchmark various models, using an LLM as a judge, we found that in the 30B params range, Qwen 3 tops it out, followed by Gemma 4. It feels like newer Qwens are optimized to perform agenti…

↯ Gemma↯ Gemma 4gemmaqwenagentic

↯ Gemma↯ Gemma 4gemma

Have a crap IGPU 64 gb AMD. THis model works pretty good.

↯ Gemmagemma

↯ Gemma↯ Gemma 4gemma

Unsloth Gemma 4 QAT MTP assistant models now available They're both available as q8_0 models named mtp-gemma-4-*.gguf on the root of the directory and in both q8 and larger quants within an MTP folder. https://huggingface.co/unsloth/gemma-…

↯ Gemma↯ Gemma 4gemma

I never used to think it was all that important! But I’m using it for Svelte 5 and it ACTUALLY knows runes out of the box.

↯ Gemmagemma

Introducing Gemma 4 12B: a unified, encoder-free multimodal model Today, we are introducing Gemma 4 12B, our latest model designed to bring agentic multimodal intelligence directly to laptops. Bridging the gap between our edge-friendly E4B…

↯ Gemma↯ Gemma 4gemmaagentic

I had a huge LLM server, and now I have a tiny one! I had a Jetson Orin NX gathering dust from a long dead robotics project, from back in the Llama-7B days.

↯ Llama↯ Gemma↯ Qwen 3.6gemmamoellama+1

I'm just getting started using local LLMs for code. I'm not interested vibe coding, but I am hoping to increase my productivity in the publish or perish world of academia.

↯ Gemma↯ Qwen 3.6gemmaqwen

Hi everyone, I am comparing the standard (non-QAT) iq4_xs and q3_k_m quants with this QAT q4_k_xl model. (All of them are Unsloth versions)(gemma-4-26B-A4B-it-GGUF via lmstudio).

↯ Gemma↯ Gemma 4gemma

I'm trying to find out if anyone has done any benchmarking comparing the Gemma 4 4-bit QAT models (via Unsloth) against standard 8-bit non-QAT quants. I know QAT is supposed to retain a ton of accuracy compared to the baseline BF16, but I'…

↯ Gemma↯ Gemma 4gemma

Hopefully this isn't too low effort of a post. I just finished the benchmarks and I figured I'd post them online because they certainly were insightful for me.

↯ Gemma↯ Gemini 3.1gemmagemini

Original post: https://www.reddit.com/r/LocalLLaMA/comments/1txwff3/comment/oq1e0jt/?context=3 TL;DR: Migrated to WSL2 to test Linux (several people suggested it). Embedded MTP on the UD model: 25.8 tok/s.

↯ Gemmagemma

Did anyone manage to launch that in LMStudio? I am on the most recent update with the most recent llama.cpp available in LMStudio.

↯ Llama↯ Gemma↯ Gemma 4gemmallama

Hi all Been loving the QAT models but honestly what is up with the assistant models, any ggufs and ways to make em work with vanilla llamacpp and if this way of MTP is different than the one am17an developed for llamacpp. Followup question…

↯ Llama↯ Gemma↯ Gemma 4gemmallama

the MLX version of the QAT 4bit is like 27gb but the none QAT version is 17gb and the regular 4bit MLX version is also 17gb… anyone know why?

↯ Gemma↯ Gemma 4gemma

↯ Llama↯ Gemma↯ Gemma 4gemmallama

Ran a small, focused eval on three on-device models and the result was backwards from what I expected, so sharing the method and numbers. The task: tell the model "my dog is named Pablo," then add N turns of unrelated filler (shuffled gene…

↯ Gemma↯ Gemma 4gemmamoe

These last few weeks have been godsend for 24GB (and below) gpu poor peeps. Killer models released (Gemma 4 / Qwen 3.6) Free intelligence via QAT Bonus speed via MTP We're at the tipping point where GPU poor (24gb and below) people are act…

↯ Gemma↯ Qwen 3.6gemmaqwen

↯ Gemma↯ Gemma 4gemma

I spent the last few days trying to get consistent tool calling out of the new Gemma 4 12b QAT model and had to give up. When the model actually works, it works great, but for my specific use case and workflows it is just not for me.

↯ Gemma↯ Gemma 4gemmaqwen

Not talking about 31b. In terms of creative tasks, writing, chatting, not necessarily coding but can still be included, Does Gemma 12b outperform in any way?

↯ Gemma↯ Gemma 4gemma

I wanted to try new QATs and opened two collections on HF (which HF found for me): https://huggingface.co/collections/google/gemma-4-qat-q4-0 https://huggingface.co/collections/unsloth/gemma-4-qat One strange thing caught my attention, for…

↯ Gemma↯ Gemma 4gemma

↯ Fine Tuning↯ Gemma↯ Gemma 4gemmafine-tuning

↯ Gemmagemmaqwenagentic

when running by using transformers it runs by using vllm some weird error come up plese can any body share the command of running it on vllm ?

↯ Vllm↯ Gemma↯ Gemma 4vllmgemma

How are you using it? Quantized?

↯ Gemma↯ Gemma 4gemma

Running into something annoying with llama-server in router mode (`--models-preset`) and I can't tell if I'm missing a flag or if this is just how it works. My rig is 2x 3090, 2x 4060 Ti (one's unplugged at the moment, riser got repurposed…

↯ Llama↯ Gemmagemmallama

I am using llama.cpp version b9549 with this arguments as recommended: llama-server --temp 1.0 --top-p 0.95 --top-k 64 -hf ... Here is what I got on chessboard svg test https://www.reddit.com/r/LocalLLaMA/comments/1t53dhp/quality_compariso…

↯ Llama↯ Gemma↯ Gemma 4gemmallama

Hate to ask a simple question, but I’ve looked around and I see plenty of smaller models that *can* tool call, but none of them seem to do so appropriately or agentically. Referring to this.

↯ Gemmagemmaqwen

I’m trying to run: unsloth/gemma-4-31B-it-qat-GGUF gemma-4-31B-it-qat-UD-Q4_K_XL.gguf on an RTX 5090 32GB using llama.cpp Gemma 4 MTP PR branch. Main model loads.

↯ Llama↯ Gemma↯ Gemma 4gemmallama

I just came across the following post, where a user found some confusing divergence results between Q4 quants of the original and QAT models with a Q8/unquantized reference of the original model. https://www.reddit.com/r/LocalLLaMA/comment…

↯ Gemma↯ Gemma 4gemma

I've been running LLMs on my old potato i5-8500 with 32GB of RAM and *no GPU* for awhile now, running up to 12B dense models which run slow but perfectly useable. But this Gemma-4-26B-A4B simply flies on this CPU - only machine using Kobol…

↯ Gemma↯ Gemma 4gemma

Just kidding. Are there any distills that actually improve a model's quality?

↯ Anthropic Mythos↯ Gemma↯ Qwen 3.6gemmamythosgpt-5+1

I'll be upfront: I vibe-benched and vibe-reported this with Claude Sonnet 4.6, but I reviewed and edited everything before posting (too lazy to take out all the AI EM dash —), so hopefully nobody considers this AI slop. And more importantl…

↯ Gemma↯ Sonnet 4.6gemmasonnet

Title: Gemma 4 QAT MTP assistant heads now public on HuggingFace + PARALLEL=2 crash fix + 12B 2-slot bench (Strix Halo / Vulkan) Three things in one update: the converted QAT-matched draft heads are now uploaded for anyone to use, we found…

↯ Llama↯ Gemma↯ Gemma 4gemmallama

First we never saw an upgraded Air model after 4.5. Then GLM 4.7 Turbo was great, but quickly surpassed for coding.

↯ Glm↯ Gemma↯ Qwen 3.6glmgemmaqwen+1

Google just released the QAT (Quantization-Aware Training) variant of their Gemma 4 models, including 12B, so it was only natural for me to benchmark it on my 12GB GPU since it fits entirely in VRAM. I was pleasantly surprised of the resul…

↯ Llama↯ Gemma↯ Gemma 4gemmallama

Now someone needs to quantize them to 4bit, also I have intentionally kept the divergence and refusal different from original Gemma 4 heretic collection, so you can even try these as alternative to original model.

↯ Gemma↯ Gemma 4gemma

Table from https://unsloth.ai/docs/models/gemma-4/qat#qat-analysis I heard that MoE models are usually more susceptible to quantization error, but what happened with the 12B? I thought lower-parameter models usually quantized worse and yet…

↯ Gemma↯ Gemma 4gemmamoe

I’m a web developer doing mostly coding, but also project management, requirements analysis, testing, etc. I recently started experimenting with local LLMs, mostly because agentic stuff finally made them feel useful.

↯ Gemma↯ Qwen 3.6gemmamoeqwen+1

Gemma 4 QAT Q4_0 Bench on Strix Halo These are Google's official Gemma 4 QAT Q4_0 GGUF models, served locally through llama.cpp Vulkan/RADV on a Strix Halo APU. QAT means quantization-aware training.

↯ Llama↯ Gemma↯ Gemma 4gemmallama

While everyone's obsessing over giant cloud-based AI models, a quiet revolution is happening in local AI. We're seeing the emergence of extremely token-efficient, super-small system prompts, and modular agents designed specifically for loc…

↯ Gemma↯ Gemma 4gemma

I picked models I consider local (usable on 3×3090), so there are no 300B models, and you should probably skip 200B models too (but MiniMax and Step are pretty fast in Q3) Gemma-4 12B is still missing

↯ Minimax↯ Gemma↯ Gemma 4minimaxgemma

This subreddit is where I learned the most about using Local LLMs. I've been on this journey for 4 months now, and I'm already using Local LLMs in very complex pipelines.

↯ Gemmagemmaqwen

What's with the switch guys? now imagine if google gonna drop 128B model or a MoE version (I bet those Qwen lovers will forget Qwen even existed).

↯ Gemma↯ Gemma 4gemmamoeqwen

They started uploading to Gemma 4 MTP QAT but forgot to upload 12B quants to the Gemma 4 QAT 😭.

↯ Gemma↯ Gemma 4gemma

I’ve been doing lots of testing back and forth with this 7900xtx. All of my workloads were relying on qwen3.6 models, which are amazing fwiw, but I wanted some diversity in thought.

↯ Gemma↯ Qwen 3.6gemmaagentic

First time hearing it. I also heard about the gemma 4 qat quants and if any one of them is good for 4gb vram and 16gb ram.

↯ Gemma↯ Gemma 4gemmamoe

Posting to share my results with others, I think the big bottom line is MTP acceptance rates offering a huge speedup, during coding tasks it's over 90% acceptance! Haven't hit my soft goal results or llm as judge benchmarks yet to compare…

↯ Llama↯ Gemma↯ Gemma 4gemmallama

could not extract summary

↯ Gemma↯ Gemma 4gemma

This is a PSA for people like me who tried it and hit the wall with tool calls failing left and right, so much so that harnesses like OpenCode just didn't work: There is a fix for that. You need to pass a better chat template file, which i…

↯ Llama↯ Gemma↯ Gemma 4gemmallamaqwen

Release Blog Post: Gemma 4 with quantization-aware training HuggingFace for mobile: Gemma 4 QAT Mobile - a google Collection HuggingFace for Q4_0: Gemma 4 QAT Q4_0 - a google Collection

↯ Gemma↯ Gemma 4gemma

Their collection: https://huggingface.co/collections/unsloth/gemma-4-qat And their guide, always a very interesting read: https://unsloth.ai/docs/models/gemma-4/qat

↯ Gemma↯ Gemma 4gemma

Wanted to share a result I didn't expect to work. Running google/gemma-4-e2b locally through LM Studio, exposed via OpenAI-compatible endpoint, called from a Spring Boot app using Spring AI's ChatClient abstraction.

↯ Gemma↯ Sonnet 4.6gemmasonnetopenai

Ever since the latest Gemma releases, there is literally zero reason to use Qwen. Better architecture, cleaner code output, and it doesn't get stuck in weird multi-turn reasoning loops.

↯ Gemma↯ Qwen 3.7↯ Qwen 3.7gemmaqwen

Think Local started with a simple idea: AI should work for you, not collect from you. So I built an app that lets you run modern AI models completely on-device - privately and fully offline.

↯ Llama↯ Gemmagemmadeepseekllama+3

Hi everyone, I’m looking for advice on local AI setups. My goal is to have a local AI generate text documentation from my one-hour therapy sessions.

↯ Gemma↯ Qwen 3.6gemmaqwen

Edit: does this happen every time a newbie tries to post here. Getting roasted despite having valid results?

↯ Vllm↯ Gemma↯ Qwen 3.6vllmgemmaqwen

Hi all this is to inform you all that many frontline models like GPT, sonnet opus and or Gemma even are at stage of collapsing as they have frequently started drifting and running away from provided work either stretching that work too lon…

↯ Gemmagemmasonnetopus

Found this ACM paper on prompt injection and jailbreak attacks against open-source LLMs. The authors tested 10 open-source models across 94 prompt injection and 73 jailbreak scenarios, including Phi, Mistral, DeepSeek-R1, Llama 3.2, Qwen,…

↯ Security↯ Llama↯ Mistral↯ Gemma↯ Jailbreak↯ Llama 3.2mistraljailbreakprompt-injection+5

I can set my context length as high as 64k and the vram usage is not even remotely close to the maximum utilisation. My TPS is also 40+.

↯ Gemma↯ Gemma 4gemma

Hi, I’m having a very hard time right now. I used to use ChatGPT 4o and 5.1 Thinking for helping me write my story, and I was very happy with them.

↯ Gemma↯ ChatGPT 5.5gemmachatgpt

Due to old GPU I run on CPU and came to appreciate value of MoE. I know of MoE for Qwen 3.6 and Gemma-4, which are <40B.

↯ Gemma↯ Qwen 3.6gemmamoeqwen

Just saw this and wanted to ask the obligatory GGUF when?

↯ Gemma↯ Gemma 4gemma

Google launched its Gemma 4 open models this spring, promising a new level of power and performance for local AI. Google’s take on edge AI could be getting even faster already with the release of Multi-Token Prediction (MTP) drafters for G…

↯ Gemma↯ Gemma 4gemma

Hey everyone, ​I'm running into a weird issue and hoping someone here might have a fix or some troubleshooting ideas. I'm currently trying to run the new Gemma 4 31b-it model using vLLM (v0.20.0-cu130) deployed via Helm chart (https://gith…

↯ Vllm↯ Gemma↯ Gemma 4vllmgemmaopenai

Just tested Gemma 4 31B with the new official MTP Drafter on my H100 today and compared the approach with DFlash to help you decide which one to use. Without drafter: 13.7 tok/s.

↯ Vllm↯ Gemma↯ Gemma 4vllmgemma

Let's test the Gemma 4 MTP implementation using the HuggingFace Transformers library and the new drafter model by Google. We'll load both models and test on a couple of prompts with and without the MTP support https://www.youtube.com/live/…

↯ Gemma↯ Gemma 4gemma

Increase your CPU Thread Pool Size to your processor's max. In LM Studio, the max is 10.

↯ Gemma↯ Qwen 3.5gemmaqwen

I'm confused about what to get. I don't want to get something super expensive, but would like to have something that's "good enough" for coding etc.

↯ Gemmagemmaqwen