Hey r/LocalLLaMA, We’ve released our ByteShape Qwen 3.6 35B GGUF quantizations in two families: standard NTP (Next Token Prediction or non-MTP) and MTP. Blog / Download NTP Models / Download MTP Models TL;DR For NTP, “pick the largest quan…
#mmlu
22 items
Qwen 3.6 35B GGUF: NTP vs MTP quantization results across GPUs and CPUs (www.reddit.com) MiniMax m2.7 under 64gb for Macs - 91% MMLU (www.reddit.com) https://huggingface.co/JANGQ-AI/MiniMax-M2.7-JANGTQ Used TQ as quantization method where it matters. Finally mac users under 64 gb - esp base m5 users can get a real cloud SOTA-like level LLM running from home.
First DeepSeek V4 Flash-Base-Int4 Quant (huggingface.co via hn) DeepSeek-V4-Flash-Base INT4 A real INT4 packed-storage quantization of deepseek-ai/DeepSeek-V4-Flash-Base — a 284 B-parameter Mixture-of-Experts model. Hero numbers | Metric | This release | Community Q4KM norm | |---|---|---| | MMLU (5 su…
Show HN: Flint – A 30B model fine-tuned for less repetition (springboards.ai via hn) As frontier LLMs have very little output diversity even for open ended queries. We built Flint to see if we could reverse this.
GGUF Quants Arena for MMLU (24GB VRAM + 128GB RAM) (www.reddit.com) Dataset: MMLU subset (DEV+TEST) Llamacpp setting: 3 params only ctx 8192 , seed 42 , fa on Let me know whatelse do you want to see. Thanks.
The Two MMLU Scores: What a Benchmark Name Does Not Fix (zatona.dev via hn) The Two MMLU Scores: What a Benchmark Name Does Not Fix Date: September 6, 2026 · Author: Dmitrii Zatona TL;DR - Two MMLU accuracies, 0.781 and0.79 , for two builds of one model family under the same benchmark name; for ascore-delta query…
Show HN: Cactus Hybrid: We taught Gemma 4 to know when it's wrong (github.com via hn) Hey HN, Henry & Roman here from Cactus. A small, on-device model is fast and private, but sometimes wrong, but frontier models are getting expensive pretty fast.
Beyond Arc and GSM: Langford Coverage as a Benchmark for ASI (zenodo.org via hn) As foundational models approach Artificial Superintelligence (ASI), standard intelligence evaluations like the Massive Multitask Language Understanding (MMLU), Grade School Math 8K (GSM8K), and the Abstraction and Reasoning Corpus (ARC-AGI…
Basaltlabs Monolith-1.0 – #1 on Last Exam, AIME, GPQA Diamond, MMLU-Pro (basaltlabs.org via hn) The frontier is open. Monolith-1.0 leads every major reasoning benchmark, and its weights are on Hugging Face.
Domain-level metacognitive monitoring in frontier LLMs: A 33-model atlas (arxiv.org via hn) Aggregate metacognitive quality scores mask within-model variation across MMLU benchmark domains. We administered 1,500 MMLU items (250 per domain, under an a priori six-domain grouping) to 33 frontier LLMs from eight model families and co…
I benchmarked Jev aginst gpt-5.6-luna! (www.reddit.comhttps) I got access to TypeSafe's Jev a few days ago. It's an odd kind of model that doesn't generate text at all.
Turkish MMLU Pro: Traceable Option Augmentation and Its Validity Limits in Turkish Multiple-Choice Evaluation (arxiv.org) Adding answer options can lower multiple-choice scores without improving assessment validity. Turkish MMLU Pro examines this distinction using 12,000 Turkish-source questions across 58 sections.
What Does MMLU Actually Measure? A Psychometric Audit of Difficulty Structure in Aggregate Benchmark Scores (arxiv.org) Although MMLU is widely adopted as a benchmark for calibrating general AI capabilities, we psychometrically demonstrate that its aggregate score primarily evaluates a model's factual retrieval capacity rather than its reasoning ability. By…
LLMPEDIA: Browsing, Verifying, and Comparing the Parametric Encyclopedic Knowledge of LLMs (arxiv.org) Flagship language models appear saturated on benchmarks like MMLU (Hendrycks et al., 2021), scoring above 90% - yet benchmarks test only what the experimenter thought to ask, the availability bias of fixed question sets. LLMPEDIA makes thi…
Fidelity Is Not Safety: Gently-Compressed LLMs Pass Every Data-Free Quality Guard Yet Invent Procedure Steps in Agentic Execution (arxiv.org) Practitioners accept a compressed language model once it clears a stack of data-cheap quality guards: perplexity within a small factor of the original, downstream accuracy (for example MMLU) inside a confidence interval, and data-free outp…
Claude for daily use: my honest experience, plus references to public benchmarks (www.reddit.com via reddit) I've been using Claude 3.5 Sonnet for several weeks as my primary AI assistant. I wanted to share my genuine, hands‑on experience – and because this involves a comparison with another model, I've also included relevant public benchmark dat…
Building a European Multilingual Evaluation Dataset: The MMLU Localisation Project within the EMT Network (arxiv.org) This paper reports on a collaboration between the Directorate-General for Translation (DGT) and the European Master's in Translation (EMT) to localise the MMLU dataset into 11 European languages. Beyond creating a more inclusive benchmark…
Meta-Benchmarks for Financial-Services LLM Evaluation (arxiv.org) Public LLM leaderboards optimise for global average performance and do not capture the specific cognitive demands of financial-services work: a model that leads on MMLU-Pro may underperform on document-grounded compliance reasoning, and a…
LLMpedia: A Transparent Framework to Materialize an LLM's Encyclopedic Knowledge at Scale (arxiv.org) Benchmarks like MMLU suggest flagship language models approach factuality saturation above 90\%. \emph{LLMpedia} shows this picture is incomplete.
I built a 1v1 nuclear strategy game to benchmark LLM reasoning (instead of just QCMs) — Age of LLM (www.reddit.com via reddit) In 2017, I watched OpenAI Five destroy pro players at Dota 2. That moment taught me something: games are the ultimate test of emergent intelligence.
UrduMMLU: A Massive Multitask Benchmark for Urdu Language Understanding (arxiv.org) Meaningful multilingual evaluation must test models in the target language and educational context. Urdu, spoken by more than 230 million people, lacks a broad MMLU-style benchmark built from native educational sources.
We open-sourced Chaperone-Thinking-LQ-1.0 — a 4-bit GPTQ + QLoRA fine-tuned DeepSeek-R1-32B that hits 84% on MedQA in ~20GB (www.reddit.com) Hey everyone, We just open-sourced our reasoning model, Chaperone-Thinking-LQ-1.0, on Hugging Face. It's built on DeepSeek-R1-Distill-Qwen-32B but goes well beyond a simple quantization — here's what we actually did: The pipeline: 4-bit GP…