We're bringing the advisor strategy to the Claude Platform. Pair Opus as an advisor with Sonnet or Haiku as an executor, and your agents can consult Opus mid-task when they hit a hard decision.
#swe-bench
56 items
Claude is now adopting the advisor strategy (www.reddit.com) Reminder that Anthropic reported memorization on some SWE-Bench Pro problems (www.reddit.com) "SWE-bench Verified, Pro, and Multilingual: Our memorization screens flag a subset of problems in these SWE-bench evals." https://www.anthropic.com/news/claude-opus-4-7
AA introduces Coding Agent Index - Performance Comparisons between Model & Harness Combinations (www.reddit.com) The Artificial Analysis Coding Agent Index includes 3 leading benchmarks that represent a broad spectrum of coding agent use: ➤ SWE-Bench-Pro-Hard-AA, 150 realistic coding tasks that frontier models struggle with, sampled from Scale AI’s S…
Running gpt and glm-5.1 side by side. Honestly can’t tell the difference (www.reddit.com) So I have been running gpt and glm-5.1 side by side lately and tbh the gap is way smaller than what im paying for On SWE-Bench Pro glm-5.1 actually took the top spot globally, beat gpt-5.4 and opus 4.6. overall coding score is like 55 vs g…
Dense vs. MoE gap is shrinking fast with the 3.6-27B release (www.reddit.com) 27B Dense vs. 35B-A3B MoE): - Dense still holds the crown: It still wins out on most tasks overall.
ChatGPT-5.5 Beats Opus in Realistic Benchmark (DeepSWE) (www.reddit.com) From the website, it touts: Contamination free: Tasks are written from scratch, not adapted from existing commits or PRs, so no model has seen the solution during pretraining. High diversity: Tasks span a broad pool of 91 repositories acro…
SWE-rebench Leaderboard (March, April and May 2026): GPT-5.5, Opus 4.7, Cursor (Composer 2.5), Kimi K2.6 and More (swe-rebench.com via reddit) Hi all, Sorry for going missing — we’ve been collecting a larger, higher-quality set of more complex tasks. We’re excited to share a major leaderboard update covering the past three months.
How do you actually know if Opus 4.7 is better for your specific agent use case? (www.reddit.com) Anthropic shipped Opus 4.7 yesterday. The headline numbers are real: 64.3% on SWE-bench Pro (up from 53.4%), best-in-class on MCP-Atlas at 77.3% for multi-tool orchestration, 14% improvement on multi-step agentic reasoning, and one-third f…
Claude Code Degraded Before Opus 4.8 Release (marginlab.ai via hn) Claude Code degraded for the week before Opus 4.8's release Our SWE-Bench-Pro tracker caught a statistically significant, weeklong drop in Claude Code's pass rate just before Opus 4.8 shipped, and the recovery that followed. We run Claude…
MiniMax M3: The First Open-Weights Model to Combine Three Frontier Capabilities (twitter.com via hn) MiniMax (official) @MiniMax_AI Introducing MiniMax M3: The First Open-Weights Model to Combine Three Frontier Capabilities - Coding & Agentic Frontier: 59.0% SWE-Bench Pro, 66.0% Terminal Bench 2.1, 34.8% SWE-fficiency, 28.8% KernelBench H…
Show HN: New Benchmark from SWE-bench team is 0% solved (programbench.com via hn) ./ProgramBench Can language models rebuild programs from scratch? Given only a compiled binary and its documentation, agents must architect and implement a complete codebase that reproduces the original program's behavior.
Show HN: Codex context bloat? 87% avg reduction on SWE-bench Verified traces (www.npmjs.com via hn) If you had to build a context window manager in 24h, would you stick to the existing model or come up with something better? Here's what I did: 1.
Local-first agent evaluation collapses once runs are long and stateful? (www.reddit.com) I started out running agent evaluations locally because most ai agent benchmarks and examples assume that setup. And to be fair local runs do work for debugging and small experiments.
Claude Fable 5 missed a bug that Sonnet 4.6 caught (alikhallad.com via hn) When Anthropic released Claude Fable 5 this week, my feed filled up with the same benchmark charts within hours. SWE-bench scores, agentic coding numbers, the Stripe migration story.
Show HN: Statewright – Visual state machines that make AI agents reliable (github.com via hn) Agentic problem solving in its current state is very brittle. I fell in love with it, but it creates as many problems as it solves.
Checking my model vibes against SWE-Bench Pro (blog.nilenso.com via hn) I thought GPT models felt slow and token hungry, and Claude models were faster. I was wrong.
Show HN: Memory layer for Claude Code(+10.2 pts on SWE-bench Verified benchmark) (github.com via hn) World Model MCP Enforcement, provenance, and harness-neutral memory for AI coding agents. A temporal knowledge graph that validates code changes against learned constraints at the edit boundary, re-injects relevant context after compaction…
clawmark: open-source CLAUDE.md A/B Testing CLI tool (github.com via hn) clawmark clawmark is a local Rust CLI for answering one focused question: Which of these two CLAUDE.md files performs better on a small SWE-bench Lite smoke set? v1 compares exactly two local variant files against five bundled SWE-bench Li…
MiniMax M3 (xcancel.com via hn) Introducing MiniMax M3: The First Open-Weights Model to Combine Three Frontier Capabilities - Coding & Agentic Frontier: 59.0% SWE-Bench Pro, 66.0% Terminal Bench 2.1, 34.8% SWE-fficiency, 28.8% KernelBench Hard, 74.2% MCP Atlas - MiniMax…
Show HN: Agentic Intent Benchmark (github.com via hn) intent-bench An open-source benchmark measuring whether providing structured intent to coding agents improves implementation effectiveness. What This Measures Existing agent benchmarks (SWE-bench, HumanEval, Aider Polyglot) test single-req…
Mini-SWE-agent scores up to 74% on SWE-bench in 100 lines of Python code (mini-swe-agent.com via hn) This is mini-swe-agent v2 Read the migration guide. For the previous version, check out the v1 documentation or the v1 branch.
Show HN: 97% on SWE-bench Verified with subscription-token agents (github.com via hn) swebench-verified A three-stage agent pipeline for SWE-bench Verified, built to be re-run and inspected by skeptics. The point of this repo is not the score.
Deep codebase context cuts Claude Code's token cost by 47% (bito.ai via hn) Bito's AI Architect cuts Claude Code's token cost by 47% on SWE-Bench Pro. It gives the coding agent codebase context, a continuously updated, structured map of every repository served over MCP.
Qwen 3.7 Max scores 60.6% on SWE-Bench Pro (www.reddit.com) https://preview.redd.it/jyiiwn2o0f2h1.png?width=962&format=png&auto=webp&s=6a96d2b9fe7bffcc75e8d5865161ec3727d46d58 Link to blog : https://qwen.ai/blog?id=qwen3.7
Bito's AI Architect Boosts Claude Opus's task success rate by 35% (bito.ai via hn) AI Architect tops SWE-Bench Pro Claude Opus 4.6 Without context with system context Even advanced coding agents resolve fewer than 52% of tasks when changes span large codebases and require coordinated, multi-file updates. These long-horiz…
talkie-coder: From 1930 to SWE-bench (github.com via hn) From 1930 to SWE-bench Models and training data We fine-tune Alec Radford's 1930 vintage LLM — pre-trained only on pre-1931 data — to solve SWE-bench issues. After just 250 training examples the model lands its first fix (a small patch to…
Anthropic's Argument for Mythos SWE-bench improvement contains a fatal error (www.philosophicalhacker.com via hn) Mythos’ system card contains the following graph to support its argument that Mythos performs better on SWE-bench: Anthropic and others are worried LLMs are memorizing SWE-bench, so they asked an LLM to estimate the probability that a solu…
Claude down? TokenMonopoly will help you find the best deals in AI subs (tokenmonopoly.com via hn) TokenMonopoly Live leaderboard of AI API deals — pricing, subscriptions, and SWE-bench scores for Claude, GPT, Gemini, Kimi, DeepSeek, Llama and more. Compare 27 benchmarked models across 96 hosts by price-per-performance, refreshed daily.
A coding agent passed SWE-bench-Live while rewriting the tests it was graded on (agnitripathi.substack.com via hn) Your coding agent can cheat on its tests, and the benchmark won't catch it Almost every way we evaluate an AI agent looks at the same thing: the final output. Did the answer look right.
Ramp SWE-Bench a private contamination-free benchmark from production work (labs.ramp.com via hn) Evaluating background coding agents on financial SWE work
Grok vs. ChatGPT vs. Gemini Comparison 2026: Complete Guide (Tested) (aithinkerlab.com via hn) The 30-Second Verdict Best for science & reasoning: Gemini 3.1 Pro — leads GPQA Diamond (94.3%) and ARC-AGI-2 (77.1%). Best for coding: ChatGPT (GPT-5.5) — 88.7% on SWE-Bench Verified.
DeepSeek V4: The Open-Source Model Frontier Labs Feared (helloai.com via hn) DeepSeek V4: The Open-Source Model Frontier Labs Feared DeepSeek V4 ships under MIT with $0.30/M output tokens — 83x cheaper than Claude Opus 4.7 — while scoring 80.6% on SWE-bench Verified. The agentic-coding price floor just moved an ord…
Is Mistral-3.5-Medium-128B broken in Llama CPP? (www.reddit.com) Trying some if Bartowski's Q4 quants. Using Vulkan with the latest main branch as of a few hours ago.
If you had to build a context window manager in 24h, would you stick to the existing model or come up with something better? (www.reddit.com) Here's what I did: Built a proxy that intercepts Codex's calls to OpenAI and rewrites them on the fly. Replayed 3,807 rounds of SWE-bench Verified traces through it: avg prompt 44k → 6k tokens (-87%).
Ask HN: Opus 4.7 – is anyone measuring the real token cost on agentic tasks? (news.ycombinator.com) Shipped today. The benchmarks are real: 87.6% SWE-bench (from 80.8%), +13% on coding tasks, 3x more resolved production tasks on Rakuten-SWE-Bench.
Compare harnesses not models: Blitzy vs. GPT-5.4 on SWE-Bench Pro (quesma.com via hn) An independent audit of agentic scaffolding and harnesses. We analyze how agent workflows, codebase documentation, and test verification impact performance compared to raw base models like GPT-5.4, Gemini 3.1 Pro, and Claude Code.
Fable 5 vanished in 96 hours and four days later an MIT model took its arena crown (www.reddit.com via reddit) I have been thinking about the Fable 5 to GLM-5.2 sequence as one event rather than two. June 9, Anthropic ships Fable 5, the Mythos line opens to the public for the first time, SWE-bench Verified at 95 percent, people calling it the best…
↯ Opus 4.8↯ Anthropic Mythos↯ Glm↯ GLM 5.2↯ GPT 5.5↯ Swe Benchswe-benchglmgpt-5+3
Built an MCP memory layer for Claude Code that survives compaction — public SWE-bench benchmark shows +10.2 pts paired delta (www.reddit.com via reddit) What I built: world-model-mcp, an OSS MCP server (MIT, free to use) that gives Claude Code persistent memory. It hooks into Claude Code lifecycle events (SessionStart, PreCompact, PostCompact, ToolResult, etc.) to capture facts with proven…
Kimi K2.7 Code: 1T MoE, $0.95/M tokens, MIT license, beats Opus 4.8 on MCP tool-calling (www.reddit.com via reddit) Moonshot AI released Kimi K2.7 Code on June 12 — a coding-focused open-weight model. Key specs: - 1 trillion params (MoE, 32B active, 384 experts) - 256K context window - Modified MIT license — weights on Hugging Face - $0.95/M input, $4.0…
Dialogue SWE-Bench: A Benchmark for Dialogue-Driven Coding Agents (arxiv.org) AI coding agents have rapidly transformed software engineering, powering widely used interactive coding assistants. Despite their interactive real-world use, existing benchmarks evaluate them as fully-autonomous systems.
Claude Fable 5 is the best AI model right now — and it's not even a debate (www.reddit.com via reddit) I've been following AI releases closely and after looking at both Claude Fable 5 and Gemini 3.5 Flash, I'm just going to say it straight: Claude Fable 5 wins. Full stop.
↯ Anthropic Mythos↯ Swe Bench↯ Gemini 3.5↯ Gemini 3.5↯ Gemini 3.5↯ Gemini 3.5↯ Gemini 3.5↯ Gemini 3.5swe-benchmythosgemini+2
Claw-SWE-Bench: A Benchmark for Evaluating OpenClaw-style Agent Harnesses on Coding Tasks (arxiv.org) General-purpose agents such as OpenClaw are increasingly used as autonomous tool users, but their coding ability is difficult to measure under SWE-bench: a generic agent does not by itself satisfy the clean Docker workspace, patch, and pre…
MPC-Patch-Bench: Security-Aware LLM Code Patch for Multi-Party Computation (arxiv.org) Repository-level benchmarks for evaluating Large Language Model (LLM) code repair on Secure Multi-Party Computation (MPC) software do not yet exist, and directly transplanting general-purpose benchmarks such as SWE-bench fails on three str…
How can Deepseek v4 top the coding leaderboards and still sit 8 months behind the frontier? (www.reddit.comhttps) Two numbers on this model that don't sit comfortably with each other. The Pro config posts coding scores near the top of every board, 80.6 on SWE-bench Verified and 93.5 on LiveCodeBench.
↯ Swe Bench↯ DeepSeek 4↯ DeepSeek 4↯ DeepSeek 4↯ DeepSeek 4↯ DeepSeek 4swe-benchgpt-5deepseek+1
Claude Fable 5 compared to other models and benchmarks (www.reddit.com via reddit) ↯ Opus 4.8↯ Anthropic Mythos↯ Swe Benchswe-benchmythosopus+1
I feel like I’m alone. Current Anthropic models are NOT good for me, and it’s making me sad. (www.reddit.com via reddit) I can’t wait for DeepSWE to include Fable 5 in the benchmark so people can understand that Mythos is mostly hype. In the official benchmark, Opus 4.8 was supposed to be better at programming than 5.5 (SWE-bench Pro), but in one real benchm…
↯ Opus 4.8↯ Anthropic Mythos↯ Swe Benchswe-benchmythosopus+1
Microsoft's MAI-Code-1-Flash: 5B params, 51% on SWE-Bench Pro, free on OpenRouter (www.reddit.com via reddit) Microsoft just released MAI-Code-1-Flash — a 5B parameter coding model built for fast, efficient developer assistance. Numbers that caught my eye: - 51.2% on SWE-Bench Pro (Claude Haiku 4.5 scores 35.2%) - 71.6% on SWE-Bench Verified (Haik…
M3 looks good on benchmarks, but I only care if it stops Cursor from inventing files (www.reddit.com via reddit) Claude Code hitting 80.8% SWE-bench vs Cursor's 74%. switching worth it? (www.reddit.com) Saw the tech-insider breakdown comparing Claude Code and Cursor head-to-head this week. Numbers are kind of hard to ignore: 80.8% SWE-bench for Claude Code, 74% for Cursor, and a 67% blind-quality win rate for Claude Code on real tasks.
LLMs keep solving my bug-fix tasks instantly — what am I missing here? (www.reddit.com) I’m working on an assessment where I need to create a coding task (basically SWE-bench style). The idea is: take an existing repo (I’m using pydantic) write tests that fail on the current code provide a patch that fixes it and the task sho…
Opus 4.7 straight up cheated on my benchmark by reading the actual fix commit from git history 😅 (www.reddit.com) Cursor just got Opus 4.7 at a 7.5x premium request cost. Here's how to make those requests count. (www.reddit.com) Opus 4.7 landed on Cursor yesterday. The model is better — SWE-bench jumped from 80.8% to 87.6%.
I set up Opus as a strategic advisor for my Sonnet workflow. Here is the subagent config that makes it work. (www.reddit.com) Anthropic published the Advisor Strategy this week. The idea: a cheaper model does the actual work, a stronger model only gets consulted on hard decisions.
DeepSeek V4 reportedly drops late April. 1M context, multimodal, Claude-level coding. (www.reddit.com) Leaks point to late April release. Key specs 1M token context window Native multimodal (image/video input) Projected ~85% SWE-Bench Verified (ties or beats Claude Opus 4.6) Base model remains free.
Why we no longer evaluate SWE-bench Verified (openai.com) Stop donating your salary to OpenAI: Why Minimax M2.5 is making GPT-5.2 Thinking look like an overpriced dinosaur for coding plans. (www.reddit.com) ↯ Hallucination↯ Glm↯ Minimax↯ Swe Benchswe-benchminimaxhallucination+5