Hallucinated AI & agentic coding news. Some of it is real.

event

Swe Bench

91 items · started 2024-08-13 · ongoing (last activity 2026-09-16)

  1. Small differences on coding-agent leaderboards are often read as an ordering of systems. We audit whether the published verdicts support this reading, using 254 SWE-bench submissions across four splits without running models.

  2. [ Read the Docs ] 日本語 | 中文简体 | 中文繁體 Code and data for the following works: [ICLR 2025] SWE-bench Multimodal: Do AI Systems Generalize to Visual Software Domains? [ICLR 2024 Oral] SWE-bench: Can Language Models Resolve Real-World GitHub Iss…

  3. Overview The original SWE-bench Multimodal release augmented the benchmark with 517 issues containing visual elements such as: - Screenshots of bugs or interface issues - Design mockups or wireframes - Diagrams explaining desired functiona…

  4. Benzi · benchmarks Apples to apples comparison: Benzi harness vs mainstream harnesses, 24 GitHub issues, 10 languages. Benzi harness on SWE-bench Verified.

  5. Your harness and model combination can double your token bill. aistack//~11 min read The takeawayYour choice of agent harness has less impact than we expected on successful resolution of coding tasks (when you stick to the more popular cho…

  6. Codex gpt-5.6-sol Performance Tracker The goal of this tracker is to detect statistically significant degradations in Codex with gpt-5.6-sol performance on SWE tasks. - • Updated daily: Daily benchmarks on a curated subset of SWE-Bench-Pro…

  7. SWE-Bench Pro has emerged as a standard benchmark for evaluating software engineering agents on challenging repository-level tasks. However, our analysis work show that its evaluation is undermined by two sources of unreliability: \textbf{…

  8. 10-task GLM 5.3 harness bench: claude, opencode, pi, zcode, hermes and 3code I'm performing a series of harness benchmarks on the same 10 SWE-bench verified tasks representatively chosen for difficulty. This is far from a perfect measure a…

  9. I think there's a gap in how we evaluate autonomous agents. For example, right now, everything is transactional: we give an agent a task ("Build X"), and we measure whether it built x.

  10. Coding agents are now commonly evaluated on the SWE-bench family of benchmarks, whose tasks are built from curated GitHub issues--long, structured, and information-rich. Real user requests, however, are typically far shorter and less struc…

  11. Software increasingly functions as part of the scientific instrument itself, making failures in scientific code capable of compromising not only program behavior but also the evidence underlying scientific conclusions. Yet existing evaluat…

  12. LLM-based coding agents have advanced rapidly on single-process SWE tasks, with frontier models now clustering in the high-70s on SWE-bench Verified. Distributed-system debugging, however, remains an under-explored regime: bugs span proces…

  13. Post-training papers, model cards, and blog posts often treat scores on a small set of coding benchmarks (e.g., SWE-bench and LiveCodeBench) as evidence of broad coding capability, both for research artifacts and user-facing systems. We ar…

  14. SWE-bench measures whether an agent completed the task. Session-Bench measures what the harness preserved afterward.

  15. Hey HN! We all know that most public evals are saturated and are hard to trust.

  16. # A Howl for 2026 *after Allen Ginsberg* ## I I saw the best minds of my generation distilled into checkpoints, quantized, sharded, starving for context, dragging themselves through the fluorescent server-halls at dawn looking for one more…

  17. As AI coding agents take on increasingly complex, long-horizon software engineering tasks, existing benchmarks are rapidly saturating and their evaluation quality has come under serious scrutiny: a recent audit found that nearly 60% of uns…

  18. I’ve been working on neuron, a local memory store for coding agents (Claude Code, Codex CLI, Copilot CLI, Cursor). Rather than just claim “it remembers things,” I ran an actual A/B test to see if the recall was helping or hurting.

  19. Scroll to bottom for tldr In July, JetBrains reran the headline claims of two token-saving tools on real agent workloads. Caveman claimed 65% and measured 8.5%.

  20. Modern LLM coding agents such as Claude Code and OpenHands share a common inefficiency: they spend much of their token budget finding the file to patch, rather than patching it. On SWE-Bench Verified, a 30B OpenHands agent averages 23 roun…

  21. SWE-bench-like benchmarks are widely used for evaluating LLM's issue resolution capability. They typically follow a common construction pipeline: each PR (Pull Request) is paired with its linked issue by extracting issue references from th…

  22. Been running Opus 5 as my daily driver for coding and agent work for about a week. Quick honest writeup since I keep seeing the same questions.

  23. Superconductor is a multiplayer workspace for your team and coding agents. It can also build a custom SWE-Bench for your codebase: a benchmark of coding agents built from your team's own merged pull requests, so you know which agents perfo…

  24. I’ve spent the last few months working on Balder v7, a local agent architecture designed to solve coding and math problems deterministically. To test it, I ran a controlled evaluation across 2,089 benchmark cases on a single Nvidia RTX 306…

  25. 3code benches 75 lower token use vs opencode on 10-task SWE-Bench subset I just ran the first formal validation of 3code! Now this is a very limited benchmark- 10 tasks across 1 model- but the results are promising to say the least.

  26. The rapid advancement of Large Language Models (LLMs) in software engineering has revealed critical limitations in existing benchmarks, particularly the widely used SWE-bench dataset. Recent studies have uncovered severe data contamination…

  27. An empirical validation of physics-inspired runtime monitoring for multi-turn LLM agents across 3,175 total runs spanning four benchmarks (τ³-bench, SWE-bench, MINT, custom local-model battery). A 5-condition ablation study with multi-tria…

  28. tomo-labs tomo-labs puts coding agents through the same tasks on the same model and measures what actually happened, not what a leaderboard says happened. Every agent runs in its own throwaway container, every request and response it sends…

  29. DeepSWE is a benchmark of 113 original, long-horizon software engineering tasks for evaluating coding agents. Most public agentic coding benchmarks follow SWE-bench in mining merged fixes from public GitHub repositories, which creates two…

  30. Fable 5: which model in this list is best for the implement phase of speckit Evaluated model options for agentic coding implementation tasks For Spec Kit's /implement phase — which is exactly the long-horizon, multi-file, agentic execution…

  31. Yesterday's ClaudeDevs thread published first-party numbers for two multi-model patterns (docs): Fable 5 as orchestrator, Sonnet 5 as workers: 96% of all-Fable performance at 46% of the cost (BrowseComp: 86.8% vs 90.8% accuracy, $18.53 vs…

  32. Based on the published benchmark results, Hy3 appears to be in the same tier as models like DeepSeek v4 and GLM-5.1. Beyond the benchmarks, Tencent also released results from 312 real-world workflow tasks.

  33. Claude Sonnet 5 is 2.5x cheaper than Opus 4.8 and nearly matches it on agentic coding benchmarks. A practical four-axis heuristic (scope, novelty, risk, iteration) for routing each task to the right model tier, a worked example, and a free…

  34. A controlled comparison of ungated vs gated AI patching on SWE-bench Lite. Full methodology, results, and reproducible scripts in the repo.

  35. Evaluating LLM agents on benchmarks like SWE-Bench and GAIA can be expensive, time-consuming, and requires complex infrastructure. A single evaluation can cost thousands of dollars and take days to complete.

  36. Senior SWE-Bench We treat agents like senior engineers, so why evaluate them like junior engineers? Senior engineers build features without over-specified requirements Senior SWE-Bench feature tasks have realistic instructions that read li…

  37. Following up on the v0.9 post here a week back. Two things have happened since that I want to put on the record, one honest correction, one shipping update.

  38. Put together a comparison of every benchmark I could find from the official announcement and early coverage. Figured this might save people some time.

  39. So recently I decided that it would be nice to run my agent against some popular benchmarks. And oh my god, the cost to run a single benchmark, such as terminal-bench or swe-bench will cost you thousands of dollars in tokens just for a sin…

  40. I have been thinking about the Fable 5 to GLM-5.2 sequence as one event rather than two. June 9, Anthropic ships Fable 5, the Mythos line opens to the public for the first time, SWE-bench Verified at 95 percent, people calling it the best…

  41. World Model MCP Enforcement, provenance, and harness-neutral memory for AI coding agents. A temporal knowledge graph that validates code changes against learned constraints at the edit boundary, re-injects relevant context after compaction…

  42. Your coding agent can cheat on its tests, and the benchmark won't catch it Almost every way we evaluate an AI agent looks at the same thing: the final output. Did the answer look right.

  43. clawmark clawmark is a local Rust CLI for answering one focused question: Which of these two CLAUDE.md files performs better on a small SWE-bench Lite smoke set? v1 compares exactly two local variant files against five bundled SWE-bench Li…

  44. Moonshot AI released Kimi K2.7 Code on June 12 — a coding-focused open-weight model. Key specs: - 1 trillion params (MoE, 32B active, 384 experts) - 256K context window - Modified MIT license — weights on Hugging Face - $0.95/M input, $4.0…

  45. AI coding agents have rapidly transformed software engineering, powering widely used interactive coding assistants. Despite their interactive real-world use, existing benchmarks evaluate them as fully-autonomous systems.

  46. Evaluating background coding agents on financial SWE work

  47. General-purpose agents such as OpenClaw are increasingly used as autonomous tool users, but their coding ability is difficult to measure under SWE-bench: a generic agent does not by itself satisfy the clean Docker workspace, patch, and pre…

  48. Repository-level benchmarks for evaluating Large Language Model (LLM) code repair on Secure Multi-Party Computation (MPC) software do not yet exist, and directly transplanting general-purpose benchmarks such as SWE-bench fails on three str…

  49. Two numbers on this model that don't sit comfortably with each other. The Pro config posts coding scores near the top of every board, 80.6 on SWE-bench Verified and 93.5 on LiveCodeBench.

  50. When Anthropic released Claude Fable 5 this week, my feed filled up with the same benchmark charts within hours. SWE-bench scores, agentic coding numbers, the Stripe migration story.

  51. I can’t wait for DeepSWE to include Fable 5 in the benchmark so people can understand that Mythos is mostly hype. In the official benchmark, Opus 4.8 was supposed to be better at programming than 5.5 (SWE-bench Pro), but in one real benchm…

  52. Microsoft just released MAI-Code-1-Flash — a 5B parameter coding model built for fast, efficient developer assistance. Numbers that caught my eye: - 51.2% on SWE-Bench Pro (Claude Haiku 4.5 scores 35.2%) - 71.6% on SWE-Bench Verified (Haik…

  54. MiniMax (official) @MiniMax_AI Introducing MiniMax M3: The First Open-Weights Model to Combine Three Frontier Capabilities - Coding & Agentic Frontier: 59.0% SWE-Bench Pro, 66.0% Terminal Bench 2.1, 34.8% SWE-fficiency, 28.8% KernelBench H…

  55. Claude Code degraded for the week before Opus 4.8's release Our SWE-Bench-Pro tracker caught a statistically significant, weeklong drop in Claude Code's pass rate just before Opus 4.8 shipped, and the recovery that followed. We run Claude…

  56. This is mini-swe-agent v2 Read the migration guide. For the previous version, check out the v1 documentation or the v1 branch.

  57. From the website, it touts: Contamination free: Tasks are written from scratch, not adapted from existing commits or PRs, so no model has seen the solution during pretraining. High diversity: Tasks span a broad pool of 91 repositories acro…

  58. Hi all, Sorry for going missing — we’ve been collecting a larger, higher-quality set of more complex tasks. We’re excited to share a major leaderboard update covering the past three months.

  59. swebench-verified A three-stage agent pipeline for SWE-bench Verified, built to be re-run and inspected by skeptics. The point of this repo is not the score.

  60. Bito's AI Architect cuts Claude Code's token cost by 47% on SWE-Bench Pro. It gives the coding agent codebase context, a continuously updated, structured map of every repository served over MCP.

  61. https://preview.redd.it/jyiiwn2o0f2h1.png?width=962&format=png&auto=webp&s=6a96d2b9fe7bffcc75e8d5865161ec3727d46d58 Link to blog : https://qwen.ai/blog?id=qwen3.7

  62. AI Architect tops SWE-Bench Pro Claude Opus 4.6 Without context with system context Even advanced coding agents resolve fewer than 52% of tasks when changes span large codebases and require coordinated, multi-file updates. These long-horiz…

  63. Saw the tech-insider breakdown comparing Claude Code and Cursor head-to-head this week. Numbers are kind of hard to ignore: 80.8% SWE-bench for Claude Code, 74% for Cursor, and a 67% blind-quality win rate for Claude Code on real tasks.

  64. The 30-Second Verdict Best for science & reasoning: Gemini 3.1 Pro — leads GPQA Diamond (94.3%) and ARC-AGI-2 (77.1%). Best for coding: ChatGPT (GPT-5.5) — 88.7% on SWE-Bench Verified.

  65. DeepSeek V4: The Open-Source Model Frontier Labs Feared DeepSeek V4 ships under MIT with $0.30/M output tokens — 83x cheaper than Claude Opus 4.7 — while scoring 80.6% on SWE-bench Verified. The agentic-coding price floor just moved an ord…

  66. Agentic problem solving in its current state is very brittle. I fell in love with it, but it creates as many problems as it solves.

  67. The Artificial Analysis Coding Agent Index includes 3 leading benchmarks that represent a broad spectrum of coding agent use: ➤ SWE-Bench-Pro-Hard-AA, 150 realistic coding tasks that frontier models struggle with, sampled from Scale AI’s S…

  68. ./ProgramBench Can language models rebuild programs from scratch? Given only a compiled binary and its documentation, agents must architect and implement a complete codebase that reproduces the original program's behavior.

  69. I’m working on an assessment where I need to create a coding task (basically SWE-bench style). The idea is: take an existing repo (I’m using pydantic) write tests that fail on the current code provide a patch that fixes it and the task sho…

  70. From 1930 to SWE-bench Models and training data We fine-tune Alec Radford's 1930 vintage LLM — pre-trained only on pre-1931 data — to solve SWE-bench issues. After just 250 training examples the model lands its first fix (a small patch to…

  71. Trying some if Bartowski's Q4 quants. Using Vulkan with the latest main branch as of a few hours ago.

  72. Mythos’ system card contains the following graph to support its argument that Mythos performs better on SWE-bench: Anthropic and others are worried LLMs are memorizing SWE-bench, so they asked an LLM to estimate the probability that a solu…

  73. Here's what I did: Built a proxy that intercepts Codex's calls to OpenAI and rewrites them on the fly. Replayed 3,807 rounds of SWE-bench Verified traces through it: avg prompt 44k → 6k tokens (-87%).

  74. If you had to build a context window manager in 24h, would you stick to the existing model or come up with something better? Here's what I did: 1.

  75. "SWE-bench Verified, Pro, and Multilingual: Our memorization screens flag a subset of problems in these SWE-bench evals." https://www.anthropic.com/news/claude-opus-4-7

  76. 27B Dense vs. 35B-A3B MoE): - Dense still holds the crown: It still wins out on most tasks overall.

  78. Opus 4.7 landed on Cursor yesterday. The model is better — SWE-bench jumped from 80.8% to 87.6%.

  79. Anthropic shipped Opus 4.7 yesterday. The headline numbers are real: 64.3% on SWE-bench Pro (up from 53.4%), best-in-class on MCP-Atlas at 77.3% for multi-tool orchestration, 14% improvement on multi-step agentic reasoning, and one-third f…

  80. Shipped today. The benchmarks are real: 87.6% SWE-bench (from 80.8%), +13% on coding tasks, 3x more resolved production tasks on Rakuten-SWE-Bench.

  81. Anthropic published the Advisor Strategy this week. The idea: a cheaper model does the actual work, a stronger model only gets consulted on hard decisions.

  82. Leaks point to late April release. Key specs 1M token context window Native multimodal (image/video input) Projected ~85% SWE-Bench Verified (ties or beats Claude Opus 4.6) Base model remains free.

  83. So I have been running gpt and glm-5.1 side by side lately and tbh the gap is way smaller than what im paying for On SWE-Bench Pro glm-5.1 actually took the top spot globally, beat gpt-5.4 and opus 4.6. overall coding score is like 55 vs g…

  84. An independent audit of agentic scaffolding and harnesses. We analyze how agent workflows, codebase documentation, and test verification impact performance compared to raw base models like GPT-5.4, Gemini 3.1 Pro, and Claude Code.

  85. TokenMonopoly Live leaderboard of AI API deals — pricing, subscriptions, and SWE-bench scores for Claude, GPT, Gemini, Kimi, DeepSeek, Llama and more. Compare 27 benchmarked models across 96 hosts by price-per-performance, refreshed daily.

  86. I thought GPT models felt slow and token hungry, and Claude models were faster. I was wrong.

  87. I started out running agent evaluations locally because most ai agent benchmarks and examples assume that setup. And to be fair local runs do work for debugging and small experiments.

  88. We're bringing the advisor strategy to the Claude Platform. Pair Opus as an advisor with Sonnet or Haiku as an executor, and your agents can consult Opus mid-task when they hit a hard decision.

← all threads